ci: Update Earlgrey test lists & add new regression run
#251
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The `rsa_3072_verify` test was removed from Upstream OpenTitan Earlgrey due to being obsolete, but was previously expected to be passing here. Remove it from the list of passing tests. Signed-off-by: Alex Jones <alex.jones@lowrisc.org>
AlexJones0
force-pushed
the
update_eg_tests
branch
3 times, most recently
from
4d690bd to
050c17d
Compare
rivos-eblot
requested changes
Oct 22, 2025
AlexJones0
force-pushed
the
update_eg_tests
branch
2 times, most recently
from
f5e41a8 to
31256dc
Compare
We expect this test to pass, but recent upstream changes increase the time to emulate this test by a lot and it will often take 2-5+ minutes to execute depending on host performance and load. Mark it flaky instead of removing it because it could theoretically pass if the runner is fast enough, but it is not expected to pass with the standard 60 second timeout we give. Signed-off-by: Alex Jones <alex.jones@lowrisc.org>
Add a couple more OpenTitan tests that are commonly seen as being flaky in the logs (previously were not expected to pass). Signed-off-by: Alex Jones <alex.jones@lowrisc.org>
When the script was extended to support specifying different execution environments and to collect all the Earlgrey test statuses together in one place, I forgot to update the accompanying documentation. Make sure that is up to date with this commit. Signed-off-by: Alex Jones <alex.jones@lowrisc.org>
With appropriate upstream OpenTitan support, we now expect a large amount of ROM end-to-end tests to pass in QEMU. Add all of these tests to the test list and mark them as expected passes to catch regressions. Signed-off-by: Alex Jones <alex.jones@lowrisc.org>
AlexJones0
force-pushed
the
update_eg_tests
branch
3 times, most recently
from
7b9470b to
55566c8
Compare
When testing a specific execution environment, you can set a maximum test timeout to use for that environment. For environments where tests are expected to be quick this can be used to decrease CI times (tests using unimplemented features may timeout, so this provides a smaller upper bound on test execution). For environments where tests may take a while when executed in parallel on a standard GitHub runner (maybe longer than 60 seconds), this can be used to increase the timeout for just executions in that environment. Signed-off-by: Alex Jones <alex.jones@lowrisc.org>
With upstream support, there are now a variety of ROM_EXT end-to-end tests that we might expect to pass or at least be flaky. To ensure that regressions on these more complicated tests are being caught, add a regression run for the non-SiVal ROM_EXT environment, which most of these tests use. Since this is a separate job in the regression workflow it will be run in parallel by a separate runner, and so should not increase existing CI times (unless the time to execute all ROM_EXT test overshadows the time for executing other existing execution environments). Signed-off-by: Alex Jones <alex.jones@lowrisc.org>
Add newly passing ROM_EXT tests to be checked in Earlgrey regressions now that these are properly supported in upstream OpenTitan. Uses the increased timeout to ensure that even when running in parallel on slower runners, these tests should pass most of the time. Mark the couple of tests that are most likely to be flaky due to strict timing checks in the ROM as such, to hopefully avoid CI flakiness. Signed-off-by: Alex Jones <alex.jones@lowrisc.org>
AlexJones0
force-pushed
the
update_eg_tests
branch
from
55566c8 to
85a4857
Compare
rivos-eblot
approved these changes
Oct 23, 2025
jwnrt
approved these changes
Oct 23, 2025
6 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
earlgrey_1.0.0(remove a test target that was removed, add newly passing end-to-end ROM and ROM_EXT tests).