Skip to content

OpenTitan regression script bug fixes #166

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Sep 4, 2025
Merged

OpenTitan regression script bug fixes #166

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
8dba282
[ot] .github/workflows: opentitan_regression: ensure test step failur…
jwnrt Sep 3, 2025
d171f72
[ot] scripts/opentitan: bazel: handle test list sorting in script
jwnrt Sep 3, 2025
d1cb726
[ot] scripts/opentitan: bazel: constrain passing test regex
jwnrt Sep 3, 2025
0e9e1a9
[ot] scripts/opentitan: bazel: add more tests to the flaky test list
jwnrt Sep 3, 2025
67feaa2
[ot] docs/opentitan: regressions: add docs
jwnrt Sep 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/opentitan_regression.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,9 @@ jobs:
ninja -C build qemu-system-riscv32

- name: Run OpenTitan regressions
shell: bash
run: |
set -o pipefail
./qemu/scripts/opentitan/run-bazel-tests.sh ./ qemu \
| tee test_results.txt

Expand Down
26 changes: 26 additions & 0 deletions docs/opentitan/regressions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# OpenTitan regressions

The `run-bazel-tests.sh` script and accompanying CI workflow allow us to run
regression tests against the OpenTitan repository. Currently only Earlgrey tests
are supported.

With a checkout of OpenTitan, the script can be run like this:

```sh
./scripts/opentitan/run-bazel-tests.sh path/to/opentitan path/to/qemu
```

The script will execute all QEMU-compatible tests using QEMU as it was built
at the given path. The test results will be compared against two lists checked
into this repository:

* `scripts/opentitan/tests-passing.txt`
* `scripts/opentitan/tests-flaky.txt`

All tests in `tests-passing.txt` are expected to pass. Failures indicate a
regression in either QEMU or Earlgrey. The script will fail if there is a
mismatch between the tests that we expect to pass and the actual results.

Some tests may be flaky and pass or fail on different runs of the same QEMU
and OpenTitan checkouts. These tests can be added to the `tests-flaky.txt` list
to cause the script to ignore them.
2 changes: 2 additions & 0 deletions docs/opentitan/tools.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,8 @@ of options and the available features.
* `treillis/` directory contains the test application to test the [GPIO](gpio.md) device.
* [`uartmux.py`](uartmux.md) is a tiny stream wrapper to help dealing with multiple QEMU output
streams, typically multiple virtual UARTs.
* [`run-bazel-tests.sh`](regressions.md) is a script for running the OpenTitan test suite against
a local build of QEMU.

## Python modules

Expand Down
14 changes: 4 additions & 10 deletions scripts/opentitan/run-bazel-tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,12 +37,6 @@ qemu_path="$(realpath "$qemu_path")"
passing_tests_path="${qemu_path}/scripts/opentitan/tests-passing.txt"
flaky_tests_path="${qemu_path}/scripts/opentitan/tests-flaky.txt"

# Check the test lists are sorted:
if ! sort --check "$passing_tests_path" || ! sort --check "$flaky_tests_path"; then
echo >&2 "ERROR: test lists must be sorted!"
exit 1
fi

# Ensure QEMU has already been built in `./build`:
if [ ! -x "${qemu_path}/build/qemu-system-riscv32" ]; then
echo >&2 "ERROR: expected QEMU binary at '${qemu_path}/build/qemu-system-riscv32'"
Expand Down Expand Up @@ -80,14 +74,14 @@ cd "$opentitan_path" >/dev/null

## COMPARE RESULTS

# Write the flaky test list to a flat file:
printf "%s\n" "$flaky_tests_path" | sort > "$flaky"
# Ensure the flaky tests are sorted with the current locale.
sort "$flaky_tests_path" > "$flaky"

# Load the list of passing tests and strip flaky tests:
comm -23 "$passing_tests_path" "$flaky" > "$expected"
sort "$passing_tests_path" | comm -23 - "$flaky" > "$expected"

# Find all the tests which passed in Bazel:
grep "PASSED" "$results" | cut -d' ' -f1 | sort > "$all_passed"
grep "PASSED[^:]" "$results" | cut -d' ' -f1 | sort > "$all_passed"

# Filter out the flaky tests:
comm -23 "$all_passed" "$flaky" > "$passed"
Expand Down
10 changes: 10 additions & 0 deletions scripts/opentitan/tests-flaky.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,15 @@
//sw/device/silicon_creator/lib/ownership:owner_verify_functest_sim_qemu_rom_with_fake_keys
//sw/device/silicon_creator/lib/sigverify:mod_exp_ibex_functest_hardcoded_sim_qemu_rom_with_fake_keys
//sw/device/silicon_creator/lib/sigverify/sphincsplus/test:verify_test_kat1_sim_qemu_rom_with_fake_keys
//sw/device/silicon_creator/lib/sigverify/sphincsplus/test:verify_test_kat2_sim_qemu_rom_with_fake_keys
//sw/device/silicon_creator/lib/sigverify/sphincsplus/test:verify_test_kat3_sim_qemu_rom_with_fake_keys
//sw/device/silicon_creator/lib/sigverify/sphincsplus/test:verify_test_kat4_sim_qemu_rom_with_fake_keys
//sw/device/silicon_creator/lib/sigverify/sphincsplus/test:verify_test_kat5_sim_qemu_rom_with_fake_keys
//sw/device/silicon_creator/lib/sigverify/sphincsplus/test:verify_test_kat6_sim_qemu_rom_with_fake_keys
//sw/device/silicon_creator/lib/sigverify/sphincsplus/test:verify_test_kat7_sim_qemu_rom_with_fake_keys
//sw/device/silicon_creator/lib/sigverify/sphincsplus/test:verify_test_kat9_sim_qemu_rom_with_fake_keys
Comment on lines +3 to +10
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find these failures surprising since I've never seen them fail personally. I think it's fine to add for now though if you've seen this, and we can remove them again later if needed.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For me it fails more often than not:

//sw/device/silicon_creator/lib/sigverify/sphincsplus/test:verify_test_kat4_sim_qemu_rom_with_fake_keys TIMEOUT in 48 out of 50 in 60.2s

It must be timing related

Copy link

@AlexJones0 AlexJones0 Sep 3, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I think it might be timing related - but not in the way you think. Running that test myself locally:

INFO: Build completed successfully, 6 total actions
//sw/device/silicon_creator/lib/sigverify/sphincsplus/test:verify_test_kat4_sim_qemu_rom_with_fake_keys PASSED in 10.6s
  Stats over 5 runs: max = 10.6s, min = 10.6s, avg = 10.6s, dev = 0.0s

I think it heavily depends upon the speed of the host machine. This could explain why you keep seeing errors on the CI runner (and locally) but I don't? What is the failure mode - is it just a genuine timeout?

If so, these tests probably need tagging with longer timeouts in OpenTitan.

//sw/device/tests/autogen:alert_test_sim_qemu_rom_with_fake_keys
//sw/device/tests/crypto:aes_gcm_timing_test_sim_qemu_rom_with_fake_keys
//sw/device/tests/crypto:rsa_2048_keygen_functest_sim_qemu_rom_with_fake_keys
//sw/device/tests:edn_auto_mode_sim_qemu_rom_with_fake_keys
//sw/device/tests:rv_core_ibex_rnd_test_sim_qemu_rom_with_fake_keys
50 changes: 25 additions & 25 deletions scripts/opentitan/tests-passing.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,10 @@
//sw/device/silicon_creator/lib/drivers:rstmgr_functest_sim_qemu_rom_with_fake_keys
//sw/device/silicon_creator/lib/drivers:uart_functest_sim_qemu_rom_with_fake_keys
//sw/device/silicon_creator/lib/ownership:owner_verify_functest_sim_qemu_rom_with_fake_keys
//sw/device/silicon_creator/lib/sigverify:ecdsa_p256_verify_functest_sim_qemu_rom_with_fake_keys
//sw/device/silicon_creator/lib/sigverify:rsa_verify_functest_sim_qemu_rom_with_fake_keys
//sw/device/silicon_creator/lib/sigverify:sigverify_dynamic_functest_hardcoded_sim_qemu_rom_with_fake_keys
//sw/device/silicon_creator/lib/sigverify:sigverify_dynamic_functest_wycheproof_sim_qemu_rom_with_fake_keys
//sw/device/silicon_creator/lib/sigverify/sphincsplus/test:fors_test_sim_qemu_rom_with_fake_keys
//sw/device/silicon_creator/lib/sigverify/sphincsplus/test:mgf1_test_sim_qemu_rom_with_fake_keys
//sw/device/silicon_creator/lib/sigverify/sphincsplus/test:thash_test_sim_qemu_rom_with_fake_keys
Expand All @@ -26,11 +30,23 @@
//sw/device/silicon_creator/lib/sigverify/sphincsplus/test:verify_test_kat8_sim_qemu_rom_with_fake_keys
//sw/device/silicon_creator/lib/sigverify/sphincsplus/test:verify_test_kat9_sim_qemu_rom_with_fake_keys
//sw/device/silicon_creator/lib/sigverify/sphincsplus/test:wots_test_sim_qemu_rom_with_fake_keys
//sw/device/silicon_creator/lib/sigverify:ecdsa_p256_verify_functest_sim_qemu_rom_with_fake_keys
//sw/device/silicon_creator/lib/sigverify:rsa_verify_functest_sim_qemu_rom_with_fake_keys
//sw/device/silicon_creator/lib/sigverify:sigverify_dynamic_functest_hardcoded_sim_qemu_rom_with_fake_keys
//sw/device/silicon_creator/lib/sigverify:sigverify_dynamic_functest_wycheproof_sim_qemu_rom_with_fake_keys
//sw/device/silicon_creator/lib/sigverify:spx_verify_functest_sim_qemu_rom_with_fake_keys
//sw/device/tests:aes_entropy_test_sim_qemu_rom_with_fake_keys
//sw/device/tests:aes_idle_test_sim_qemu_rom_with_fake_keys
//sw/device/tests:aes_smoketest_sim_qemu_rom_with_fake_keys
//sw/device/tests:alert_handler_lpg_clkoff_test_sim_qemu_rom_with_fake_keys
//sw/device/tests:alert_handler_lpg_reset_toggle_test_sim_qemu_rom_with_fake_keys
//sw/device/tests:alert_handler_ping_ok_test_sim_qemu_rom_with_fake_keys
//sw/device/tests:aon_timer_irq_test_sim_qemu_rom_with_fake_keys
//sw/device/tests:clkmgr_jitter_frequency_test_sim_qemu_rom_with_fake_keys
//sw/device/tests:clkmgr_jitter_test_sim_qemu_rom_with_fake_keys
//sw/device/tests:clkmgr_off_aes_trans_test_sim_qemu_rom_with_fake_keys
//sw/device/tests:clkmgr_off_hmac_trans_test_sim_qemu_rom_with_fake_keys
//sw/device/tests:clkmgr_off_kmac_trans_test_sim_qemu_rom_with_fake_keys
//sw/device/tests:clkmgr_off_otbn_trans_test_sim_qemu_rom_with_fake_keys
//sw/device/tests:clkmgr_smoketest_sim_qemu_rom_with_fake_keys
//sw/device/tests:coverage_test_sim_qemu_rom_with_fake_keys
//sw/device/tests:crt_test_sim_qemu_rom_with_fake_keys
//sw/device/tests/crypto:aes_functest_sim_qemu_rom_with_fake_keys
//sw/device/tests/crypto:aes_gcm_functest_sim_qemu_rom_with_fake_keys
//sw/device/tests/crypto:aes_kwp_functest_sim_qemu_rom_with_fake_keys
Expand Down Expand Up @@ -60,22 +76,6 @@
//sw/device/tests/crypto:sha384_functest_sim_qemu_rom_with_fake_keys
//sw/device/tests/crypto:sha512_functest_sim_qemu_rom_with_fake_keys
//sw/device/tests/crypto:symmetric_keygen_functest_sim_qemu_rom_with_fake_keys
//sw/device/tests:aes_entropy_test_sim_qemu_rom_with_fake_keys
//sw/device/tests:aes_idle_test_sim_qemu_rom_with_fake_keys
//sw/device/tests:aes_smoketest_sim_qemu_rom_with_fake_keys
//sw/device/tests:alert_handler_lpg_clkoff_test_sim_qemu_rom_with_fake_keys
//sw/device/tests:alert_handler_lpg_reset_toggle_test_sim_qemu_rom_with_fake_keys
//sw/device/tests:alert_handler_ping_ok_test_sim_qemu_rom_with_fake_keys
//sw/device/tests:aon_timer_irq_test_sim_qemu_rom_with_fake_keys
//sw/device/tests:clkmgr_jitter_frequency_test_sim_qemu_rom_with_fake_keys
//sw/device/tests:clkmgr_jitter_test_sim_qemu_rom_with_fake_keys
//sw/device/tests:clkmgr_off_aes_trans_test_sim_qemu_rom_with_fake_keys
//sw/device/tests:clkmgr_off_hmac_trans_test_sim_qemu_rom_with_fake_keys
//sw/device/tests:clkmgr_off_kmac_trans_test_sim_qemu_rom_with_fake_keys
//sw/device/tests:clkmgr_off_otbn_trans_test_sim_qemu_rom_with_fake_keys
//sw/device/tests:clkmgr_smoketest_sim_qemu_rom_with_fake_keys
//sw/device/tests:coverage_test_sim_qemu_rom_with_fake_keys
//sw/device/tests:crt_test_sim_qemu_rom_with_fake_keys
//sw/device/tests:csrng_kat_test_sim_qemu_rom_with_fake_keys
//sw/device/tests:csrng_smoketest_sim_qemu_rom_with_fake_keys
//sw/device/tests:edn_sw_mode_sim_qemu_rom_with_fake_keys
Expand Down Expand Up @@ -131,19 +131,19 @@
//third_party/riscv-compliance:C-J_sim_qemu_rom_with_fake_keys
//third_party/riscv-compliance:C-LI_sim_qemu_rom_with_fake_keys
//third_party/riscv-compliance:C-LUI_sim_qemu_rom_with_fake_keys
//third_party/riscv-compliance:C-LWSP_sim_qemu_rom_with_fake_keys
//third_party/riscv-compliance:C-LW_sim_qemu_rom_with_fake_keys
//third_party/riscv-compliance:C-LWSP_sim_qemu_rom_with_fake_keys
//third_party/riscv-compliance:C-MV_sim_qemu_rom_with_fake_keys
//third_party/riscv-compliance:C-OR_sim_qemu_rom_with_fake_keys
//third_party/riscv-compliance:C-SLLI_sim_qemu_rom_with_fake_keys
//third_party/riscv-compliance:C-SRAI_sim_qemu_rom_with_fake_keys
//third_party/riscv-compliance:C-SRLI_sim_qemu_rom_with_fake_keys
//third_party/riscv-compliance:C-SUB_sim_qemu_rom_with_fake_keys
//third_party/riscv-compliance:C-SWSP_sim_qemu_rom_with_fake_keys
//third_party/riscv-compliance:C-SW_sim_qemu_rom_with_fake_keys
//third_party/riscv-compliance:C-SWSP_sim_qemu_rom_with_fake_keys
//third_party/riscv-compliance:C-XOR_sim_qemu_rom_with_fake_keys
//third_party/riscv-compliance:DIVU_sim_qemu_rom_with_fake_keys
//third_party/riscv-compliance:DIV_sim_qemu_rom_with_fake_keys
//third_party/riscv-compliance:DIVU_sim_qemu_rom_with_fake_keys
//third_party/riscv-compliance:I-ADD-01_sim_qemu_rom_with_fake_keys
//third_party/riscv-compliance:I-ADDI-01_sim_qemu_rom_with_fake_keys
//third_party/riscv-compliance:I-AND-01_sim_qemu_rom_with_fake_keys
Expand Down Expand Up @@ -192,9 +192,9 @@
//third_party/riscv-compliance:I-SW-01_sim_qemu_rom_with_fake_keys
//third_party/riscv-compliance:I-XOR-01_sim_qemu_rom_with_fake_keys
//third_party/riscv-compliance:I-XORI-01_sim_qemu_rom_with_fake_keys
//third_party/riscv-compliance:MULH_sim_qemu_rom_with_fake_keys
//third_party/riscv-compliance:MULHSU_sim_qemu_rom_with_fake_keys
//third_party/riscv-compliance:MULHU_sim_qemu_rom_with_fake_keys
//third_party/riscv-compliance:MULH_sim_qemu_rom_with_fake_keys
//third_party/riscv-compliance:MUL_sim_qemu_rom_with_fake_keys
//third_party/riscv-compliance:REMU_sim_qemu_rom_with_fake_keys
//third_party/riscv-compliance:REM_sim_qemu_rom_with_fake_keys
//third_party/riscv-compliance:REMU_sim_qemu_rom_with_fake_keys
Loading