[test-triage] chip_sw_lc_ctrl_rand_to_scrap fails for some seeds

[rswarbrick]

Hierarchy of regression failure

Chip level

Failure Description

This started failing in last night's nightly (23rd April). The failure log is:

2.chip_sw_lc_ctrl_rand_to_scrap.96544019929050351620551105538803383579697781106412772125544720722752507385078
        Line 749, in log /container/opentitan-public/scratch/os_regression/chip_earlgrey_asic-sim-vcs/2.chip_sw_lc_ctrl_rand_to_scrap/latest/run.log

          UVM_ERROR @ 2923.567685 us: (chip_sw_base_vseq.sv:770) [uvm_test_top.env.virtual_sequencer.chip_sw_lc_ctrl_scrap_vseq] Check failed (status_val) >> dummy.num() == 0 (45948 [0xb37c] vs 0 [0x0]) Unexpected status error b37c000
          UVM_INFO @ 2923.567685 us: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
          --- UVM Report catcher Summary ---

Steps to Reproduce

• GitHub Revision: 41bc3e0

Tests with similar or related failures

No response

[rswarbrick]

I've just done a minimal bit of triage and have reproduced with the current head of master (f914807). To see the error, run:

util/dvsim/dvsim.py hw/top_earlgrey/dv/chip_sim_cfg.hjson -i chip_sw_lc_ctrl_rand_to_scrap --tool=vcs --fixed-seed=96544019929050351620551105538803383579697781106412772125544720722752507385078

This takes a reasonable amount of time (5m30s on my laptop)

[rswarbrick]

I've done a bit of bisection and found the first commit where this particular seed fails: fc3a071.

This change is fundamentally just changing the timing in the chip-level sequence, so I suspect it's just moving around the choice of seeds to make the test fail. Not sure that bisection will help much more. :-(

(Bisection run:

• fc98685 (Jan 2nd): Good
• 2b9338d (Mar 2nd): Good
• 2e5f341 (Mar 26th): Good
• fc3a071 (Mar 26th): Bad
• c173674 (Mar 27th): Bad
• 816e06d (Mar 28th): Bad
• 3b4bd98 (Mar 29th): Bad
• 94f2cf5 (Apr 2nd): Bad
• 559db22 (Apr 11th): Bad)

[rswarbrick]

I just ran a bunch of tests on the head of my "work" branch (which should be pretty much equivalent to f914807). I get one failure in 32 seeds (which will explain why the nightly failures are a bit sporadic: we're running 3 seeds each time)

[antmarzam]

I saw the following related failure when triaging the nightly regression:

UVM_ERROR @ * us: (chip_sw_base_vseq.sv:770) [chip_sw_lc_ctrl_scrap_vseq] Check failed (status_val) >> dummy.num() == * (* [*] vs * [*]) Unexpected status error *c* has 1 failures:

Test chip_sw_lc_ctrl_rma_to_scrap has 1 failures.
0.chip_sw_lc_ctrl_rma_to_scrap.76330856294524163685711952422459750785499940999804025761801481247434007489402
Line 776, in log /container/opentitan-public/scratch/os_regression/chip_earlgrey_asic-sim-vcs/0.chip_sw_lc_ctrl_rma_to_scrap/latest/run.log

  UVM_ERROR @ 2873.518058 us: (chip_sw_base_vseq.sv:770) [uvm_test_top.env.virtual_sequencer.chip_sw_lc_ctrl_scrap_vseq] Check failed (status_val) >> dummy.num() == 0 (507904 [0x7c000] vs 0 [0x0]) Unexpected status error 7c000000
  UVM_INFO @ 2873.518058 us: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---