Potentially Wrong Golden Values For Some Tests

[AmirhosseinPoolad]

We're currently working on restarting nightly tests and did a test run last night. Unfortunately there were some QOR errors so the tests failed. I don't think it's an issue with the workflow so I'm thinking that some golden values weren't updated. Here's the log for the failed tests:

Parsing test results...
scripts/parse_vtr_task.py -l vtr-verilog-to-routing/vtr-verilog-to-routing/vtr_flow/tasks/regression_tests/vtr_reg_nightly_test7/task_list.txt
Elapsed time: 34.47 seconds

Calculating QoR results...

regression_tests/vtr_reg_nightly_test7/titan_other_run_flat...[Fail]
[Fail]
stratixiv_arch.timing.xml/carpat_stratixiv_arch_timing.blif/common num_LAB relative value 1.063959390862944 outside of range [0.95,1.05] and not equal to golden value: 985.0

regression_tests/vtr_reg_nightly_test7/titan_other_run_flat...[Fail]
[Fail]
stratixiv_arch.timing.xml/uoft_raytracer_stratixiv_arch_timing.blif/common num_LAB relative value 1.146516393442623 outside of range [0.95,1.05] and not equal to golden value: 976.0
regression_tests/vtr_reg_nightly_test7/verify_rr_graph_run_flat...[Pass]

regression_tests/vtr_reg_nightly_test7/3d_cb_titan_other_auto_bb...[Fail]
[Fail]
3d_full_OPIN_inter_die_stratixiv_arch.timing.xml/carpat_stratixiv_arch_timing.blif/common num_LAB relative value 1.063959390862944 outside of range [0.95,1.05] and not equal to golden value: 985.0

regression_tests/vtr_reg_nightly_test7/3d_cb_titan_other_auto_bb...[Fail]
[Fail]
3d_full_OPIN_inter_die_stratixiv_arch.timing.xml/smithwaterman_stratixiv_arch_timing.blif/common num_LAB relative value 1.080906148867314 outside of range [0.95,1.05] and not equal to golden value: 2163.0

regression_tests/vtr_reg_nightly_test7/3d_cb_titan_other_auto_bb...[Fail]
[Fail]
3d_full_OPIN_inter_die_stratixiv_arch.timing.xml/uoft_raytracer_stratixiv_arch_timing.blif/common num_LAB relative value 1.146516393442623 outside of range [0.95,1.05] and not equal to golden value: 976.0

regression_tests/vtr_reg_nightly_test7/3d_cb_titan_other_cube_bb...[Fail]
[Fail]
3d_full_OPIN_inter_die_stratixiv_arch.timing.xml/carpat_stratixiv_arch_timing.blif/common num_LAB relative value 1.063959390862944 outside of range [0.95,1.05] and not equal to golden value: 985.0

regression_tests/vtr_reg_nightly_test7/3d_cb_titan_other_cube_bb...[Fail]
[Fail]
3d_full_OPIN_inter_die_stratixiv_arch.timing.xml/smithwaterman_stratixiv_arch_timing.blif/common num_LAB relative value 1.080906148867314 outside of range [0.95,1.05] and not equal to golden value: 2163.0

regression_tests/vtr_reg_nightly_test7/3d_cb_titan_other_cube_bb...[Fail]
[Fail]
3d_full_OPIN_inter_die_stratixiv_arch.timing.xml/uoft_raytracer_stratixiv_arch_timing.blif/common num_LAB relative value 1.146516393442623 outside of range [0.95,1.05] and not equal to golden value: 976.0

regression_tests/vtr_reg_nightly_test7/3d_sb_titan_other_auto_bb...[Fail]
[Fail]
3d_SB_inter_die_stratixiv_arch.timing.xml/carpat_stratixiv_arch_timing.blif/common num_LAB relative value 1.0618034447821683 outside of range [0.95,1.05] and not equal to golden value: 987.0

regression_tests/vtr_reg_nightly_test7/3d_sb_titan_other_auto_bb...[Fail]
[Fail]
3d_SB_inter_die_stratixiv_arch.timing.xml/smithwaterman_stratixiv_arch_timing.blif/common num_LAB relative value 1.0759318913943856 outside of range [0.95,1.05] and not equal to golden value: 2173.0

regression_tests/vtr_reg_nightly_test7/3d_sb_titan_other_auto_bb...[Fail]
[Fail]
3d_SB_inter_die_stratixiv_arch.timing.xml/uoft_raytracer_stratixiv_arch_timing.blif/common num_LAB relative value 1.1453428863868986 outside of range [0.95,1.05] and not equal to golden value: 977.0

regression_tests/vtr_reg_nightly_test7/3d_sb_titan_other_auto_bb...[Fail]
[Fail]
3d_SB_inter_die_stratixiv_arch.timing.xml/murax_stratixiv_arch_timing.blif/common num_LAB relative value 1.0684931506849316 outside of range [0.95,1.05] and not equal to golden value: 73.0

regression_tests/vtr_reg_nightly_test7/3d_sb_titan_other_per_layer_bb...[Fail]
[Fail]
3d_SB_inter_die_stratixiv_arch.timing.xml/carpat_stratixiv_arch_timing.blif/common num_LAB relative value 1.0618034447821683 outside of range [0.95,1.05] and not equal to golden value: 987.0

regression_tests/vtr_reg_nightly_test7/3d_sb_titan_other_per_layer_bb...[Fail]
[Fail]
3d_SB_inter_die_stratixiv_arch.timing.xml/fir_cascade_stratixiv_arch_timing.blif/common placed_setup_WNS_est relative value 1.4807930004683074 outside of range [0.5,1.4] and not equal to golden value: -3.90769

regression_tests/vtr_reg_nightly_test7/3d_sb_titan_other_per_layer_bb...[Fail]
[Fail]
3d_SB_inter_die_stratixiv_arch.timing.xml/smithwaterman_stratixiv_arch_timing.blif/common num_LAB relative value 1.0759318913943856 outside of range [0.95,1.05] and not equal to golden value: 2173.0

regression_tests/vtr_reg_nightly_test7/3d_sb_titan_other_per_layer_bb...[Fail]
[Fail]
3d_SB_inter_die_stratixiv_arch.timing.xml/uoft_raytracer_stratixiv_arch_timing.blif/common num_LAB relative value 1.1453428863868986 outside of range [0.95,1.05] and not equal to golden value: 977.0

regression_tests/vtr_reg_nightly_test7/3d_sb_titan_other_per_layer_bb...[Fail]
[Fail]
3d_SB_inter_die_stratixiv_arch.timing.xml/murax_stratixiv_arch_timing.blif/common num_LAB relative value 1.0684931506849316 outside of range [0.95,1.05] and not equal to golden value: 73.0

Test 'vtr_reg_nightly_test7' had 17 qor test failures

Test 'vtr_reg_nightly_test7' had 0 run failures

Error: 18 tests failed

[AlexandreSinger]

My gut feeling is that we can just regenerate the golden results for these since they are so close to the golden solution for all of them and it could just be a small change causing these; however, all of these are from either the run-flat or the 3D testcases and they all show a regression of 5%-10%.

@vaughnbetz What do you think? Should we just regenerate the golden results?

[vaughnbetz]

These look like moderate changes to LAB count on a few mostly smaller designs. We changed the default clustering output limit a little while ago -- @MohamedElgammal maybe nightly test 7's golden results weren't updated?

I think it is Ok to regenerate/update these golden results.

[AlexandreSinger]

@AmirhosseinPoolad

https://docs.verilogtorouting.org/en/latest/README.developers/#generating-new-qor-golden-result

How to update golden results^^^