Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add candidate_viability scenario #5935

Merged
merged 21 commits into from
Feb 19, 2024

Conversation

eddyashton
Copy link
Member

@eddyashton eddyashton commented Jan 18, 2024

This ends up being another flavour of suffix_collision, but naming it as candidate_viability to capture the original intent/investigation.

This acts as a regression scenario for the spec issue spotted in #5934. This is a scenario which we believe is valid, and can trace match, but for which MoreUpToDateCorrectInv is not an invariant.

This adds a few new driver actions:

  • dispatch_single,{src},{dst} lets us examine the state after single messages at the scenario level. Previously we could only flush at the node-to-node level message.
  • summarise_log(s_all) prints the log state. Primarily for debugging, similar to state.

@eddyashton eddyashton requested a review from a team as a code owner January 18, 2024 10:41
@ccf-bot
Copy link
Collaborator

ccf-bot commented Jan 18, 2024

candidate_viability_scenario@81980 aka 20240219.15 vs main ewma over 20 builds from 81610 to 81973

Click to see table

main

build_id build_number pi_basic_mt_sgx_cft^ pi_basic_mt_sgx_cft_mem Commit latency factor tpcc_sgx_cft^ tpcc_sgx_cft_mem pi_basic_mt_virtual_cft^ tpcc_virtual_cft^ ls_virtual_cft^ pi_ls_virtual_cft^ pi_basic_virtual_cft^ ls_sgx_cft^ ls_sgx_cft_mem pi_ls_sgx_cft^ pi_ls_sgx_cft_mem pi_basic_js_virtual_cft^ pi_basic_sgx_cft^ pi_basic_sgx_cft_mem ls_jwt_virtual_cft^ pi_ls_jwt_virtual_cft^ ls_js_virtual_cft^ ls_full_js_virtual_cft^ ls_js_jwt_virtual_cft^ pi_basic_js_sgx_cft^ pi_basic_js_sgx_cft_mem ls_jwt_sgx_cft^ ls_jwt_sgx_cft_mem pi_ls_jwt_sgx_cft^ pi_ls_jwt_sgx_cft_mem ls_js_sgx_cft^ ls_js_sgx_cft_mem hist_sgx_cft^ ls_full_js_sgx_cft^ ls_full_js_sgx_cft_mem ls_js_jwt_sgx_cft^ ls_js_jwt_sgx_cft_mem tlc_3node_fixed_duration_s tlc_3node_fixed_states tlc_atomic_reconfig_duration_s tlc_atomic_reconfig_states tlc_reconfig_duration_s tlc_reconfig_states RB put (/s)^ CHAMP put (/s)^ RB get (/s)^ CHAMP get (/s)^ tlc_sim_traces tlc_sim_levelmean
81610 20240209.43 28113.5 2.51822e+07 0.831113 5452.05 8.59996e+07 74800.8 17312.9 53334.8 56377.1 63345.5 13971.3 1.88908e+07 14055.1 1.05021e+07 4549.3 15316.7 1.46964e+07 20875.3 21727.8 20872.7 17287.8 11740.9 1411.3 1.25993e+07 6769.48 1.67936e+07 6829.1 6.30784e+06 5785.06 1.67936e+07 41491.4 5431.98 1.67936e+07 3983.75 1.67936e+07 5 58806 272 6.79836e+06 894 1.90055e+07 828263 1.1861e+06 8.14735e+06 3.07e+07 46780 365
81624 20240212.1 28160.1 2.30851e+07 0.840579 5618.33 8.59996e+07 76732.9 17250 52989.9 56371.4 63069.6 14131.6 1.88908e+07 14237.7 1.05021e+07 4614.8 15703.9 1.46964e+07 21193.5 21333.3 20726.4 17516.9 11696.1 1433.2 1.25993e+07 6854.74 1.67936e+07 6976.9 6.30784e+06 5807.12 1.67936e+07 47466.5 5733.35 1.88908e+07 4003.65 1.67936e+07 5 58806 278 6.79836e+06 900 1.90055e+07 817866 1.18164e+06 8.13644e+06 3.12543e+07 46044 359
81633 20240212.5 27902.2 2.51822e+07 0.858976 5644.36 8.59996e+07 94529.7 17328.9 52996.2 56459.2 63330.5 14075.5 1.88908e+07 14236.7 1.05021e+07 4619.8 15713.4 1.25993e+07 20718.5 21627.2 20938.2 17612.2 11855.3 1437.2 1.25993e+07 7254.33 1.67936e+07 6987.9 6.30784e+06 5760 1.67936e+07 42607.2 5489.04 1.88908e+07 3987.18 1.67936e+07 6 58806 280 6.79836e+06 892 1.90055e+07 829013 1.18545e+06 8.17033e+06 3.04839e+07 46925 365
81655 20240212.13 27898.6 2.30851e+07 0.803768 5563.8 8.59996e+07 76460 17316.4 52962.6 47729.3 62963 13990.4 1.88908e+07 14131.4 1.05021e+07 4609.6 15607.9 1.25993e+07 20826.8 22215.8 20982.2 17201.3 11702 1430.6 1.25993e+07 6798.34 1.67936e+07 7014.8 6.30784e+06 5801.86 1.67936e+07 45338.9 5707.17 1.67936e+07 3984.33 1.67936e+07 6 58806 282 6.79836e+06 857 1.90055e+07 837177 1.18565e+06 8.12705e+06 3.05914e+07 45375 356
81698 20240212.26 27512.9 2.51822e+07 0.801797 5544.31 8.59996e+07 81841.4 17343.5 53339.3 56997.9 63496.3 13988.3 1.88908e+07 14007.3 1.05021e+07 4628.1 15458.2 1.46964e+07 20830.9 22081.4 21241.5 17273.1 11733.4 1419.4 1.25993e+07 6793.95 1.67936e+07 6814.1 6.30784e+06 5799.08 1.67936e+07 47856 5451.72 1.67936e+07 3966.96 1.67936e+07 6 58806 263 6.79836e+06 854 1.90055e+07 838135 1.1831e+06 8.17395e+06 3.08327e+07 46408 359
81713 20240212.29 28092.6 2.30851e+07 0.813847 5580.79 8.59996e+07 88445.5 17102.9 53091.7 57051.2 60375.5 13971 1.88908e+07 14066.3 1.05021e+07 4517.3 15530.5 1.46964e+07 17241 21863.2 17389.8 17423 11630.8 1403.1 1.25993e+07 6787.68 1.88908e+07 6820.5 6.30784e+06 5749.76 1.67936e+07 38811.1 5445.28 1.67936e+07 3992.32 1.67936e+07 5 58806 274 6.79836e+06 876 1.90055e+07 838511 1.18495e+06 8.13641e+06 3.08983e+07 48318 358
81732 20240212.37 27888.8 2.51822e+07 0.845989 5518.64 8.59996e+07 70549.7 17229.3 53165.2 57484.1 62768.9 13910.9 1.88908e+07 14011.2 1.05021e+07 4595.7 15394.5 1.25993e+07 20604.4 22223.7 20891.8 17526.8 11852.5 1415.1 1.25993e+07 6801.08 1.88908e+07 6957.2 6.30784e+06 5787.7 1.67936e+07 45010.5 5473.48 1.67936e+07 3997.15 1.67936e+07 5 58806 278 6.79836e+06 866 1.90055e+07 838282 1.18275e+06 8.12582e+06 3.07715e+07 48346 365
81747 20240213.1 28031.1 2.30851e+07 0.783706 5598.88 8.59996e+07 84199.3 17205 52771.1 56937.6 62139.6 14025.9 1.88908e+07 14118.6 1.05021e+07 4593.7 15557.8 1.25993e+07 21109.9 22089.2 17259.7 17604.4 11729.5 1425.8 1.25993e+07 7227.43 1.67936e+07 7096.7 6.30784e+06 5807.81 1.67936e+07 42851.5 5488.4 1.67936e+07 4002.5 1.67936e+07 5 58806 278 6.79836e+06 884 1.90055e+07 841331 1.17805e+06 8.14091e+06 3.08462e+07 45775 360
81773 20240213.12 28346.5 2.30851e+07 0.806758 5612.8 8.59996e+07 88445.4 17755.6 53442 56959.3 63159.7 14079.7 1.88908e+07 14147.7 1.05021e+07 4585.3 15604.4 1.25993e+07 20190.2 22218.3 21017.4 17330.1 11817.1 1424.6 1.25993e+07 6832.37 1.88908e+07 6972.8 6.30784e+06 5778.15 1.67936e+07 45384.7 5492.64 1.67936e+07 4001.02 1.67936e+07 6 58806 323 6.79836e+06 995 1.90055e+07 832511 1.1804e+06 8.15066e+06 3.08694e+07 37551 353
81795 20240213.20 28128.7 2.51822e+07 0.771904 5553.43 8.59996e+07 99598 17327.1 55995.6 57880.1 63521.1 13925 1.88908e+07 14099.2 1.05021e+07 4658 15403.6 1.46964e+07 20659.2 21285.7 21165.5 17234.2 11886.9 1407.6 1.25993e+07 6762.8 1.67936e+07 6873.1 6.30784e+06 5799.14 1.67936e+07 40785.6 5467.56 1.67936e+07 3987.27 1.67936e+07 6 58806 333 6.79836e+06 1023 1.90055e+07 831204 1.18179e+06 8.1411e+06 3.21623e+07 36847 359
81806 20240213.24 27887.3 2.51822e+07 0.81354 5527.63 8.59996e+07 78720.8 17516.6 53363 48120.9 61690.2 13996.4 1.67936e+07 13946.8 1.05021e+07 4649.6 15450.2 1.46964e+07 20920.8 21147 17259.4 17639.9 11657.8 1421.3 1.25993e+07 6835.72 1.67936e+07 6884 6.30784e+06 5795.51 1.67936e+07 44086.7 5489 1.67936e+07 3994.6 1.67936e+07 6 58806 320 6.79836e+06 1048 1.90055e+07 826669 1.18252e+06 8.12582e+06 3.2006e+07 37461 349
81814 20240214.3 27856.8 2.30851e+07 0.805022 5610.84 8.59996e+07 72270.6 17388.6 52805.8 57042.8 61624.1 14037.2 1.88908e+07 14172.8 1.05021e+07 4608.2 15580.1 1.25993e+07 21043.9 22035.7 20851.1 17513.5 11759.5 1430 1.25993e+07 7237.01 1.67936e+07 6933.7 6.30784e+06 5803.73 1.67936e+07 45304.2 5745.23 1.67936e+07 3995.11 1.67936e+07 6 58806 318 6.79836e+06 1048 1.90055e+07 825397 1.1803e+06 8.13451e+06 3.12186e+07 37073 358
81835 20240215.1 28260.7 2.51822e+07 0.810838 5556.59 8.59996e+07 97762.8 17546.5 53371.7 57971.7 62812.8 14014.7 1.88908e+07 14159.4 1.05021e+07 4669.2 15548.8 1.25993e+07 21349.3 21626.3 21004.2 17406.6 11723.3 1433.9 1.05021e+07 6825.13 1.67936e+07 6961.7 6.30784e+06 5808.4 1.67936e+07 46516.7 5482.12 1.88908e+07 3991.88 1.67936e+07 6 58806 309 6.79836e+06 1045 1.90055e+07 826714 1.18256e+06 8.10367e+06 3.07965e+07 38798 357
81866 20240215.12 27973.7 2.51822e+07 0.798433 5633.12 8.59996e+07 94340.9 17307.5 55718 57706.4 62491.5 13963.8 1.88908e+07 14054.9 1.05021e+07 4631.6 15537.9 1.25993e+07 21333.3 22140.5 17722.6 17434.4 11614.9 1427.4 1.25993e+07 6822.14 1.67936e+07 6975.1 6.30784e+06 5800.23 1.67936e+07 45649.5 5477.44 1.67936e+07 3995.34 1.67936e+07 6 57915 300 6.61457e+06 1004 1.86602e+07 833692 1.17884e+06 8.14943e+06 3.10162e+07 36692 369
81878 20240216.2 28163.4 2.51822e+07 0.804098 5614.03 8.59996e+07 70152.6 17209.5 55821.2 57015.5 62429.9 14037.1 1.88908e+07 14165.9 1.05021e+07 4640.4 15456.9 1.25993e+07 21240.7 22158.7 17694.1 17450.6 11629 1437.9 1.25993e+07 6855.03 1.88908e+07 7071.3 6.30784e+06 5766.22 1.67936e+07 46518.3 5726.45 1.67936e+07 4000.06 1.67936e+07 6 57915 306 6.61457e+06 1019 1.86602e+07 834938 1.17903e+06 8.12502e+06 3.05102e+07 37141 354
81902 20240216.12 28187.3 2.30851e+07 0.83745 5662.17 8.59996e+07 74042.2 17213.4 53156.5 57689.4 62247.5 14100 1.67936e+07 14186.8 1.05021e+07 4598.3 15626.4 1.25993e+07 20992.9 22396.4 17768.7 17443.1 11714.3 1434.3 1.25993e+07 7270.68 1.67936e+07 7081.9 6.30784e+06 5779.11 1.67936e+07 47922.8 5776.12 1.67936e+07 3994.94 1.67936e+07 6 57915 309 6.61457e+06 999 1.86602e+07 835906 1.18276e+06 8.15144e+06 3.14202e+07 34208 353
81943 20240219.1 27621.8 2.51822e+07 0.782095 5667.55 8.59996e+07 83079 17277.9 56116.8 57788.8 62804 14044.5 1.88908e+07 14183.2 1.05021e+07 4644.4 15445.6 1.46964e+07 20651.1 21669.4 17591.4 17552.7 11815.2 1429.8 1.25993e+07 6908.51 1.88908e+07 7031.7 6.30784e+06 5804.6 1.67936e+07 45735.4 5488.26 1.67936e+07 3992.56 1.67936e+07 6 57915 300 6.61457e+06 995 1.86602e+07 835133 1.18877e+06 8.12592e+06 3.07914e+07 37416 364
81950 20240219.4 28273.1 2.51822e+07 0.788681 5634.8 8.59996e+07 77771.5 17256.4 56177.9 57713.7 63579.7 14033 1.88908e+07 14105 1.05021e+07 4652.1 15646.6 1.25993e+07 20835.7 21736.8 17628.5 17569.5 11699.6 1437.4 1.25993e+07 7249.6 1.67936e+07 6937.1 6.30784e+06 5800.48 1.67936e+07 47813 5775.71 1.67936e+07 4004.55 1.67936e+07 6 57915 297 6.61457e+06 1001 1.86602e+07 816727 1.18306e+06 8.17317e+06 3.13601e+07 35189 364
81956 20240219.6 28198.5 2.51822e+07 0.841847 5590.12 8.59996e+07 82111.5 17256.7 56076.3 57868 63536.2 14039 1.67936e+07 14103.3 1.05021e+07 4621.3 15575 1.46964e+07 20475.6 21545.2 20790.2 17549.8 11816.2 1422.6 1.25993e+07 6839.08 1.88908e+07 7081.8 6.30784e+06 5802.2 1.67936e+07 42003.1 5733.21 1.67936e+07 3958.81 1.67936e+07 6 57915 320 6.61457e+06 977 1.86602e+07 836148 1.18263e+06 8.1505e+06 3.065e+07 37053 366
81973 20240219.13 28012.5 2.30851e+07 0.792366 5654.35 8.59996e+07 99659.4 17348.1 53111.9 57636.9 62666.2 14061.5 1.88908e+07 14134.1 1.05021e+07 4614.4 15558.3 1.46964e+07 20487.2 21719.8 20838.9 17377.1 11885.9 1427.3 1.25993e+07 6817.71 1.67936e+07 6933.9 6.30784e+06 5768.53 1.67936e+07 44508.3 5478.15 1.88908e+07 3965.28 1.67936e+07 6 57915 300 6.61457e+06 993 1.86602e+07 833146 1.18178e+06 8.15641e+06 3.08151e+07 36649 359

candidate_viability_scenario

build_id build_number pi_basic_mt_sgx_cft^ pi_basic_mt_sgx_cft_mem Commit latency factor tpcc_sgx_cft^ tpcc_sgx_cft_mem ls_sgx_cft^ ls_sgx_cft_mem pi_ls_sgx_cft^ pi_ls_sgx_cft_mem pi_basic_sgx_cft^ pi_basic_sgx_cft_mem pi_basic_js_sgx_cft^ pi_basic_js_sgx_cft_mem ls_jwt_sgx_cft^ ls_jwt_sgx_cft_mem pi_ls_jwt_sgx_cft^ pi_ls_jwt_sgx_cft_mem ls_js_sgx_cft^ ls_js_sgx_cft_mem ls_full_js_sgx_cft^ ls_full_js_sgx_cft_mem ls_js_jwt_sgx_cft^ ls_js_jwt_sgx_cft_mem hist_sgx_cft^ RB put (/s)^ CHAMP put (/s)^ RB get (/s)^ CHAMP get (/s)^ pi_basic_mt_virtual_cft^ tpcc_virtual_cft^ tlc_sim_traces tlc_sim_levelmean ls_virtual_cft^ pi_ls_virtual_cft^ pi_basic_virtual_cft^ pi_basic_js_virtual_cft^ ls_jwt_virtual_cft^ pi_ls_jwt_virtual_cft^ ls_js_virtual_cft^ ls_full_js_virtual_cft^ ls_js_jwt_virtual_cft^ tlc_3node_fixed_duration_s tlc_3node_fixed_states tlc_atomic_reconfig_duration_s tlc_atomic_reconfig_states tlc_reconfig_duration_s tlc_reconfig_states
80571 20240119.14 28042.4 2.30851e+07 0.828912 5556.2 8.59996e+07 14073.1 1.88908e+07 14086.7 1.05021e+07 15506 1.25993e+07 1422.4 1.25993e+07 7237.1 1.67936e+07 7014.6 6.30784e+06 5774.74 1.67936e+07 5455.73 1.67936e+07 4003.53 1.67936e+07 45898.7 826073 1.17924e+06 8.15381e+06 3.09174e+07 81230.4 16823 48171 395 53007.2 57557.3 60745.2 4580.8 21454.8 21628.2 20850.9 17407.5 11818.2 6 77678 907 2.53976e+07 673 1.594e+07
80674 20240122.17 28119.9 2.30851e+07 0.809552 5601.75 8.59996e+07 14102 1.88908e+07 14178.1 1.05021e+07 15612.8 1.46964e+07 1438.2 1.25993e+07 7255.39 1.67936e+07 6996.5 6.30784e+06 5821.13 1.67936e+07 5712.71 1.88908e+07 4001.31 1.67936e+07 45741.5 822758 1.18154e+06 8.14593e+06 3.21901e+07 67994.9 17293.2 51403 400 55829.4 57979.1 62724.8 4636.7 20985.3 21856.5 21026.3 17689.4 11726.2 6 77678 872 2.53976e+07 627 1.594e+07
81346 20240206.14 28079.2 2.30851e+07 0.784107 5620.47 8.59996e+07 14024.5 1.88908e+07 14197.8 1.05021e+07 15577.5 1.46964e+07 1435.7 1.25993e+07 6885.69 1.88908e+07 6999.7 6.30784e+06 5795.88 1.67936e+07 5486.71 1.67936e+07 4001.14 1.67936e+07 45413.6 838141 1.17423e+06 8.14253e+06 3.08629e+07 85039.7 17329.7 48789 353 53278.1 56484.7 63131.6 4652.2 20798.9 20849 17710.6 18020.6 11724.9 5 58806 171 4.51944e+06 568 1.35909e+07
81980 20240219.15 28057.3 2.51822e+07 0.835131 5602.3 8.59996e+07 14093.3 1.88908e+07 14154.2 1.05021e+07 15631.9 1.46964e+07 1425 1.25993e+07 7250.13 1.67936e+07 6975.1 6.30784e+06 5813.52 1.67936e+07 5491.26 1.67936e+07 4000.68 1.67936e+07 43398.1 837144 1.18427e+06 8.07886e+06 3.08132e+07 68817.6 17304.8 36982 362 56095.3 57022.6 63573.3 4597 20886.8 22276.2 17149.2 17592.5 11841.5 6 57915 299 6.61457e+06 1016 1.86602e+07

images

@eddyashton eddyashton marked this pull request as draft January 22, 2024 16:08
@achamayou achamayou mentioned this pull request Jan 30, 2024
7 tasks
@eddyashton eddyashton marked this pull request as ready for review February 19, 2024 14:37
@eddyashton eddyashton enabled auto-merge (squash) February 19, 2024 14:38
@eddyashton eddyashton merged commit 41c8cb9 into microsoft:main Feb 19, 2024
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants