Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow mutation tests to expect one of many error codes. #1446

Merged
merged 1 commit into from
May 22, 2024

Conversation

locallycompact
Copy link
Contributor

@locallycompact locallycompact commented May 22, 2024

This accounts for situations where mutation tests could fail for multiple reasons depending on the evaluation order of the validator.


  • CHANGELOG updated or not needed
  • Documentation updated or not needed
  • Haddocks updated or not needed
  • No new TODOs introduced or explained herafter

@locallycompact locallycompact force-pushed the lc/multiple-errors-in-mutation-tests branch from bfbf8c7 to ff45f34 Compare May 22, 2024 09:11
Copy link

github-actions bot commented May 22, 2024

Transactions Costs

Sizes and execution budgets for Hydra protocol transactions. Note that unlisted parameters are currently using arbitrary values and results are not fully deterministic and comparable to previous runs.

Metadata
Generated at 2024-05-22 10:36:49.430517253 UTC
Max. memory units 14000000
Max. CPU units 10000000000
Max. tx size (kB) 16384

Script summary

Name Hash Size (Bytes)
νInitial bccf2a430c016bc960fbf31b02694011cd399d20da8882aac9d33611 4110
νCommit 56b0f0b597150e619c76bed60683f3b1e42d7bc0685ed951b882bfc5 1975
νHead 86bff95ba20e9d1d1b34899a56d86bbacc9fed999260b27dcc92d128 9351
μHead 88f533cf67cd0fc93d7d9ccf0a8b1d69ffd1208a825efbebbc1d36ba* 4213
  • The minting policy hash is only usable for comparison. As the script is parameterized, the actual script is unique per Head.

Cost of Init Transaction

Parties Tx size % max Mem % max CPU Min fee ₳
1 4795 8.97 3.44 0.46
2 4993 10.82 4.14 0.49
3 5197 12.64 4.83 0.52
5 5597 16.64 6.37 0.58
10 6605 26.31 10.09 0.73
48 14244 99.96 38.36 1.86

Cost of Commit Transaction

This is using ada-only outputs for better comparability.

UTxO Tx size % max Mem % max CPU Min fee ₳
1 559 10.24 4.04 0.29
2 748 13.88 5.64 0.34
3 937 17.66 7.29 0.39
5 1309 25.66 10.74 0.49
10 2247 48.19 20.30 0.78
19 3920 97.83 40.79 1.41

Cost of CollectCom Transaction

Parties UTxO (bytes) Tx size % max Mem % max CPU Min fee ₳
1 57 544 16.93 6.68 0.36
2 114 654 26.39 10.53 0.47
3 171 764 38.13 15.34 0.60
4 228 878 53.37 21.54 0.78
5 282 984 66.61 27.14 0.93
6 338 1099 84.66 34.58 1.13

Cost of Close Transaction

Parties Tx size % max Mem % max CPU Min fee ₳
1 583 8.34 4.61 0.28
2 744 8.96 5.65 0.30
3 881 9.70 6.73 0.32
5 1229 11.84 9.48 0.37
10 1840 15.12 14.58 0.46
50 4662 38.83 23.67 0.84

Cost of Contest Transaction

Parties Tx size % max Mem % max CPU Min fee ₳
1 624 8.65 4.73 0.28
2 759 9.38 5.80 0.30
3 969 10.56 7.25 0.33
5 1281 12.34 9.69 0.38
10 1936 16.29 15.30 0.48
50 8123 51.74 64.07 1.39

Cost of Abort Transaction

Some variation because of random mixture of still initial and already committed outputs.

Parties Tx size % max Mem % max CPU Min fee ₳
1 4584 13.97 5.78 0.51
2 4748 24.68 10.61 0.64
3 4858 37.69 16.33 0.79
4 5008 56.62 24.81 1.01
5 5343 79.05 34.91 1.28

Cost of FanOut Transaction

Involves spending head output and burning head tokens. Uses ada-only UTxO for better comparability.

Parties UTxO UTxO (bytes) Tx size % max Mem % max CPU Min fee ₳
5 0 0 4627 8.06 3.37 0.44
5 1 57 4661 9.21 4.08 0.46
5 5 284 4796 13.82 6.93 0.52
5 10 570 4968 19.16 10.31 0.60
5 20 1138 5305 31.11 17.62 0.76
5 30 1708 5647 42.43 24.67 0.92
5 40 2278 5988 53.96 31.81 1.08
5 50 2845 6325 65.50 38.95 1.24
5 80 4553 7342 99.94 60.31 1.72

End-To-End Benchmark Results

This page is intended to collect the latest end-to-end benchmarks results produced by Hydra's Continuous Integration system from the latest master code.

Please take those results with a grain of salt as they are currently produced from very limited cloud VMs and not controlled hardware. Instead of focusing on the absolute results, the emphasis should be on relative results, eg. how the timings for a scenario evolve as the code changes.

Generated at 2024-05-22 10:39:03.10638118 UTC

Baseline Scenario

Number of nodes 1
Number of txs 3000
Avg. Confirmation Time (ms) 4.186614488
P99 9.985628529999996ms
P95 5.565110799999998ms
P50 3.9108405ms
Number of Invalid txs 0

Three local nodes

Number of nodes 3
Number of txs 9000
Avg. Confirmation Time (ms) 22.188729506
P99 113.68374131000002ms
P95 31.65402519999999ms
P50 19.3607765ms
Number of Invalid txs 0

Copy link

Test Results

427 tests  ±0   419 ✅ ±0   14m 58s ⏱️ -29s
139 suites ±0     8 💤 ±0 
  2 files   ±0     0 ❌ ±0 

Results for commit ff45f34. ± Comparison against base commit b7fbf19.

@locallycompact locallycompact requested a review from a team May 22, 2024 09:22
This accounts for situations where mutation tests could fail for multiple reasons depending on the evaluation order of the validator.
@locallycompact locallycompact force-pushed the lc/multiple-errors-in-mutation-tests branch from ff45f34 to 90e18c4 Compare May 22, 2024 10:34
@locallycompact locallycompact merged commit d82a44f into master May 22, 2024
20 checks passed
@locallycompact locallycompact deleted the lc/multiple-errors-in-mutation-tests branch May 22, 2024 11:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants