Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
2030: Add benchmark for `RndState` and add non-null tracers in benchmark r=KtorZ a=KtorZ # Issue Number <!-- Put here a reference to the issue this PR relates to and which requirements it tackles --> #1997 # Overview <!-- Detail in a few bullet points the work accomplished in this PR --> - 36bfac4 📍 **extend benchmark with RndState and byron address storage** I also did some re-structuring of the benchmark fixtures, so that the creation of benchmark data happens outside of the actual benchmarks. Also, I reviewed some of the work bench to use numbers that reflects a bit more reality (we actually always store checkpoints one by one, so the time it takes to store N checkpoints isn't quite interesting). <details> <summary>see full results</summary> - NVMe SSD Controller SM981/PM981 - Intel(R) Core(TM) i7-8565U CPU @ 1.80GHz - 256KiB L1 cache - 1MiB L2 cache - 8MiB L3 cache - 32GiB System Memory - 16GiB SODIMM DDR4 Synchronous 2400 MHz (0,4 ns) - 16GiB SODIMM DDR4 Synchronous 2400 MHz (0,4 ns) ``` benchmarking UTxO (Write)/1 CP x 0 UTxO time 3.182 ms (2.943 ms .. 3.493 ms) 0.943 R² (0.902 R² .. 0.975 R²) mean 3.228 ms (3.071 ms .. 3.439 ms) std dev 433.0 μs (329.2 μs .. 563.3 μs) variance introduced by outliers: 69% (severely inflated) benchmarking UTxO (Write)/1 CP x 10 UTxO time 3.583 ms (3.407 ms .. 3.821 ms) 0.970 R² (0.943 R² .. 0.988 R²) mean 3.602 ms (3.446 ms .. 3.839 ms) std dev 463.5 μs (307.3 μs .. 695.7 μs) variance introduced by outliers: 65% (severely inflated) benchmarking UTxO (Write)/1 CP x 100 UTxO time 6.134 ms (5.834 ms .. 6.451 ms) 0.986 R² (0.970 R² .. 0.994 R²) mean 6.002 ms (5.849 ms .. 6.173 ms) std dev 392.8 μs (310.1 μs .. 569.3 μs) variance introduced by outliers: 29% (moderately inflated) benchmarking UTxO (Write)/1 CP x 1000 UTxO time 29.62 ms (28.15 ms .. 31.61 ms) 0.990 R² (0.976 R² .. 0.998 R²) mean 29.83 ms (29.04 ms .. 30.76 ms) std dev 1.768 ms (1.312 ms .. 2.395 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking UTxO (Write)/1 CP x 10000 UTxO time 271.5 ms (258.9 ms .. 285.0 ms) 0.998 R² (0.996 R² .. 1.000 R²) mean 268.5 ms (258.0 ms .. 274.1 ms) std dev 9.334 ms (3.659 ms .. 12.75 ms) variance introduced by outliers: 16% (moderately inflated) benchmarking UTxO (Write)/1 CP x 100000 UTxO time 2.844 s (2.552 s .. 3.177 s) 0.998 R² (0.994 R² .. 1.000 R²) mean 2.768 s (2.691 s .. 2.811 s) std dev 73.91 ms (17.62 ms .. 98.78 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking UTxO (Write)/10 CP x 1000 UTxO time 316.2 ms (222.6 ms .. 417.3 ms) 0.981 R² (0.979 R² .. 1.000 R²) mean 316.9 ms (292.7 ms .. 341.0 ms) std dev 30.48 ms (14.55 ms .. 38.38 ms) variance introduced by outliers: 22% (moderately inflated) benchmarking UTxO (Write)/100 CP x 1000 UTxO time 2.565 s (2.239 s .. 2.766 s) 0.998 R² (0.996 R² .. 1.000 R²) mean 3.037 s (2.804 s .. 3.226 s) std dev 261.6 ms (100.5 ms .. 357.2 ms) variance introduced by outliers: 22% (moderately inflated) benchmarking UTxO (Read)/1 CP x 0 UTxO time 5.069 ms (4.804 ms .. 5.363 ms) 0.967 R² (0.923 R² .. 0.993 R²) mean 4.829 ms (4.619 ms .. 5.102 ms) std dev 533.6 μs (333.9 μs .. 780.3 μs) variance introduced by outliers: 54% (severely inflated) benchmarking UTxO (Read)/1 CP x 10 UTxO time 4.009 ms (3.720 ms .. 4.293 ms) 0.969 R² (0.931 R² .. 0.991 R²) mean 4.577 ms (4.387 ms .. 4.897 ms) std dev 552.9 μs (331.6 μs .. 889.7 μs) variance introduced by outliers: 58% (severely inflated) benchmarking UTxO (Read)/1 CP x 100 UTxO time 5.092 ms (4.932 ms .. 5.252 ms) 0.995 R² (0.991 R² .. 0.998 R²) mean 5.137 ms (5.054 ms .. 5.201 ms) std dev 158.1 μs (120.2 μs .. 222.4 μs) benchmarking UTxO (Read)/1 CP x 1000 UTxO time 19.79 ms (18.20 ms .. 22.09 ms) 0.953 R² (0.878 R² .. 0.990 R²) mean 19.86 ms (18.82 ms .. 21.49 ms) std dev 2.510 ms (1.607 ms .. 4.136 ms) variance introduced by outliers: 48% (moderately inflated) benchmarking UTxO (Read)/1 CP x 10000 UTxO time 161.8 ms (105.9 ms .. 180.0 ms) 0.986 R² (0.937 R² .. 1.000 R²) mean 151.8 ms (138.0 ms .. 159.9 ms) std dev 13.51 ms (5.826 ms .. 19.63 ms) variance introduced by outliers: 18% (moderately inflated) benchmarking UTxO (Read)/1 CP x 100000 UTxO time 1.428 s (1.369 s .. 1.555 s) 0.999 R² (0.998 R² .. 1.000 R²) mean 1.381 s (1.368 s .. 1.406 s) std dev 23.90 ms (130.2 μs .. 28.16 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking SeqState/1 CP x 10 addr time 3.379 ms (3.170 ms .. 3.572 ms) 0.976 R² (0.954 R² .. 0.991 R²) mean 3.311 ms (3.195 ms .. 3.430 ms) std dev 284.2 μs (212.5 μs .. 396.9 μs) variance introduced by outliers: 47% (moderately inflated) benchmarking SeqState/1 CP x 100 addr time 5.443 ms (5.138 ms .. 5.713 ms) 0.979 R² (0.956 R² .. 0.992 R²) mean 5.466 ms (5.308 ms .. 5.672 ms) std dev 427.8 μs (291.6 μs .. 592.0 μs) variance introduced by outliers: 39% (moderately inflated) benchmarking SeqState/1 CP x 1000 addr time 26.29 ms (25.03 ms .. 27.87 ms) 0.993 R² (0.986 R² .. 0.999 R²) mean 24.41 ms (23.76 ms .. 25.15 ms) std dev 1.436 ms (1.024 ms .. 2.145 ms) variance introduced by outliers: 18% (moderately inflated) benchmarking SeqState/1 CP x 10000 addr time 217.6 ms (143.9 ms .. 283.6 ms) 0.965 R² (0.948 R² .. 1.000 R²) mean 249.6 ms (231.4 ms .. 266.7 ms) std dev 22.71 ms (13.88 ms .. 31.51 ms) variance introduced by outliers: 18% (moderately inflated) benchmarking SeqState/1 CP x 100000 addr time 2.629 s (2.361 s .. 2.815 s) 0.999 R² (0.996 R² .. 1.000 R²) mean 2.487 s (2.314 s .. 2.552 s) std dev 120.2 ms (4.664 ms .. 149.2 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking SeqState/10 CP x 1000 addr time 49.44 ms (44.08 ms .. 54.40 ms) 0.975 R² (0.946 R² .. 0.997 R²) mean 51.44 ms (49.24 ms .. 54.66 ms) std dev 4.991 ms (2.424 ms .. 6.888 ms) variance introduced by outliers: 32% (moderately inflated) benchmarking SeqState/100 CP x 1000 addr time 295.9 ms (261.5 ms .. 343.9 ms) 0.992 R² (0.981 R² .. 1.000 R²) mean 287.5 ms (280.0 ms .. 295.1 ms) std dev 10.26 ms (5.842 ms .. 14.28 ms) variance introduced by outliers: 16% (moderately inflated) benchmarking RndState/1 CP x 10 addr time 2.544 ms (2.375 ms .. 2.710 ms) 0.963 R² (0.929 R² .. 0.983 R²) mean 2.614 ms (2.515 ms .. 2.778 ms) std dev 309.7 μs (192.6 μs .. 439.4 μs) variance introduced by outliers: 66% (severely inflated) benchmarking RndState/1 CP x 100 addr time 3.527 ms (3.271 ms .. 3.754 ms) 0.963 R² (0.931 R² .. 0.987 R²) mean 3.505 ms (3.340 ms .. 3.797 ms) std dev 548.9 μs (308.4 μs .. 1.013 ms) variance introduced by outliers: 77% (severely inflated) benchmarking RndState/1 CP x 1000 addr time 8.955 ms (8.512 ms .. 9.397 ms) 0.987 R² (0.977 R² .. 0.994 R²) mean 8.904 ms (8.689 ms .. 9.127 ms) std dev 502.1 μs (427.8 μs .. 606.0 μs) variance introduced by outliers: 25% (moderately inflated) benchmarking RndState/1 CP x 10000 addr time 59.16 ms (54.70 ms .. 61.69 ms) 0.988 R² (0.963 R² .. 0.997 R²) mean 60.60 ms (58.74 ms .. 63.37 ms) std dev 3.753 ms (2.618 ms .. 4.804 ms) variance introduced by outliers: 18% (moderately inflated) benchmarking RndState/1 CP x 100000 addr time 570.5 ms (549.2 ms .. 602.6 ms) 1.000 R² (0.999 R² .. 1.000 R²) mean 585.9 ms (576.4 ms .. 598.2 ms) std dev 13.13 ms (3.653 ms .. 17.76 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking RndState/10 CP x 1000 addr time 561.2 ms (494.1 ms .. 708.5 ms) 0.992 R² (0.984 R² .. 1.000 R²) mean 513.5 ms (500.1 ms .. 539.3 ms) std dev 25.48 ms (685.7 μs .. 30.17 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking RndState/100 CP x 1000 addr time 52.21 s (48.98 s .. 54.44 s) 1.000 R² (0.999 R² .. 1.000 R²) mean 51.66 s (51.02 s .. 52.21 s) std dev 797.5 ms (352.6 ms .. 1.031 s) variance introduced by outliers: 19% (moderately inflated) benchmarking TxHistory (Write)/1 w/ 1i + 1o [1..10] time 2.018 ms (1.821 ms .. 2.215 ms) 0.926 R² (0.865 R² .. 0.967 R²) mean 2.526 ms (2.375 ms .. 2.704 ms) std dev 415.1 μs (339.3 μs .. 510.2 μs) variance introduced by outliers: 77% (severely inflated) benchmarking TxHistory (Write)/10 w/ 1i + 1o [1..10] time 2.959 ms (2.738 ms .. 3.186 ms) 0.961 R² (0.932 R² .. 0.982 R²) mean 3.299 ms (3.147 ms .. 3.471 ms) std dev 400.0 μs (301.7 μs .. 564.0 μs) variance introduced by outliers: 63% (severely inflated) benchmarking TxHistory (Write)/10 w/ 10i + 10o [1..10] time 7.899 ms (7.527 ms .. 8.293 ms) 0.989 R² (0.981 R² .. 0.995 R²) mean 7.345 ms (7.138 ms .. 7.553 ms) std dev 511.7 μs (425.2 μs .. 624.8 μs) variance introduced by outliers: 31% (moderately inflated) benchmarking TxHistory (Write)/10 w/ 50i + 100o [1..10] time 53.62 ms (47.93 ms .. 57.53 ms) 0.978 R² (0.931 R² .. 0.997 R²) mean 54.73 ms (51.41 ms .. 59.62 ms) std dev 7.023 ms (3.806 ms .. 10.75 ms) variance introduced by outliers: 44% (moderately inflated) benchmarking TxHistory (Write)/10 w/ 255i + 255o [1..100] time 166.2 ms (145.8 ms .. 180.3 ms) 0.992 R² (0.983 R² .. 1.000 R²) mean 151.8 ms (145.0 ms .. 157.7 ms) std dev 9.344 ms (6.262 ms .. 13.87 ms) variance introduced by outliers: 13% (moderately inflated) benchmarking TxHistory (Write)/100 w/ 10i + 10o [1..100] time 65.11 ms (56.94 ms .. 76.31 ms) 0.969 R² (0.915 R² .. 0.998 R²) mean 65.39 ms (62.09 ms .. 69.57 ms) std dev 6.323 ms (4.347 ms .. 8.712 ms) variance introduced by outliers: 28% (moderately inflated) benchmarking TxHistory (Write)/100 w/ 50i + 100o [1..100] time 546.7 ms (387.3 ms .. 704.4 ms) 0.989 R² (0.960 R² .. 1.000 R²) mean 526.1 ms (507.0 ms .. 558.2 ms) std dev 30.39 ms (7.684 ms .. 38.90 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking TxHistory (Write)/100 w/ 255i + 255o [1..100] time 1.731 s (1.481 s .. 1.897 s) 0.998 R² (0.992 R² .. 1.000 R²) mean 1.621 s (1.522 s .. 1.677 s) std dev 96.13 ms (29.17 ms .. 130.9 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking TxHistory (Write)/1000 w/ 10i + 10o [1..1000] time 554.0 ms (512.9 ms .. 583.7 ms) 0.999 R² (0.998 R² .. 1.000 R²) mean 559.0 ms (552.8 ms .. 566.8 ms) std dev 8.040 ms (3.100 ms .. 11.02 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking TxHistory (Write)/1000 w/ 50i + 100o [1..1000] time 4.592 s (4.014 s .. 4.937 s) 0.998 R² (0.995 R² .. 1.000 R²) mean 4.842 s (4.663 s .. 5.131 s) std dev 283.6 ms (3.899 ms .. 360.3 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking TxHistory (Write)/1000 w/ 255i + 255o [1..1000] time 14.42 s (13.98 s .. 15.46 s) 0.999 R² (0.999 R² .. 1.000 R²) mean 14.10 s (14.00 s .. 14.28 s) std dev 177.2 ms (5.722 ms .. 212.1 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking TxHistory (Write)/10000 w/ 10i + 10o [1..10000] time 5.793 s (5.746 s .. 5.837 s) 1.000 R² (1.000 R² .. 1.000 R²) mean 5.814 s (5.800 s .. 5.838 s) std dev 22.72 ms (3.755 ms .. 29.17 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking TxHistory (Read)/1000 [1..100] DESC - * time 3.111 s (2.894 s .. 3.342 s) 0.999 R² (NaN R² .. 1.000 R²) mean 3.351 s (3.230 s .. 3.447 s) std dev 121.3 ms (59.69 ms .. 160.2 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking TxHistory (Read)/1000 [1..100] ASC - * time 3.182 s (2.844 s .. 3.351 s) 0.999 R² (0.997 R² .. 1.000 R²) mean 3.371 s (3.275 s .. 3.427 s) std dev 93.88 ms (27.85 ms .. 127.6 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking TxHistory (Read)/1000 [1..1000] DESC - * time 3.518 s (3.237 s .. 3.736 s) 0.999 R² (0.999 R² .. 1.000 R²) mean 3.458 s (3.389 s .. 3.493 s) std dev 66.33 ms (32.05 ms .. 80.19 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking TxHistory (Read)/1000 [1..100] DESC pending * time 955.7 ms (712.4 ms .. 1.289 s) 0.982 R² (0.976 R² .. 1.000 R²) mean 829.7 ms (779.1 ms .. 897.1 ms) std dev 68.89 ms (23.29 ms .. 93.21 ms) variance introduced by outliers: 21% (moderately inflated) benchmarking TxHistory (Read)/1000 [1..100] DESC - 40..60 time 135.4 ms (94.69 ms .. 166.4 ms) 0.986 R² (0.980 R² .. 1.000 R²) mean 154.6 ms (140.7 ms .. 160.1 ms) std dev 9.667 ms (1.847 ms .. 12.70 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking TxHistory (Read)/1000 [1..10000] DESC - 42..1337 time 3.182 s (2.610 s .. 4.009 s) 0.993 R² (0.978 R² .. 1.000 R²) mean 3.286 s (3.125 s .. 3.388 s) std dev 167.7 ms (87.87 ms .. 236.0 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking TxHistory (Read)/10000 [1..100] DESC - 40..60 time 21.48 s (17.82 s .. 23.75 s) 0.997 R² (0.990 R² .. 1.000 R²) mean 19.03 s (17.94 s .. 20.13 s) std dev 1.290 s (1.083 s .. 1.455 s) variance introduced by outliers: 20% (moderately inflated) benchmarking TxHistory (Read)/10000 [1..10000] DESC - 42..1337 time 6.274 s (6.142 s .. 6.343 s) 1.000 R² (1.000 R² .. 1.000 R²) mean 6.246 s (6.205 s .. 6.262 s) std dev 29.77 ms (459.4 μs .. 36.66 ms) variance introduced by outliers: 19% (moderately inflated) -- Database disk space usage tests for UTxO File size /100 CP x 0 UTxO bench22158-7.db 4 KB bench22158-7.db-shm 32 KB bench22158-7.db-wal 249 KB bench22158-7.db (closed) 192 KB File size /1000 CP x 0 UTxO bench22158-8.db 4 KB bench22158-8.db-shm 32 KB bench22158-8.db-wal 555 KB bench22158-8.db (closed) 496 KB File size /10 CP x 10 UTxO bench22158-9.db 4 KB bench22158-9.db-shm 32 KB bench22158-9.db-wal 257 KB bench22158-9.db (closed) 192 KB File size /100 CP x 10 UTxO bench22158-10.db 4 KB bench22158-10.db-shm 32 KB bench22158-10.db-wal 639 KB bench22158-10.db (closed) 572 KB File size /1000 CP x 10 UTxO bench22158-11.db 4 MB bench22158-11.db-shm 32 KB bench22158-11.db-wal 4 MB bench22158-11.db (closed) 4 MB File size /10 CP x 100 UTxO bench22158-12.db 4 KB bench22158-12.db-shm 32 KB bench22158-12.db-wal 599 KB bench22158-12.db (closed) 532 KB File size /100 CP x 100 UTxO bench22158-13.db 3 MB bench22158-13.db-shm 32 KB bench22158-13.db-wal 4 MB bench22158-13.db (closed) 3 MB File size /1000 CP x 100 UTxO bench22158-14.db 37 MB bench22158-14.db-shm 96 KB bench22158-14.db-wal 38 MB bench22158-14.db (closed) 37 MB File size /10 CP x 1000 UTxO bench22158-15.db 3 MB bench22158-15.db-shm 32 KB bench22158-15.db-wal 3 MB bench22158-15.db (closed) 3 MB File size /100 CP x 1000 UTxO bench22158-16.db 37 MB bench22158-16.db-shm 96 KB bench22158-16.db-wal 37 MB bench22158-16.db (closed) 37 MB File size /1000 CP x 1000 UTxO bench22158-17.db 378 MB bench22158-17.db-shm 768 KB bench22158-17.db-wal 380 MB bench22158-17.db (closed) 378 MB Database disk space usage tests for TxHistory File size /100 w/ 10i + 20o bench22158-18.db 4 KB bench22158-18.db-shm 32 KB bench22158-18.db-wal 1 MB bench22158-18.db (closed) 1 MB File size /1000 w/ 10i + 20o bench22158-19.db 9 MB bench22158-19.db-shm 32 KB bench22158-19.db-wal 9 MB bench22158-19.db (closed) 9 MB File size /10000 w/ 10i + 20o bench22158-20.db 92 MB bench22158-20.db-shm 192 KB bench22158-20.db-wal 92 MB bench22158-20.db (closed) 92 MB File size /100000 w/ 10i + 20o bench22158-21.db 920 MB bench22158-21.db-shm 1 MB bench22158-21.db-wal 926 MB bench22158-21.db (closed) 920 MB File size /100 w/ 50i + 100o bench22158-22.db 4 MB bench22158-22.db-shm 32 KB bench22158-22.db-wal 4 MB bench22158-22.db (closed) 4 MB File size /1000 w/ 50i + 100o bench22158-23.db 47 MB bench22158-23.db-shm 96 KB bench22158-23.db-wal 47 MB bench22158-23.db (closed) 47 MB File size /10000 w/ 50i + 100o bench22158-24.db 471 MB bench22158-24.db-shm 960 KB bench22158-24.db-wal 474 MB bench22158-24.db (closed) 471 MB File size /100000 w/ 50i + 100o bench22158-25.db 4 GB bench22158-25.db-shm 9 MB bench22158-25.db-wal 4 GB bench22158-25.db (closed) 4 GB Benchmark db: FINISH Success! Waiting for next file change. ``` </details> - 3b6234f 📍 **generate Byron addresses for benchmark on RndState (instead of Jörmungandr delegation addresses...** - 33bcee0 📍 **enable logging in benchmarks, with minimum severity at Error.** Benchmark should somewhat reflect the production setup, which has a non-null logger enabled. In principle, logging should not impact the performances of the application, but this commit shows that it currently does, and a lot! - dff0168 📍 **benchmarks: filter logs before creating a LogObject** This has a dramatic impact on performances. Here are some comparisons of the database benchmarks with and without this fix, as well a comparison with the original benchmark behavior when using a `nullTracer`. benchmark | before | after | nullTracer --- | --- | --- | --- UTxO (Write)/1 CP x 100000 UTxO | 3.839s | 2.238s | 2.225s UTxO (Read)/1 CP x 100000 UTxO | 1.074s | 0.998s | 1.072s SeqState/1 CP x 100000 addr | 4.281s | 1.887s | 2.020s RndState/1 CP x 100000 addr x 100000 | 3.996s | 0.886s | 0.849s TxHistory (Write)/10000 w/ 10i + 10o | 8.032s | 5.793s | 5.587s TxHistory (Read)/10000 | 17.08s | 16.11s | 16.47s # Comments <!-- Additional comments or screenshots to attach if any --> There's an upcoming commit that fixes the severity filtering as per #2028 , I am however re-running the benchmark without the fix first to have a clear comparison of results before and after the fix. <!-- Don't forget to: ✓ Self-review your changes to make sure nothing unexpected slipped through ✓ Assign yourself to the PR ✓ Assign one or several reviewer(s) ✓ Once created, link this PR to its corresponding ticket ✓ Assign the PR to a corresponding milestone ✓ Acknowledge any changes required to the Wiki --> Co-authored-by: KtorZ <matthias.benkort@gmail.com> Co-authored-by: IOHK <devops+stack-project@iohk.io>
- Loading branch information
