Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add small 'realistic' workload benchmark integration test #714

Merged
merged 3 commits into from
Jun 13, 2024

Conversation

mamcx
Copy link
Contributor

@mamcx mamcx commented Jan 10, 2024

Description of Changes

A small integration test that benchmarks based on "realistic" workloads we are focusing on improving.

Expected complexity level and risk

1

@mamcx mamcx self-assigned this Jan 10, 2024
@joshua-spacetime
Copy link
Collaborator

Is there a reason you put these in their own module rather than in modules/benchmarks? Also can you add one for updates?

@cloutiertyler
Copy link
Contributor

Is there a reason you put these in their own module rather than in modules/benchmarks? Also can you add one for updates?

@mamcx I agree with Joshua here. This should not require a new module I don't think.

@mamcx
Copy link
Contributor Author

mamcx commented Jan 11, 2024

Is there a reason you put these in their own module rather than in modules/benchmarks? Also can you add one for updates?

The original is tied into the criterion benchmarks in crates/bench.

@mamcx mamcx requested a review from kazimuth January 11, 2024 18:39
@mamcx mamcx marked this pull request as ready for review January 12, 2024 18:36
@kazimuth
Copy link
Contributor

benchmarks please :)

Copy link

github-actions bot commented Jan 12, 2024

Benchmark results

Benchmark Report

Legend:

  • load: number of rows pre-loaded into the database
  • count: number of rows touched by the transaction
  • index types:
    • unique: a single index on the id column
    • non_unique: no indexes
    • multi_index: non-unique index on every column
  • schemas:
    • person(id: u32, name: String, age: u64)
    • location(id: u32, x: u64, y: u64)

All throughputs are single-threaded.

Empty transaction

db on disk new latency old latency new throughput old throughput
sqlite 💿 - 437.5±2.17ns - -
sqlite 🧠 - 431.8±1.41ns - -
stdb_module 💿 17.7±0.94µs 18.3±1.38µs - -
stdb_module 🧠 17.7±0.87µs 17.4±0.79µs - -
stdb_raw 💿 935.0±2.53ns 933.3±1.83ns - -
stdb_raw 🧠 898.9±3.84ns 889.5±4.16ns - -

Single-row insertions

db on disk schema index type load new latency old latency new throughput old throughput
sqlite 💿 location multi_index 0 - 14.9±1.02µs - 65.7 Ktx/sec
sqlite 💿 location multi_index 1000 - 15.7±0.16µs - 62.1 Ktx/sec
sqlite 💿 location non_unique 0 - 7.1±0.04µs - 137.4 Ktx/sec
sqlite 💿 location non_unique 1000 - 6.9±0.03µs - 141.1 Ktx/sec
sqlite 💿 location unique 0 - 7.2±0.13µs - 136.0 Ktx/sec
sqlite 💿 location unique 1000 - 7.0±0.04µs - 139.3 Ktx/sec
sqlite 💿 person multi_index 0 - 14.3±0.02µs - 68.4 Ktx/sec
sqlite 💿 person multi_index 1000 - 16.1±0.11µs - 60.8 Ktx/sec
sqlite 💿 person non_unique 0 - 7.3±0.05µs - 134.2 Ktx/sec
sqlite 💿 person non_unique 1000 - 7.2±0.04µs - 136.4 Ktx/sec
sqlite 💿 person unique 0 - 7.3±0.09µs - 133.7 Ktx/sec
sqlite 💿 person unique 1000 - 7.2±0.04µs - 135.2 Ktx/sec
sqlite 🧠 location multi_index 0 - 4.1±0.02µs - 239.0 Ktx/sec
sqlite 🧠 location multi_index 1000 - 5.1±0.03µs - 189.8 Ktx/sec
sqlite 🧠 location non_unique 0 - 1860.3±4.71ns - 525.0 Ktx/sec
sqlite 🧠 location non_unique 1000 - 1922.5±12.88ns - 508.0 Ktx/sec
sqlite 🧠 location unique 0 - 1857.1±4.07ns - 525.9 Ktx/sec
sqlite 🧠 location unique 1000 - 1952.6±5.48ns - 500.1 Ktx/sec
sqlite 🧠 person multi_index 0 - 3.7±0.01µs - 261.0 Ktx/sec
sqlite 🧠 person multi_index 1000 - 5.5±0.04µs - 176.2 Ktx/sec
sqlite 🧠 person non_unique 0 - 1964.3±6.51ns - 497.2 Ktx/sec
sqlite 🧠 person non_unique 1000 - 2.1±0.01µs - 473.7 Ktx/sec
sqlite 🧠 person unique 0 - 1938.9±6.11ns - 503.7 Ktx/sec
sqlite 🧠 person unique 1000 - 2.1±0.01µs - 469.8 Ktx/sec
stdb_module 💿 location multi_index 0 52.9±5.07µs 52.8±4.25µs 18.5 Ktx/sec 18.5 Ktx/sec
stdb_module 💿 location multi_index 1000 298.5±41.14µs 250.9±79.96µs 3.3 Ktx/sec 3.9 Ktx/sec
stdb_module 💿 location non_unique 0 47.4±4.11µs 44.3±4.85µs 20.6 Ktx/sec 22.1 Ktx/sec
stdb_module 💿 location non_unique 1000 155.3±16.33µs 213.9±31.16µs 6.3 Ktx/sec 4.6 Ktx/sec
stdb_module 💿 location unique 0 49.6±5.06µs 47.5±5.29µs 19.7 Ktx/sec 20.5 Ktx/sec
stdb_module 💿 location unique 1000 194.5±90.86µs 148.6±25.54µs 5.0 Ktx/sec 6.6 Ktx/sec
stdb_module 💿 person multi_index 0 70.6±4.49µs 64.8±7.93µs 13.8 Ktx/sec 15.1 Ktx/sec
stdb_module 💿 person multi_index 1000 109.5±42.52µs 321.7±126.53µs 8.9 Ktx/sec 3.0 Ktx/sec
stdb_module 💿 person non_unique 0 48.8±5.61µs 51.4±5.01µs 20.0 Ktx/sec 19.0 Ktx/sec
stdb_module 💿 person non_unique 1000 284.7±89.33µs 173.2±10.66µs 3.4 Ktx/sec 5.6 Ktx/sec
stdb_module 💿 person unique 0 60.3±5.28µs 53.6±6.97µs 16.2 Ktx/sec 18.2 Ktx/sec
stdb_module 💿 person unique 1000 113.4±28.11µs 160.1±2.97µs 8.6 Ktx/sec 6.1 Ktx/sec
stdb_module 🧠 location multi_index 0 41.3±3.42µs 39.1±3.19µs 23.7 Ktx/sec 25.0 Ktx/sec
stdb_module 🧠 location multi_index 1000 200.9±51.46µs 184.9±8.11µs 4.9 Ktx/sec 5.3 Ktx/sec
stdb_module 🧠 location non_unique 0 33.6±2.35µs 31.2±2.89µs 29.1 Ktx/sec 31.3 Ktx/sec
stdb_module 🧠 location non_unique 1000 122.9±16.53µs 123.4±10.35µs 7.9 Ktx/sec 7.9 Ktx/sec
stdb_module 🧠 location unique 0 36.6±3.58µs 33.2±3.07µs 26.7 Ktx/sec 29.4 Ktx/sec
stdb_module 🧠 location unique 1000 125.8±2.21µs 201.5±11.49µs 7.8 Ktx/sec 4.8 Ktx/sec
stdb_module 🧠 person multi_index 0 48.5±5.40µs 46.9±4.16µs 20.1 Ktx/sec 20.8 Ktx/sec
stdb_module 🧠 person multi_index 1000 218.6±7.10µs 162.4±73.25µs 4.5 Ktx/sec 6.0 Ktx/sec
stdb_module 🧠 person non_unique 0 35.9±3.19µs 31.8±2.66µs 27.2 Ktx/sec 30.7 Ktx/sec
stdb_module 🧠 person non_unique 1000 117.4±19.62µs 319.7±5.89µs 8.3 Ktx/sec 3.1 Ktx/sec
stdb_module 🧠 person unique 0 40.5±4.05µs 44.0±5.45µs 24.1 Ktx/sec 22.2 Ktx/sec
stdb_module 🧠 person unique 1000 168.1±46.32µs 130.8±5.68µs 5.8 Ktx/sec 7.5 Ktx/sec
stdb_raw 💿 location multi_index 0 7.4±0.01µs 7.4±0.51µs 131.8 Ktx/sec 132.5 Ktx/sec
stdb_raw 💿 location multi_index 1000 10.1±0.17µs 35.1±248.15µs 96.3 Ktx/sec 27.8 Ktx/sec
stdb_raw 💿 location non_unique 0 4.8±0.04µs 4.9±0.50µs 201.4 Ktx/sec 199.6 Ktx/sec
stdb_raw 💿 location non_unique 1000 6.4±0.13µs 6.5±0.16µs 153.5 Ktx/sec 149.4 Ktx/sec
stdb_raw 💿 location unique 0 6.3±0.14µs 6.2±0.02µs 154.5 Ktx/sec 158.3 Ktx/sec
stdb_raw 💿 location unique 1000 8.6±0.17µs 26.3±176.36µs 113.2 Ktx/sec 37.1 Ktx/sec
stdb_raw 💿 person multi_index 0 11.6±0.02µs 11.7±0.38µs 84.4 Ktx/sec 83.7 Ktx/sec
stdb_raw 💿 person multi_index 1000 57.0±419.87µs 47.0±316.04µs 17.1 Ktx/sec 20.8 Ktx/sec
stdb_raw 💿 person non_unique 0 5.9±0.01µs 6.0±0.39µs 164.7 Ktx/sec 163.7 Ktx/sec
stdb_raw 💿 person non_unique 1000 7.7±0.12µs 19.9±119.74µs 126.9 Ktx/sec 49.1 Ktx/sec
stdb_raw 💿 person unique 0 8.4±0.02µs 8.4±0.09µs 116.6 Ktx/sec 116.6 Ktx/sec
stdb_raw 💿 person unique 1000 38.4±272.43µs 39.1±278.81µs 25.4 Ktx/sec 25.0 Ktx/sec
stdb_raw 🧠 location multi_index 0 4.8±0.01µs 4.7±0.01µs 204.7 Ktx/sec 206.3 Ktx/sec
stdb_raw 🧠 location multi_index 1000 6.5±0.03µs 6.6±0.03µs 150.2 Ktx/sec 148.5 Ktx/sec
stdb_raw 🧠 location non_unique 0 2.3±0.01µs 2.3±0.02µs 420.3 Ktx/sec 422.3 Ktx/sec
stdb_raw 🧠 location non_unique 1000 3.0±0.03µs 3.0±0.05µs 329.4 Ktx/sec 323.8 Ktx/sec
stdb_raw 🧠 location unique 0 3.7±0.01µs 3.7±0.01µs 264.2 Ktx/sec 267.4 Ktx/sec
stdb_raw 🧠 location unique 1000 4.9±0.02µs 5.0±0.04µs 198.7 Ktx/sec 195.4 Ktx/sec
stdb_raw 🧠 person multi_index 0 8.4±0.02µs 8.4±0.02µs 116.2 Ktx/sec 116.5 Ktx/sec
stdb_raw 🧠 person multi_index 1000 10.8±0.10µs 10.7±0.10µs 90.5 Ktx/sec 91.6 Ktx/sec
stdb_raw 🧠 person non_unique 0 2.9±0.01µs 2.9±0.01µs 332.6 Ktx/sec 338.8 Ktx/sec
stdb_raw 🧠 person non_unique 1000 3.8±0.02µs 3.7±0.02µs 258.5 Ktx/sec 263.4 Ktx/sec
stdb_raw 🧠 person unique 0 5.3±0.01µs 5.2±0.01µs 185.3 Ktx/sec 187.2 Ktx/sec
stdb_raw 🧠 person unique 1000 6.9±0.24µs 6.6±0.05µs 142.0 Ktx/sec 147.0 Ktx/sec

Multi-row insertions

db on disk schema index type load count new latency old latency new throughput old throughput
sqlite 💿 location multi_index 0 100 - 132.9±2.96µs - 7.4 Ktx/sec
sqlite 💿 location multi_index 1000 100 - 205.4±20.72µs - 4.8 Ktx/sec
sqlite 💿 location non_unique 0 100 - 49.9±1.00µs - 19.6 Ktx/sec
sqlite 💿 location non_unique 1000 100 - 52.6±0.47µs - 18.6 Ktx/sec
sqlite 💿 location unique 0 100 - 51.9±0.97µs - 18.8 Ktx/sec
sqlite 💿 location unique 1000 100 - 56.3±0.27µs - 17.3 Ktx/sec
sqlite 💿 person multi_index 0 100 - 118.4±2.91µs - 8.3 Ktx/sec
sqlite 💿 person multi_index 1000 100 - 226.9±0.48µs - 4.3 Ktx/sec
sqlite 💿 person non_unique 0 100 - 48.5±1.30µs - 20.2 Ktx/sec
sqlite 💿 person non_unique 1000 100 - 58.9±0.30µs - 16.6 Ktx/sec
sqlite 💿 person unique 0 100 - 49.9±5.08µs - 19.6 Ktx/sec
sqlite 💿 person unique 1000 100 - 54.3±0.32µs - 18.0 Ktx/sec
sqlite 🧠 location multi_index 0 100 - 121.1±0.68µs - 8.1 Ktx/sec
sqlite 🧠 location multi_index 1000 100 - 170.0±0.30µs - 5.7 Ktx/sec
sqlite 🧠 location non_unique 0 100 - 43.1±0.47µs - 22.6 Ktx/sec
sqlite 🧠 location non_unique 1000 100 - 43.7±0.39µs - 22.4 Ktx/sec
sqlite 🧠 location unique 0 100 - 45.6±0.27µs - 21.4 Ktx/sec
sqlite 🧠 location unique 1000 100 - 47.2±0.22µs - 20.7 Ktx/sec
sqlite 🧠 person multi_index 0 100 - 106.3±0.40µs - 9.2 Ktx/sec
sqlite 🧠 person multi_index 1000 100 - 188.9±0.35µs - 5.2 Ktx/sec
sqlite 🧠 person non_unique 0 100 - 40.7±0.26µs - 24.0 Ktx/sec
sqlite 🧠 person non_unique 1000 100 - 44.7±0.32µs - 21.8 Ktx/sec
sqlite 🧠 person unique 0 100 - 43.8±0.65µs - 22.3 Ktx/sec
sqlite 🧠 person unique 1000 100 - 47.0±0.48µs - 20.8 Ktx/sec
stdb_module 💿 location multi_index 0 100 869.7±24.87µs 801.6±16.77µs 1149 tx/sec 1247 tx/sec
stdb_module 💿 location multi_index 1000 100 1117.0±12.87µs 990.3±85.42µs 895 tx/sec 1009 tx/sec
stdb_module 💿 location non_unique 0 100 461.8±5.99µs 426.4±53.34µs 2.1 Ktx/sec 2.3 Ktx/sec
stdb_module 💿 location non_unique 1000 100 490.4±21.24µs 563.4±68.50µs 2039 tx/sec 1775 tx/sec
stdb_module 💿 location unique 0 100 561.5±97.56µs 618.9±71.33µs 1781 tx/sec 1615 tx/sec
stdb_module 💿 location unique 1000 100 643.5±76.11µs 754.4±99.63µs 1553 tx/sec 1325 tx/sec
stdb_module 💿 person multi_index 0 100 1065.4±4.59µs 1133.5±281.13µs 938 tx/sec 882 tx/sec
stdb_module 💿 person multi_index 1000 100 1167.2±57.76µs 1239.7±87.31µs 856 tx/sec 806 tx/sec
stdb_module 💿 person non_unique 0 100 558.7±152.51µs 582.1±107.04µs 1789 tx/sec 1717 tx/sec
stdb_module 💿 person non_unique 1000 100 841.0±21.09µs 1032.8±10.68µs 1189 tx/sec 968 tx/sec
stdb_module 💿 person unique 0 100 653.6±10.71µs 716.3±8.28µs 1530 tx/sec 1396 tx/sec
stdb_module 💿 person unique 1000 100 776.3±7.58µs 1019.2±84.49µs 1288 tx/sec 981 tx/sec
stdb_module 🧠 location multi_index 0 100 548.7±16.25µs 649.3±120.30µs 1822 tx/sec 1540 tx/sec
stdb_module 🧠 location multi_index 1000 100 759.8±110.32µs 599.1±7.24µs 1316 tx/sec 1669 tx/sec
stdb_module 🧠 location non_unique 0 100 370.3±22.97µs 368.7±11.94µs 2.6 Ktx/sec 2.6 Ktx/sec
stdb_module 🧠 location non_unique 1000 100 477.2±42.05µs 443.5±15.19µs 2.0 Ktx/sec 2.2 Ktx/sec
stdb_module 🧠 location unique 0 100 531.3±69.15µs 434.9±5.20µs 1882 tx/sec 2.2 Ktx/sec
stdb_module 🧠 location unique 1000 100 600.2±73.22µs 706.4±12.45µs 1666 tx/sec 1415 tx/sec
stdb_module 🧠 person multi_index 0 100 914.7±74.55µs 823.6±19.13µs 1093 tx/sec 1214 tx/sec
stdb_module 🧠 person multi_index 1000 100 1057.3±39.45µs 1086.9±35.80µs 945 tx/sec 920 tx/sec
stdb_module 🧠 person non_unique 0 100 519.1±1.97µs 400.0±37.03µs 1926 tx/sec 2.4 Ktx/sec
stdb_module 🧠 person non_unique 1000 100 526.5±12.54µs 700.4±8.99µs 1899 tx/sec 1427 tx/sec
stdb_module 🧠 person unique 0 100 587.3±3.45µs 580.4±29.67µs 1702 tx/sec 1722 tx/sec
stdb_module 🧠 person unique 1000 100 752.9±5.22µs 753.9±17.52µs 1328 tx/sec 1326 tx/sec
stdb_raw 💿 location multi_index 0 100 344.0±0.49µs 350.0±3.94µs 2.8 Ktx/sec 2.8 Ktx/sec
stdb_raw 💿 location multi_index 1000 100 389.6±210.21µs 403.1±275.32µs 2.5 Ktx/sec 2.4 Ktx/sec
stdb_raw 💿 location non_unique 0 100 116.9±0.20µs 120.1±0.22µs 8.4 Ktx/sec 8.1 Ktx/sec
stdb_raw 💿 location non_unique 1000 100 126.9±82.89µs 133.4±113.60µs 7.7 Ktx/sec 7.3 Ktx/sec
stdb_raw 💿 location unique 0 100 241.0±0.20µs 244.6±0.26µs 4.1 Ktx/sec 4.0 Ktx/sec
stdb_raw 💿 location unique 1000 100 273.7±146.57µs 263.9±1.25µs 3.6 Ktx/sec 3.7 Ktx/sec
stdb_raw 💿 person multi_index 0 100 709.8±17.18µs 715.1±15.68µs 1408 tx/sec 1398 tx/sec
stdb_raw 💿 person multi_index 1000 100 736.4±1.20µs 781.2±369.79µs 1357 tx/sec 1280 tx/sec
stdb_raw 💿 person non_unique 0 100 217.8±0.31µs 219.8±0.71µs 4.5 Ktx/sec 4.4 Ktx/sec
stdb_raw 💿 person non_unique 1000 100 220.5±0.86µs 223.0±0.35µs 4.4 Ktx/sec 4.4 Ktx/sec
stdb_raw 💿 person unique 0 100 427.7±0.48µs 429.1±1.09µs 2.3 Ktx/sec 2.3 Ktx/sec
stdb_raw 💿 person unique 1000 100 446.4±3.47µs 447.8±2.40µs 2.2 Ktx/sec 2.2 Ktx/sec
stdb_raw 🧠 location multi_index 0 100 302.2±0.65µs 302.0±0.31µs 3.2 Ktx/sec 3.2 Ktx/sec
stdb_raw 🧠 location multi_index 1000 100 325.4±0.49µs 328.3±0.66µs 3.0 Ktx/sec 3.0 Ktx/sec
stdb_raw 🧠 location non_unique 0 100 74.9±0.09µs 73.5±0.09µs 13.0 Ktx/sec 13.3 Ktx/sec
stdb_raw 🧠 location non_unique 1000 100 76.2±0.09µs 75.1±0.16µs 12.8 Ktx/sec 13.0 Ktx/sec
stdb_raw 🧠 location unique 0 100 199.2±0.26µs 197.9±0.23µs 4.9 Ktx/sec 4.9 Ktx/sec
stdb_raw 🧠 location unique 1000 100 216.4±0.35µs 216.9±0.72µs 4.5 Ktx/sec 4.5 Ktx/sec
stdb_raw 🧠 person multi_index 0 100 616.6±3.42µs 619.1±1.00µs 1621 tx/sec 1615 tx/sec
stdb_raw 🧠 person multi_index 1000 100 644.3±0.55µs 649.7±1.13µs 1551 tx/sec 1539 tx/sec
stdb_raw 🧠 person non_unique 0 100 127.1±0.15µs 125.5±0.14µs 7.7 Ktx/sec 7.8 Ktx/sec
stdb_raw 🧠 person non_unique 1000 100 130.0±0.24µs 127.8±0.34µs 7.5 Ktx/sec 7.6 Ktx/sec
stdb_raw 🧠 person unique 0 100 336.5±0.46µs 333.1±0.75µs 2.9 Ktx/sec 2.9 Ktx/sec
stdb_raw 🧠 person unique 1000 100 356.7±0.60µs 353.0±0.31µs 2.7 Ktx/sec 2.8 Ktx/sec

Full table iterate

db on disk schema index type new latency old latency new throughput old throughput
sqlite 💿 location unique - 8.9±0.12µs - 109.1 Ktx/sec
sqlite 💿 person unique - 9.3±0.10µs - 104.8 Ktx/sec
sqlite 🧠 location unique - 7.7±0.09µs - 126.6 Ktx/sec
sqlite 🧠 person unique - 8.0±0.09µs - 121.4 Ktx/sec
stdb_module 💿 location unique 50.0±10.00µs 49.7±9.01µs 19.5 Ktx/sec 19.6 Ktx/sec
stdb_module 💿 person unique 64.5±8.99µs 67.5±8.71µs 15.2 Ktx/sec 14.5 Ktx/sec
stdb_module 🧠 location unique 55.2±6.74µs 52.9±7.52µs 17.7 Ktx/sec 18.5 Ktx/sec
stdb_module 🧠 person unique 55.2±10.88µs 55.2±11.42µs 17.7 Ktx/sec 17.7 Ktx/sec
stdb_raw 💿 location unique 9.2±0.10µs 9.0±0.02µs 106.5 Ktx/sec 108.0 Ktx/sec
stdb_raw 💿 person unique 9.2±0.01µs 9.0±0.02µs 106.6 Ktx/sec 108.0 Ktx/sec
stdb_raw 🧠 location unique 9.1±0.10µs 9.1±0.24µs 107.2 Ktx/sec 107.2 Ktx/sec
stdb_raw 🧠 person unique 9.1±0.02µs 9.0±0.07µs 107.3 Ktx/sec 108.6 Ktx/sec

Find unique key

db on disk key type load new latency old latency new throughput old throughput
sqlite 💿 u32 1000 - 2.3±0.01µs - 419.9 Ktx/sec
sqlite 🧠 u32 1000 - 1111.5±4.47ns - 878.6 Ktx/sec
stdb_module 💿 u32 1000 26.3±2.20µs 23.3±1.43µs 37.1 Ktx/sec 42.0 Ktx/sec
stdb_module 🧠 u32 1000 24.2±1.89µs 24.7±1.46µs 40.4 Ktx/sec 39.6 Ktx/sec
stdb_raw 💿 u32 1000 2.2±0.01µs 2.2±0.02µs 452.6 Ktx/sec 442.8 Ktx/sec
stdb_raw 🧠 u32 1000 2.1±0.00µs 2.1±0.00µs 464.1 Ktx/sec 457.8 Ktx/sec

Filter

db on disk key type index strategy load count new latency old latency new throughput old throughput
sqlite 💿 string indexed 1000 10 - 5.6±0.02µs - 175.8 Ktx/sec
sqlite 💿 string non_indexed 1000 10 - 48.3±0.33µs - 20.2 Ktx/sec
sqlite 💿 u64 indexed 1000 10 - 5.4±0.02µs - 181.6 Ktx/sec
sqlite 💿 u64 non_indexed 1000 10 - 33.2±0.05µs - 29.4 Ktx/sec
sqlite 🧠 string indexed 1000 10 - 4.2±0.01µs - 234.6 Ktx/sec
sqlite 🧠 string non_indexed 1000 10 - 46.7±0.23µs - 20.9 Ktx/sec
sqlite 🧠 u64 indexed 1000 10 - 4.0±0.01µs - 243.1 Ktx/sec
sqlite 🧠 u64 non_indexed 1000 10 - 31.6±0.06µs - 30.9 Ktx/sec
stdb_module 💿 string indexed 1000 10 36.8±2.51µs 36.2±3.28µs 26.5 Ktx/sec 27.0 Ktx/sec
stdb_module 💿 string non_indexed 1000 10 170.9±1.49µs 183.1±4.58µs 5.7 Ktx/sec 5.3 Ktx/sec
stdb_module 💿 u64 indexed 1000 10 35.3±2.24µs 33.9±2.33µs 27.7 Ktx/sec 28.8 Ktx/sec
stdb_module 💿 u64 non_indexed 1000 10 146.5±4.86µs 145.3±6.14µs 6.7 Ktx/sec 6.7 Ktx/sec
stdb_module 🧠 string indexed 1000 10 37.8±2.59µs 36.5±2.80µs 25.9 Ktx/sec 26.8 Ktx/sec
stdb_module 🧠 string non_indexed 1000 10 170.1±3.99µs 174.6±2.60µs 5.7 Ktx/sec 5.6 Ktx/sec
stdb_module 🧠 u64 indexed 1000 10 33.7±2.58µs 33.4±2.29µs 28.9 Ktx/sec 29.2 Ktx/sec
stdb_module 🧠 u64 non_indexed 1000 10 140.1±1.50µs 141.0±3.93µs 7.0 Ktx/sec 6.9 Ktx/sec
stdb_raw 💿 string indexed 1000 10 4.7±0.01µs 4.8±0.05µs 209.2 Ktx/sec 203.7 Ktx/sec
stdb_raw 💿 string non_indexed 1000 10 137.6±4.82µs 149.8±0.41µs 7.1 Ktx/sec 6.5 Ktx/sec
stdb_raw 💿 u64 indexed 1000 10 4.6±0.02µs 4.6±0.01µs 213.1 Ktx/sec 210.1 Ktx/sec
stdb_raw 💿 u64 non_indexed 1000 10 115.8±1.59µs 122.5±0.16µs 8.4 Ktx/sec 8.0 Ktx/sec
stdb_raw 🧠 string indexed 1000 10 4.6±0.01µs 4.7±0.05µs 212.1 Ktx/sec 207.6 Ktx/sec
stdb_raw 🧠 string non_indexed 1000 10 158.4±0.46µs 150.2±0.51µs 6.2 Ktx/sec 6.5 Ktx/sec
stdb_raw 🧠 u64 indexed 1000 10 4.5±0.01µs 4.6±0.01µs 215.0 Ktx/sec 213.7 Ktx/sec
stdb_raw 🧠 u64 non_indexed 1000 10 133.5±0.35µs 119.9±1.38µs 7.3 Ktx/sec 8.1 Ktx/sec

Serialize

schema format count new latency old latency new throughput old throughput
location bsatn 100 1641.2±29.14ns 1789.3±32.66ns 58.1 Mtx/sec 53.3 Mtx/sec
location json 100 3.1±0.01µs 3.4±0.08µs 31.1 Mtx/sec 28.0 Mtx/sec
location product_value 100 544.8±0.42ns 557.0±0.53ns 175.1 Mtx/sec 171.2 Mtx/sec
person bsatn 100 2.5±0.01µs 2.6±0.01µs 38.3 Mtx/sec 37.2 Mtx/sec
person json 100 4.7±0.02µs 5.0±0.05µs 20.2 Mtx/sec 19.1 Mtx/sec
person product_value 100 663.3±7.62ns 664.1±1.41ns 143.8 Mtx/sec 143.6 Mtx/sec

Module: invoke with large arguments

arg size new latency old latency new throughput old throughput
64KiB 65.9±6.11µs 69.9±9.29µs - -

Module: print bulk

line count new latency old latency new throughput old throughput
1 22.9±1.48µs 23.3±1.18µs - -
100 201.2±1.06µs 200.9±0.80µs - -
1000 1967.5±801.21µs 1924.9±526.78µs - -

Remaining benchmarks

name new latency old latency new throughput old throughput
special/db_game/load=10 3.5±7.36ms - - -
special/db_game/load=100 1139.5±11.74µs - - -

Copy link
Contributor

@kazimuth kazimuth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great! No complaints from me aside from one comment.

crates/bench/benches/special.rs Show resolved Hide resolved
Copy link
Contributor

@kazimuth kazimuth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, wait, I was running this locally and I thought of a few things that might be helpful.

crates/testing/tests/standalone_integration_test.rs Outdated Show resolved Hide resolved
modules/benchmarks_db/src/lib.rs Outdated Show resolved Hide resolved
crates/bench/benches/special.rs Outdated Show resolved Hide resolved
modules/benchmarks_db/src/lib.rs Outdated Show resolved Hide resolved
Cargo.toml Outdated Show resolved Hide resolved
@mamcx mamcx force-pushed the mamcx/db-perf-simple branch 2 times, most recently from 97bff79 to 4f2fee1 Compare January 25, 2024 18:43
@mamcx mamcx requested a review from kazimuth January 30, 2024 14:07
@bfops bfops added no runtime change This change does not affect the final binaries release-any To be landed in any release window labels Feb 9, 2024
Copy link
Contributor

@cloutiertyler cloutiertyler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This LGTM now!

Copy link
Contributor

@kazimuth kazimuth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general I still think it still to use numbers higher than ten thousand in a criterion benchmark

crates/testing/tests/standalone_integration_test.rs Outdated Show resolved Hide resolved
modules/benchmarks/src/synthetic.rs Show resolved Hide resolved
@kazimuth
Copy link
Contributor

benchmarks please

Copy link

github-actions bot commented Apr 15, 2024

Criterion benchmark results

Criterion benchmark report

YOU SHOULD PROBABLY IGNORE THESE RESULTS.

Criterion is a wall time based benchmarking system that is extremely noisy when run on CI. We collect these results for longitudinal analysis, but they are not reliable for comparing individual PRs.

Go look at the callgrind report instead.

empty

db on disk new latency old latency new throughput old throughput
sqlite 💿 - 421.9±2.09ns - -
sqlite 🧠 - 414.4±1.68ns - -
stdb_raw 💿 724.5±1.19ns 724.6±4.81ns - -
stdb_raw 🧠 696.5±1.62ns 695.8±1.55ns - -

insert_1

db on disk schema indices preload new latency old latency new throughput old throughput

insert_bulk

db on disk schema indices preload count new latency old latency new throughput old throughput
sqlite 💿 u32_u64_str btree_each_column 2048 256 - 521.7±0.69µs - 1916 tx/sec
sqlite 💿 u32_u64_str unique_0 2048 256 - 137.1±0.45µs - 7.1 Ktx/sec
sqlite 💿 u32_u64_u64 btree_each_column 2048 256 - 424.0±0.60µs - 2.3 Ktx/sec
sqlite 💿 u32_u64_u64 unique_0 2048 256 - 125.5±0.90µs - 7.8 Ktx/sec
sqlite 🧠 u32_u64_str btree_each_column 2048 256 - 453.3±0.96µs - 2.2 Ktx/sec
sqlite 🧠 u32_u64_str unique_0 2048 256 - 121.1±0.56µs - 8.1 Ktx/sec
sqlite 🧠 u32_u64_u64 btree_each_column 2048 256 - 370.1±0.38µs - 2.6 Ktx/sec
sqlite 🧠 u32_u64_u64 unique_0 2048 256 - 105.6±0.53µs - 9.2 Ktx/sec
stdb_raw 💿 u32_u64_str btree_each_column 2048 256 720.6±0.36µs 607.6±15.83µs 1387 tx/sec 1645 tx/sec
stdb_raw 💿 u32_u64_str unique_0 2048 256 623.6±0.54µs 470.8±35.07µs 1603 tx/sec 2.1 Ktx/sec
stdb_raw 💿 u32_u64_u64 btree_each_column 2048 256 431.7±0.39µs 403.4±16.12µs 2.3 Ktx/sec 2.4 Ktx/sec
stdb_raw 💿 u32_u64_u64 unique_0 2048 256 392.0±0.80µs 328.8±17.04µs 2.5 Ktx/sec 3.0 Ktx/sec
stdb_raw 🧠 u32_u64_str btree_each_column 2048 256 495.6±0.34µs 336.8±0.12µs 2017 tx/sec 2.9 Ktx/sec
stdb_raw 🧠 u32_u64_str unique_0 2048 256 404.2±0.27µs 257.5±0.15µs 2.4 Ktx/sec 3.8 Ktx/sec
stdb_raw 🧠 u32_u64_u64 btree_each_column 2048 256 326.9±0.67µs 279.4±0.29µs 3.0 Ktx/sec 3.5 Ktx/sec
stdb_raw 🧠 u32_u64_u64 unique_0 2048 256 293.3±0.38µs 239.0±0.39µs 3.3 Ktx/sec 4.1 Ktx/sec

iterate

db on disk schema indices new latency old latency new throughput old throughput
sqlite 💿 u32_u64_str unique_0 - 21.1±0.11µs - 46.2 Ktx/sec
sqlite 💿 u32_u64_u64 unique_0 - 19.7±0.07µs - 49.6 Ktx/sec
sqlite 🧠 u32_u64_str unique_0 - 19.6±0.12µs - 49.9 Ktx/sec
sqlite 🧠 u32_u64_u64 unique_0 - 18.3±0.05µs - 53.5 Ktx/sec
stdb_raw 💿 u32_u64_str unique_0 18.7±0.00µs 18.5±0.00µs 52.3 Ktx/sec 52.9 Ktx/sec
stdb_raw 💿 u32_u64_u64 unique_0 15.9±0.00µs 15.6±0.00µs 61.6 Ktx/sec 62.5 Ktx/sec
stdb_raw 🧠 u32_u64_str unique_0 18.7±0.00µs 18.4±0.00µs 52.3 Ktx/sec 53.0 Ktx/sec
stdb_raw 🧠 u32_u64_u64 unique_0 15.8±0.00µs 15.6±0.00µs 61.7 Ktx/sec 62.6 Ktx/sec

find_unique

db on disk key type preload new latency old latency new throughput old throughput

filter

db on disk key type index strategy load count new latency old latency new throughput old throughput
sqlite 💿 string index 2048 256 - 66.9±0.24µs - 14.6 Ktx/sec
sqlite 💿 u64 index 2048 256 - 64.4±0.07µs - 15.2 Ktx/sec
sqlite 🧠 string index 2048 256 - 64.7±0.44µs - 15.1 Ktx/sec
sqlite 🧠 u64 index 2048 256 - 60.3±0.04µs - 16.2 Ktx/sec
stdb_raw 💿 string index 2048 256 5.6±0.00µs 5.1±0.00µs 175.4 Ktx/sec 189.9 Ktx/sec
stdb_raw 💿 u64 index 2048 256 5.5±0.00µs 5.0±0.00µs 177.2 Ktx/sec 195.3 Ktx/sec
stdb_raw 🧠 string index 2048 256 5.5±0.00µs 5.1±0.00µs 176.3 Ktx/sec 190.8 Ktx/sec
stdb_raw 🧠 u64 index 2048 256 5.5±0.00µs 5.0±0.00µs 177.9 Ktx/sec 196.7 Ktx/sec

serialize

schema format count new latency old latency new throughput old throughput
u32_u64_str bflatn_to_bsatn_fast_path 100 - 3.3±0.04µs - 28.6 Mtx/sec
u32_u64_str bflatn_to_bsatn_slow_path 100 - 3.3±0.00µs - 29.0 Mtx/sec
u32_u64_str bsatn 100 2.5±0.01µs 2.4±0.00µs 38.3 Mtx/sec 39.8 Mtx/sec
u32_u64_str json 100 5.0±0.02µs 5.3±0.03µs 19.1 Mtx/sec 17.9 Mtx/sec
u32_u64_str product_value 100 648.0±1.66ns 1014.4±0.34ns 147.2 Mtx/sec 94.0 Mtx/sec
u32_u64_u64 bflatn_to_bsatn_fast_path 100 - 1395.7±2.41ns - 68.3 Mtx/sec
u32_u64_u64 bflatn_to_bsatn_slow_path 100 - 2.6±0.00µs - 36.8 Mtx/sec
u32_u64_u64 bsatn 100 1780.7±51.30ns 1764.5±36.02ns 53.6 Mtx/sec 54.0 Mtx/sec
u32_u64_u64 json 100 3.2±0.06µs 3.4±0.04µs 29.8 Mtx/sec 28.4 Mtx/sec
u32_u64_u64 product_value 100 597.8±0.36ns 1010.1±2.60ns 159.5 Mtx/sec 94.4 Mtx/sec
u64_u64_u32 bflatn_to_bsatn_fast_path 100 - 1174.4±4.06ns - 81.2 Mtx/sec
u64_u64_u32 bflatn_to_bsatn_slow_path 100 - 2.6±0.01µs - 36.8 Mtx/sec
u64_u64_u32 bsatn 100 - 1652.4±33.01ns - 57.7 Mtx/sec
u64_u64_u32 json 100 - 3.1±0.03µs - 30.4 Mtx/sec
u64_u64_u32 product_value 100 - 1010.6±0.57ns - 94.4 Mtx/sec

stdb_module_large_arguments

arg size new latency old latency new throughput old throughput
64KiB 71.7±9.01µs 68.5±5.16µs - -

stdb_module_print_bulk

line count new latency old latency new throughput old throughput
1 42.6±6.75µs 49.2±5.69µs - -
100 359.3±5.36µs 361.9±28.44µs - -
1000 3.0±0.09ms 2.9±0.40ms - -

remaining

name new latency old latency new throughput old throughput
special/db_game/circles/load=10 601.6±3.02µs - - -
special/db_game/circles/load=100 608.1±6.52µs - - -
special/db_game/ia_loop/load=10 0.0±0.00ns - - -
special/db_game/ia_loop/load=100 0.0±0.00ns - - -
sqlite/💿/update_bulk/u32_u64_str/unique_0/load=2048/count=256 - 45.6±0.18µs - 21.4 Ktx/sec
sqlite/💿/update_bulk/u32_u64_u64/unique_0/load=2048/count=256 - 40.5±0.11µs - 24.1 Ktx/sec
sqlite/🧠/update_bulk/u32_u64_str/unique_0/load=2048/count=256 - 38.7±0.29µs - 25.2 Ktx/sec
sqlite/🧠/update_bulk/u32_u64_u64/unique_0/load=2048/count=256 - 35.1±0.09µs - 27.9 Ktx/sec
stdb_module/💿/update_bulk/u32_u64_str/unique_0/load=2048/count=256 3.1±0.02ms 1308.3±27.81µs 320 tx/sec 764 tx/sec
stdb_module/💿/update_bulk/u32_u64_u64/unique_0/load=2048/count=256 2.3±0.00ms 1078.6±7.69µs 437 tx/sec 927 tx/sec
stdb_raw/💿/update_bulk/u32_u64_str/unique_0/load=2048/count=256 1115.6±1.08µs 588.2±37.27µs 896 tx/sec 1700 tx/sec
stdb_raw/💿/update_bulk/u32_u64_u64/unique_0/load=2048/count=256 763.0±1.09µs 532.0±14.97µs 1310 tx/sec 1879 tx/sec
stdb_raw/🧠/update_bulk/u32_u64_str/unique_0/load=2048/count=256 787.4±0.47µs 422.7±0.33µs 1269 tx/sec 2.3 Ktx/sec
stdb_raw/🧠/update_bulk/u32_u64_u64/unique_0/load=2048/count=256 564.1±0.32µs 391.7±0.19µs 1772 tx/sec 2.5 Ktx/sec

Copy link
Contributor

@kazimuth kazimuth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Sorry for the delay on review I missed the re-up.

@mamcx mamcx added this pull request to the merge queue Jun 13, 2024
Merged via the queue into master with commit 814dd5b Jun 13, 2024
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
no runtime change This change does not affect the final binaries release-any To be landed in any release window
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants