EPIC: Benchmarking 1/x

[mavilein]

Description: We want to convert the benchmarking suite that we used for Prisma 1 to run against Prisma 2 to know where we are performance wise.

Spec:

• Convert the existing benchmarking tooling to run against Prisma 2.
• Explore whether we can store the results in a timescaledb
• Explore whether Grafana is a good fit to visualize test results.
• Make it convenient to profile applications locally, by e.g. integrating flamegraphs.
• Create small reports on our findings, e.g.: Where are we worse and where are we better? Can we pinpoint the underlying issues?
• Create an automated setup so that we can see the change of benchmarks over time. E.g. run them daily or on each commit or ...

[pimeys]

Working on it.

• The benchmarking tools work against prisma2.
• Chose the schemaless and easier option with InfluxDB
• Grafana is a beaut. Already running graphs with it.

Now for the upcoming days I plan to:

• Test different scenarios against scala and rust versions
• Try out different executors
• Try to integrate metrics to the rust codebase so I'll see real time histograms and CPU/RAM usage
• Write that report :)

[pimeys]

Some performance tests:

• Switch the global allocator from system to jemalloc, which works quite well for multi-threaded applications, but increases the binary size a bit (700kb) https://crates.io/crates/jemallocator
• Switch from actix to tokio executor and see how it affects to the throughput

[pimeys]

https://github.com/dignifiedquire/rust-gperftools

[pimeys]

https://github.com/KDAB/hotspot/blob/master/README.md#visualize-data

[pimeys]

The results from the benchmarks and what could we do better:

https://www.notion.so/prismaio/Benchmarking-9cfa29eb09924f39a908184a49780673

[pimeys]

New issues from this:

• Test possible slog replacements, such as tracing. We might need to write our own json output module.
• Automated benchmarks. We should instrument prisma server to measure every single request and have a prometheus/grafana setup for better timescale graphs.
• Threaded connection pool with poll_fn/blocking, async/await all the way up.
• Never ever start a transaction if not really needed.

Should we close the epic? @janpio @mavilein

[janpio]

If all the work suggested here was done, this seems like a reasonable thing. But maybe @mavilein wants to add these follow up tasks.

[mavilein]

I have created the followup tasks here:

• Query Engine: Find a replacement for our current logger (performance) #91
• Query Engine: Automated Benchmarks #92
• Query Engine: Asyncify #93
• Query Engine: Start a Transaction only when needed #94
• Query Engine: Run Benchmarks continuously #95