compairing 'peak' performance

[Licenser]

I see a issue with comparing 'peak' performance especially over a set of different tests (aka different concurrency levels).

This seems a bit odd to me, I'd have expected to look at the lowest number - do a worst case "you'll get at least X" comparison instead of "when you're lucky you might even get X" best case. Especially with concurrency which is likely not to end up in the frameworks sweet spot.

Alternatives would be a average, weighted average or multiple benchmarks at different concurrency levels.

[bhauer]

In most/all cases, the lowest performance is at the lowest level of concurrency we tested (8) because in that scenario the platform/framework doesn't get to demonstrate its capacity to process new requests while I/O is happening with other requests.

That said, you can view the data we captured at various concurrency levels in the line charts and data tables. In the data tables, the peak performance is highlighted in dark red.

Most frameworks reach a plateau and then level off as concurrency rises. However, if you look at the line charts, especially on dedicated hardware, there are a couple that decline after reaching their peak.

Is the data we've provided in the line charts and tables not what you're looking for?

[Licenser]

The tables and line charts are fine, but not what the eye is guided too - I feel what we see on the first glance should be the 'most correct' interpretation of the data and peak performance seems not to be the most correct performance.

That said with the given set of concurrency levels you are entirely right, the lowest performance is currently with the lowest concurrency for most frameworks, comparing those would be further from the 'optimal interpretation' of the data then comparing peak levels.

Also please don't see this is blind criticism, I love the work you've done and think it's a great idea to have a tool for framework benchmarks - it was a ease to add additional frameworks which means this is nicely extendable too. So with a fresh set of eyes I like to address the issues I see on the benchmark with the hope to improve ont he relevance of the results.

Cheers,
Heinz

[bhauer]

Hi @Licenser. I am going to close this issue up. I think this was addressed in our conversation about concurrency levels. But let me know if you feel otherwise. I also have as a goal to add more flexibility to the tables and charts, but that will take me some time.