Clarify results

[waghanza]

Hi,

Following the #238, it COULD be awesome to report :

• Some result ranked by req/s
• Some result ranked by latency

It also COULD help clarify results. May developers argue that comparing so languages such as node and some such as elixir (which targets are completely different) are not very relevant.

This PR show results order by req/s AND by latency

Regards,

[waghanza]

@OvermindDL1 Should not we use percentile, instead of average latency to rank by latency ?
it will be more accurate no ?

[OvermindDL1]

50th percentile, 99.99th percentile, and standard deviation would be the best to show as it shows the latency range and how it grows.

[waghanza]

@OvermindDL1 my question is want metric (one ?) use to order latency tab ?

is it accurate to differentiate both table ?

[OvermindDL1]

I'd include all information on both tables, but I'd use two table, one sorted on req/s, the other sorted on 50th percentile latency.

If only Github had the ability to mark a table as sort-by-click-on-header. Could always make a dedicated HTML for that though (github pages or so).

[waghanza]

Don't you think tables will be huge ?

My final purpose is to compute result, store them in a BigQuery or else, and having a website displaying them

@tbrand @OvermindDL1 What do you think ?

[OvermindDL1]

Eh, the tables would be long sure, but that seems fine to me (especially if you can clamp the headers to the top as it scrolls).

[waghanza]

@OvermindDL1 On a race between ruby / node / php / python, I have different results (ranks) for latency :

• average
  1. roda
  2. rack-routing
  3. flame
  4. hanami
  5. sinatra
  6. polka
  7. rayo
  8. fastify
  9. japronto
  10. restify
• 50th percentile
  1. roda
  2. rack-routing
  3. flame
  4. sinatra
  5. hanami
  6. polka
  7. rayo
  8. fastify
  9. koa
  10. restify
• 90th percentile
  1. roda
  2. rack-routing
  3. flame
  4. polka
  5. sinatra
  6. rayo
  7. hanami
  8. japronto
  9. restify
  10. fastify
• 99th percentile
  1. roda
  2. rack-routing
  3. flame
  4. hanami
  5. sinatra
  6. rayo
  7. japronto
  8. polka
  9. restify
  10. fastify

I suggest then to keep existing version (sorting by average) and adjust if after using a cloud to compute results

[OvermindDL1]

Low latencies at high percentiles most likely means they generate little to no garbage in comparison. :-)

[waghanza]

@OvermindDL1 not understood

[OvermindDL1]

Just me guessing at what might cause 50 and 99+ percentile differences, ignore. ^.^

For note, this is a good chart for how to display the overarching information:

Add in req/sec as well and it would be a good detail showing for each framework.

[tbrand]

Could you remove the column which is not related to each ranking?
For example, remove Requests / s for Ranked by latency ranking.

Other things are acceptable for me 👍

[waghanza]

I've done a little change to the benchmarker code base, i know use a redis to store result (redis was picked because of simplicity but it add a dependency).

We need to refactor using an other key/value store (to avoid adding dependency) between :

• https://github.com/kostya/kyotocabinet
• https://github.com/CodeSteak/Nuummite
• https://github.com/crystal-community/kiwi

[OvermindDL1]

(redis was picked because of simplicity but it add a dependency)

As long as it's ran via docker or so and not needed to be installed. Why not just use leveldb/rockdb if it's just a KV store needed or sqlite for querying? Or just map?

[waghanza]

@OvermindDL1 I mean a redis installed on sieger machines, not targets. I need a k/v store to store all results as json (at least in order to let us decide what informations to render in each tables)

[OvermindDL1]

I need a k/v store to store all results as json (at least in order to let us decide what informations to render in each tables)

Definitely no point in using something as crazy-heavy as redis, just embed leveldb or rocksdb, I use both rather excessively, and they will far far outperform redis.

[waghanza]

@OvermindDL1 sure, it was I have said 😛
the list above was essentially for me picking an idea ^^

'cause it was late last night (and I was not very productive), I take a random tool -> I'll rewrite this part

[OvermindDL1]

@waghanza yep, 50% makes sense, for someone else then 99% may make sense, etc... etc... I really wish github had sortable column by headers, maybe need to set up a github pages so they become clickable. ^.^;

[waghanza]

@OvermindDL1 I have find that it's possible, but very hard to use / maintain (sortable table). my idea is to push results is BigQuery or else and create a website for this (github's page when #252 😛 )

[OvermindDL1]

No need for BigQuery, just generate an html file via a template and use some progressively enhancing javascript to make the table sortable via column.

[waghanza]

@OvermindDL1 I know it could be done like that, but I find it mor flexible to separate compuation from layout

e.g -> use travis to compute, and a website to display results 😜

moreover, compute and generate markdown on travis need to push from travis to github, no a big fan of this kind of solution, but I suggest we open an other issue for this question 😝

[waghanza]

@tbrand Just after merging this, I'll release 💯