Use wrk as a proper sieger #223

waghanza · 2018-06-01T13:04:25Z

Hi,

This PR enable wrk. Siege process is handled by wrk and a custom lua script, write results in a csv file.

The crystal part parse the csv and compute the rank from the average latency.

I'm not quiet sure of how to define this rank (what metric should be in).

@OvermindDL1 Any idea ?

Regards,

OvermindDL1 · 2018-06-01T15:47:59Z

Honestly I'd probably use 2 ranks:

Throughput, showing average as well as the deviation (max/min at least).
Latency, showing average as well as the deviation (max/min at least).

If only one wanted to be picked then I'd pick throughput (per second or so) as it tells 'more' information than just latency while also including some of the latency information within it where latency does not do the opposite. In addition, showing the max tested latency is a massive boon as knowing the 'max' time that a request can take (there is a lot of GC horrors out there) is very useful as if 1 out of every 100 requests on your server suddenly takes 3 seconds instead of 50ms is very bad for the user experience (hence why showing the deviation is useful to know how often it occurs).

Having both throughput and latency as rankings gives you a more complete overall picture.

waghanza · 2018-06-01T16:47:36Z

@OvermindDL1 interesting ... and is the percentile as proper information ?

OvermindDL1 · 2018-06-01T16:58:39Z

Percentile is part of the deviation yep.

waghanza · 2018-06-01T17:01:45Z

@OvermindDL1 sorry I do not understand how those metrics are calculated.
could you explain how those metrics (those defined here) are calculated (and what their are representing) ?

OvermindDL1 · 2018-06-01T17:50:04Z

@waghanza You can get the full deviation via that latency:percentile(float) function it looks like, so probably this to get the full statistical deviation:

   for _, p in pairs({ 50, 90, 99, 99.999 }) do
      n = latency:percentile(p)
      io.write(string.format("%g%%,%d\n", p, n))
   end

Which of course you can output to dot format to generate pretty charts or output to csv or whatever. :-)

And of course the summary.requests/summary.duration should get you the requests/second I think.

Oh, it is also good to report errors, only if non-0 though, those signify major issues and that test needs to be checked on if that happens, so we definitely don't want to ignore those!

OvermindDL1 · 2018-06-01T17:53:08Z

In addition to the other latency values of course*

OvermindDL1 · 2018-06-01T17:55:36Z

All of that information could be put on a whisker-chart or a some other kind of deviation chart to show how things relate in more detail.

For an overall ranking number, I'd probably just use requests-per-second, maybe with a poor weighting for ones with high latency.stdev values (as those indicate wildly fluctuating latencies).

waghanza · 2018-06-06T11:56:32Z

I've pushed a new README, with some results from wrk

Actually I only use the first rule (GET on /) to compute results (so this PR is not mergeable).

The reason I pushed the new results are to find the best way to display them.

@tbrand Do you find the results understandable ? (Results are based on a 5 minutes wrk siege)

waghanza · 2018-06-08T06:00:42Z

@tbrand I have updated the README, I think structure is clearer

DISCLAIMER results are computed with the first rule : GET on /

@OvermindDL1 the results seems more realistic 😜

tbrand · 2018-06-08T06:04:09Z

👍

OvermindDL1

Just primarily some questions.

OvermindDL1 · 2018-06-08T14:16:00Z

README.md

-OS: Linux (version: 4.16.7-200.fc27.x86_64, arch: x86_64)
-CPU Cores: 4
+OS: Linux (version: 4.16.11-100.fc26.x86_64, arch: x86_64)
+CPU Cores: 8


Is this the literal number of cores, or is it the number of hyper-threaded cores? I showed both values in my benchmarker as it is an important value to state and I'd think it should be distinguished here as well.

it the number of physical cores -> my proc is AMD FX-8320E Eight-Core Processor
in fact, I'm running on different computers / kernels

OvermindDL1 · 2018-06-08T14:16:21Z

README.md

-37. [rails](https://github.com/rails/rails) (ruby)
+35. [sinatra](https://github.com/sinatra/sinatra) (ruby)
+36. [django](https://github.com/django/django) (python)
+37. [akkahttp](https://github.com/akka/akka-http) (scala)


Wow, scala's akka one should not be that slow! I wonder if the author forget to enable threading or so... o.O

yes, I think results are wrong

OvermindDL1 · 2018-06-08T14:18:46Z

README.md

-| python                    | tornado                   |      726.349087 |      706.362335 |      718.087107 |
-| nim                       | jester                    |      246.613563 |      246.253752 |      246.379956 |
-| nim                       | mofuw                     |      147.627614 |       83.375723 |      108.593340 |
+| ruby                      | rails                     |               0 |          5244.0 |          0.88MB |


Hmm, should this list be sorted by language name, or should it be sorted by req/s or so since the above is not listing the req/s on the language?

req/s, but a think at a time ... (I'll sort after results stabilization)

OvermindDL1 · 2018-06-08T14:20:18Z

README.md

-OS: Linux (version: 4.16.7-200.fc27.x86_64, arch: x86_64)
-CPU Cores: 4
+OS: Linux (version: 4.16.11-100.fc26.x86_64, arch: x86_64)
+CPU Cores: 8
 ```

 ### Ranking (Framework)


Should the req/s and maybe throutput be listed with these as well so people can see the differences in the ranking, not just the positional ranking? Perhaps it should be a markdown table too?

OvermindDL1 · 2018-06-08T14:23:36Z

tools/src/benchmarker.cr

+    seventy_five: String,
+    ninety: String,
+    ninety_nine: String
+  )


Showing the 4 or 5 9's percentile stat is also useful and standard as it shows the extreme outliers.

waghanza · 2018-06-08T21:42:21Z

@OvermindDL1 I have published other results on https://github.com/tbrand/which_is_the_fastest#ranking-framework

16 threads
15 seconds
1000 connections

but still GET on / => next step is to combine all routes

OvermindDL1

16 saturated threads on 4 native core is going to create a LOT of contention that is going to hobble any properly multi-threaded communicative libraries (which would be most) making the ones not as well multi-threaded look like they perform better than they actually do. 4 core itself is not enough for a good test regardless unless the test is being performed across two machines with a fiber interlink and the benchmark sieger (the one running wrk) has more cores than the server being tested (assuming the server being tested has dedicated 4 native cores).

OvermindDL1 · 2018-06-08T21:50:11Z

README.md

 ```
-OS: Linux (version: 4.16.7-200.fc27.x86_64, arch: x86_64)
+OS: Linux (version: 4.16.13-300.fc28.x86_64, arch: x86_64)
 CPU Cores: 4


4 core is no where near enough for a good test! o.O

waghanza requested review from OvermindDL1 and tbrand June 1, 2018 13:04

waghanza mentioned this pull request Jun 1, 2018

Replace the client with a proper seiger #59

Closed

waghanza force-pushed the master branch 5 times, most recently from f5dbdbf to 80b7718 Compare June 5, 2018 11:56

OvermindDL1 suggested changes Jun 8, 2018

View reviewed changes

OvermindDL1 reviewed Jun 8, 2018

View reviewed changes

waghanza force-pushed the master branch from 83b717f to 79839de Compare June 9, 2018 09:49

waghanza added 8 commits June 11, 2018 12:50

use wrk as sieger

9d780bd

display latency in readme (and rank is defined by latency average)

976cf1c

display readme structure

fecd931

avoid using wrk (external) on check mode

ac8935d

upgrade cmake on travis (for c frameworks)

5db68e3

add a clear structure for readme

94e6087

fix scala benchmark (json parsing error)

6dc99a8

change benchmark metrics

74e815a

waghanza added 2 commits June 11, 2018 12:50

compute metrics using 3 rules

be7b77c

change unit for throughput

9849d50

waghanza changed the title ~~[WIP] Use wrk as a proper sieger~~ Use wrk as a proper sieger Jun 11, 2018

waghanza changed the title ~~Use wrk as a proper sieger~~ Use wrk as a proper sieger Jun 11, 2018

waghanza merged commit 1a556ed into the-benchmarker:master Jun 11, 2018

waghanza mentioned this pull request Oct 11, 2018

Skeptical about controversial results #421

Closed

waghanza mentioned this pull request Jun 13, 2019

Add documentation about sorting param #1419

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use wrk as a proper sieger #223

Use wrk as a proper sieger #223

waghanza commented Jun 1, 2018

OvermindDL1 commented Jun 1, 2018 •

edited

waghanza commented Jun 1, 2018

OvermindDL1 commented Jun 1, 2018

waghanza commented Jun 1, 2018

OvermindDL1 commented Jun 1, 2018

OvermindDL1 commented Jun 1, 2018

OvermindDL1 commented Jun 1, 2018 •

edited

waghanza commented Jun 6, 2018

waghanza commented Jun 8, 2018

tbrand commented Jun 8, 2018

OvermindDL1 left a comment

OvermindDL1 Jun 8, 2018

waghanza Jun 8, 2018

OvermindDL1 Jun 8, 2018

waghanza Jun 8, 2018

OvermindDL1 Jun 8, 2018

waghanza Jun 8, 2018

OvermindDL1 Jun 8, 2018

OvermindDL1 Jun 8, 2018

waghanza commented Jun 8, 2018

OvermindDL1 left a comment

OvermindDL1 Jun 8, 2018

Use wrk as a proper sieger #223

Use wrk as a proper sieger #223

Conversation

waghanza commented Jun 1, 2018

OvermindDL1 commented Jun 1, 2018 • edited

waghanza commented Jun 1, 2018

OvermindDL1 commented Jun 1, 2018

waghanza commented Jun 1, 2018

OvermindDL1 commented Jun 1, 2018

OvermindDL1 commented Jun 1, 2018

OvermindDL1 commented Jun 1, 2018 • edited

waghanza commented Jun 6, 2018

waghanza commented Jun 8, 2018

tbrand commented Jun 8, 2018

OvermindDL1 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

waghanza commented Jun 8, 2018

OvermindDL1 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

OvermindDL1 commented Jun 1, 2018 •

edited

OvermindDL1 commented Jun 1, 2018 •

edited