Benchmarking vignette

[schloerke]

Hi @dselivanov !

From request: the-benchmarker/web-frameworks#3334 (comment)

I've updated the document to try to have plumber do the same code style path as what RestRServe performs. I added in plumber + future into the comparison image at the bottom.

Please re-run the document chunks manually to regenerate the images using your machine (plot_results() calls ggsave()).

Other updates:

• We are not trying to benchmark json conversion, so I have each app returning text/plain.
• calc_fib(15), calc_fib(20), and calc_fib(25) had their median times scale by a factor of 10x on my machine. I figured this was a good comparison of "low", "medium", and "high" computation
• I adjusted the y scale to use the expansion parameter, rather than using a fixed addition. (Would not work well with the comparison plot.)
• The log of the background process is printed after the process has completed. This helps with debugging and making sure each process ran as expected.

I am shocked as to how good future works within plumber. I was not expecting that. The performance could be dramatically reduce if the amount of communication was increased (which RestRServe would not need to worry about).

This is your document. Please update/adjust/tweak/toss/include any of the content.

Best,
Barret

[schloerke]

btw... from my local testing of other frameworks in https://github.com/the-benchmarker/web-frameworks, I think R as a whole is going to get crushed. 😞

[dselivanov]

@schloerke thanks for PR! It will take some time to review (hopefully early next week).

[schloerke]

Had feedback to update the axis labels to avoid confusion. The plots will need to be re-rendered.

• Update images

[dselivanov]

I'm really sorry that it takes so long to review. Very busy last weeks. Mb @artemklevtsov can help with initial review?

[schloerke]

@dselivanov Take your time. I understand. Hopefully it can be of use before your next release.

[dselivanov]

@schloerke sorry once again that it took that long. I've updated couple things and pushed in your branch. Major is keep-alive parameter (which was 10000 and now -1). It seems didn't work properly on newer versions on apib (mb it is in nanoseconds?).

future seems introduce significant latency and it doesn't look it worth to use it when computation requirement is moderate...

[dselivanov]

Also I've switched to desktop machine for benchmarks. On laptop they are not reliable at all - system changes CPU frequency very aggressively depending on temperature and number if cores used (each time I run results were considerably different).

[dselivanov]

Another observation is that plumber with keep alive disabled performs better compared to keep alive enabled... Which is counter intuitive and generally should not be the case.

[t-wojciech]

Here are the new results from web frameworks benchmarker.

Pos | Language | Framework | Speed (64) | Speed (256) | Speed (512)
107 | r (4.0) | rserve (1.7) | 41 968 | 29 638 | 29 848
189 | r (4.0) | restrserve (0.3) | 2 754 | 2 652 | 2 582
192 | r (4.0) | httpuv (1.5) | 1 912 | 1 814 | 1 736
200 | r (4.0) | plumber (1.0) | 429 | 438 | 432

CC @vikram-rawat