23Q4 Performance Regressions as Revealed in TechEmpower Results

[mkvalor]

This issue isn't really about TechEmpower rankings, it's about a recent performance regression revealed by those rankings. I'm simply curious if there was a commit near a certain point of time (see below) which may have unintentionally caused this.

On the page of periodic auto runs of the TechEmpower benchmarks, at https://tfb-status.techempower.com/ , I noticed that the runs which started up till November 12, 2023 showed higher requests-per-second results than we see in more recent runs (starting with the one that began on November 17, 2023 and until the present time).

For instance, in the runs for Nov 12 and prior, we see RPS results like this:

Fortunes:
ntex [tokio,db]      559,042
ntex [async-std,db]  374,854

Data Updates:
ntex [async-std,db]   23,704
ntex [tokio,db]       22,757

However, in runs starting Nov 17 and since then, we see RPS results like this:

Fortunes:
ntex [tokio,db]      406,407   (~150k less)
ntex [async-std,db]  352,372

Data Updates:
ntex [async-std,db]   16,000   (~5k less)
ntex [tokio,db]       15,793

Can anyone think of any code commit -- or maybe benchmark configuration update in their repo at https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks/Rust/ntex which may have contributed to these performance regressions? From the Releases page here, this would have coincided in the vicinity of the following versions:

ntex       0.7.10
ntex-io    0.3.7
ntex-tokio 0.3.1

I totally understand that improvements to ntex may have necessitated these regressions. Just wanted to discover whether this outcome had been noticed by the committers and accepted as a necessary trade-off for improvements, or if this is a surprise.

[fakeshadow]

You can clone the benchmark and run it locally to find out what commit/change is the cause.

[fafhrd91]

i dont remember drop in performance from ntex upgrade, could you include links to results. i remember there was some problems with techempower environment at some point.

[mkvalor]

Hi @fafhrd91

Yes, sure.

Here are links to several random intermittent runs in 2023, up to and including the one that started Nov 12 2023. They all have Data Update category results for ntex well above 20K/s:
May 7: https://www.techempower.com/benchmarks/#section=test&runid=bc7f7570-a88e-48e3-9874-06d7dc0a0f74
Sep 19: https://www.techempower.com/benchmarks/#section=test&runid=de12400a-863e-4f9e-9b93-dea9eb260422
Oct 24: https://www.techempower.com/benchmarks/#section=test&runid=0d656625-036a-4b2f-b255-cd3ca188a42c
Nov 12: https://www.techempower.com/benchmarks/#section=test&runid=99e0b150-7cac-4fe7-9b10-5eb526d877ef

And here are the only two finished runs that started beginning Nov 19, 2023. Both of them show ntex with ~5K/s fewer Data Updates (more than 20% less) compared to all the tests above:
Nov 19: https://www.techempower.com/benchmarks/#section=test&runid=502e3495-dda5-4232-a89a-7c08755fcf0e
Nov 26: https://www.techempower.com/benchmarks/#section=test&runid=341a0547-80f0-45f8-9245-04ae0e0e1bcf

All other top-20 frameworks stayed within the top 20 across the Nov 12 and Nov 19 results, including Axum, Drogon, and Just-JS (all written in different langs). Only ntex fell from #1 on Nov 12 to #30 on Nov 19 and #32 on Nov 26.

PS: I just noticed TechEmpower hasn't completed any more intermittent runs since the one that started Nov 26, though they have one incomplete run that began in December. Before that, they made several runs per month. A banner at the top of the page at https://tfb-status.techempower.com/ explains they are getting new server hardware (hence the pause).

[fafhrd91]

i remember now, i did experiment with number of worker threads

TechEmpower/FrameworkBenchmarks#8535

i changed it back but new run never completed
TechEmpower/FrameworkBenchmarks#8569

in next run numbers should recover

[mkvalor]

Awesome. Glad this was taken care of!