fasthttp benchmark degraded significantly despite no code change

[ww9]

fasthttp ranked #2 at 7 million requests per second in Round 17 2018-10-30.

There was no code change for fasthttp benchmarking as can be seen in the commit history yet the results have been degrading ever since and currently sits at 4.5 million requests per second.

Why is that?

[sebastienros]

And the base docker image is a fixed number, so can't be it. BTW is 1.10.1 old, latest is 1.11.5.

[msmith-techempower]

I will not pretend to understand how Go applications are built, but this line suggests to me that it is going to fetch from the FastHTTP Github repository for a release every time it is built, which probably means that run to run the underlying framework changes.

We can close this issue if someone with Go experience can confirm my assumption.

[ww9]

Indeed the dockerfile fetches latest commit from fasthttp master:

RUN go get -d -u github.com/valyala/fasthttp/...

Perhaps @valyala could shine some light on why the benchmark performance degraded?

I find it quite unlikely that fasthttp made significant performance tradeoffs that could justify requests per second dropping from 7 million to 4.5 million. Specially because this framework is well known for going to great lengths to optimize for performance over other characteristics.

[valyala]

cc'ing @erikdubbelboer and @kirillDanshin

[NateBrady23]

We've had this problem in the past with some Go frameworks and people asking this question or the tests breaking randomly. I don't know much about Go either, but it'd be nice, if it's easy, to get specific releases of all those github dependencies.

[dividedbynil]

Comparing go frameworks latest result with round 17, only fasthttp is affected.

https://www.techempower.com/benchmarks/#section=test&runid=75c6319a-1a25-450c-8165-be0a70eb79ff&hw=ph&test=plaintext&l=zijocf-7
https://www.techempower.com/benchmarks/#section=data-r17&hw=ph&test=plaintext&l=zijocf-1

[kirillDanshin]

@ww9 thanks for pointing out.

That looks like fasthttp's issue. I've created valyala/fasthttp#533 for tracking this on our side.

[waghanza]

@msmith-techempower

I will not pretend to understand how Go applications are built, but this line suggests to me that it is going to fetch from the FastHTTP Github repository for a release every time it is built

That's how go work. However you can use go modules to point to a version (tag) instead of master

[NateBrady23]

Going to close this here since it's being tracked in the proper place

[erikdubbelboer]

This has been fixed, the next run should have a high request per second for the plaintext case again!