New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fasthttp benchmark degraded significantly despite no code change #4410

Closed
ww9 opened this Issue Feb 4, 2019 · 10 comments

Comments

Projects
None yet
9 participants
@ww9
Copy link

ww9 commented Feb 4, 2019

fasthttp ranked #2 at 7 million requests per second in Round 17 2018-10-30.

There was no code change for fasthttp benchmarking as can be seen in the commit history yet the results have been degrading ever since and currently sits at 4.5 million requests per second.

Why is that?

@sebastienros

This comment has been minimized.

Copy link
Contributor

sebastienros commented Feb 4, 2019

And the base docker image is a fixed number, so can't be it. BTW is 1.10.1 old, latest is 1.11.5.

@msmith-techempower

This comment has been minimized.

Copy link
Member

msmith-techempower commented Feb 4, 2019

I will not pretend to understand how Go applications are built, but this line suggests to me that it is going to fetch from the FastHTTP Github repository for a release every time it is built, which probably means that run to run the underlying framework changes.

We can close this issue if someone with Go experience can confirm my assumption.

@ww9

This comment has been minimized.

Copy link
Author

ww9 commented Feb 5, 2019

Indeed the dockerfile fetches latest commit from fasthttp master:

RUN go get -d -u github.com/valyala/fasthttp/...

Perhaps @valyala could shine some light on why the benchmark performance degraded?

I find it quite unlikely that fasthttp made significant performance tradeoffs that could justify requests per second dropping from 7 million to 4.5 million. Specially because this framework is well known for going to great lengths to optimize for performance over other characteristics.

@valyala

This comment has been minimized.

Copy link
Contributor

valyala commented Feb 5, 2019

@nbrady-techempower

This comment has been minimized.

Copy link
Member

nbrady-techempower commented Feb 5, 2019

We've had this problem in the past with some Go frameworks and people asking this question or the tests breaking randomly. I don't know much about Go either, but it'd be nice, if it's easy, to get specific releases of all those github dependencies.

@kirillDanshin

This comment has been minimized.

Copy link

kirillDanshin commented Feb 6, 2019

@ww9 thanks for pointing out.

That looks like fasthttp's issue. I've created valyala/fasthttp#533 for tracking this on our side.

@waghanza

This comment has been minimized.

Copy link
Contributor

waghanza commented Feb 8, 2019

@msmith-techempower

I will not pretend to understand how Go applications are built, but this line suggests to me that it is going to fetch from the FastHTTP Github repository for a release every time it is built

That's how go work. However you can use go modules to point to a version (tag) instead of master

@nbrady-techempower

This comment has been minimized.

Copy link
Member

nbrady-techempower commented Feb 11, 2019

Going to close this here since it's being tracked in the proper place

@erikdubbelboer

This comment has been minimized.

Copy link
Contributor

erikdubbelboer commented Feb 12, 2019

This has been fixed, the next run should have a high request per second for the plaintext case again!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment