New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: add timeouts to avoid goroutine leaks in net/http #14995
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't see it being applied. PTAL.
Limits looks good. It is virtually impossible to protect against this type of attacks at this level. It just takes a trivial bump in attack rate to get around this.
But whatever we do, there are still slow-lorris attacks which can eat up connections. Virtually impossible to guard against, though it will require authentication.
It does the total connection build up becomes lower automatically, they grow up to some value and then taper off. However yes there is just no way in reality to have this to some extent. That's why it's configurable as a hidden value. |
59b6efa
to
e1e1053
Compare
|
@klauspost correct I meant to keep these as reasonable protection, of course you can basically make enough fast connections before server considers them idle - timeouts a bloody game :-) |
Following code can reproduce an unending go-routine buildup, while keeping connections established due to lack of client not closing the connections. https://gist.github.com/harshavardhana/2d00e6f909054d2d2524c71485ad02e1 Without this PR all MinIO deployments can be put into denial of service attacks, causing entire service to be unavailable. We bring in two timeouts at this stage to control such go-routine build ups, new change - IdleTimeout (to kill off idle connections) - ReadHeaderTimeout (to kill off connections that are too slow) This new change also brings two hidden options to make any additional relevant changes if desired in some setups.
e1e1053
to
f5ea4dc
Compare
Mint Automation
14995-f5ea4dc/mint-large-bucket.sh.log:Deleting image on docker hub |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Description
fix: add timeouts to avoid goroutine leaks in net/http
Motivation and Context
Following code can reproduce an unending go-routine buildup,
while keeping connections established due to lack of client
not closing the connections.
https://gist.github.com/harshavardhana/2d00e6f909054d2d2524c71485ad02e1
Without this PR all MinIO deployments can be put into
denial of service attacks, causing the entire service to be
unavailable.
We bring in two timeouts at this stage to control such
go-routine build-ups, new change
This new change also brings two hidden options to make any
additional relevant changes if desired in some setups.
How to test this PR?
Test with the
gistshared with the latest master and observethe connection build-up locally, this can be reproduced with
single MinIO running.
This eventually causes all incoming API calls to slowdown
or not respond - eventually requiring MinIO service to be
restarted.
Types of changes
Checklist:
commit-idorPR #here)