Broken pipes #382
Comments
|
We're also seeing similar errors when calling writing objects. |
|
I feel like HTTP.jl should retry on IOErrors, but I'm not sure at what level the retry should happen. Maybe the |
|
https://github.com/JuliaWeb/HTTP.jl/blob/master/src/RetryRequest.jl#L34 I believe there are some conditions where it already retries on IOError |
|
I'm a little confused about when |
|
Oh, this might be an issue with how AWSCore.jl is using HTTP.jl? https://github.com/JuliaCloud/AWSCore.jl/blob/master/src/http.jl#L31 |
|
The default stack includes a |
|
It could be that it's not cleaning up bad connections properly and all retries attempt to use the same bad connection from the pool? |
|
Oh, at least 2 of the errors posted are throwing a broken pipe error when closing the connection in a I feel like |
|
Can you link the location of that block? |
|
Seems like |
|
I'll also note that the StreamRequest code also doesn't wrap the error in an https://github.com/JuliaWeb/HTTP.jl/blob/master/src/StreamRequest.jl#L54 |
|
On a related note, it's kinda hard to differentiate |
HTTP.jl can often get `Base.IOError`s (e.g., broken pipe when closing a connection) which aren't caught and wrapped in an `HTTP.IOError`. JuliaWeb/HTTP.jl#382
HTTP.jl can often get `Base.IOError`s (e.g., broken pipe when closing a connection) which aren't caught and wrapped in an `HTTP.IOError`. JuliaWeb/HTTP.jl#382
HTTP.jl can often get `Base.IOError`s (e.g., broken pipe when closing a connection) which aren't caught and wrapped in an `HTTP.IOError`. JuliaWeb/HTTP.jl#382
HTTP.jl can often get `Base.IOError`s (e.g., broken pipe when closing a connection) which aren't caught and wrapped in an `HTTP.IOError`. JuliaWeb/HTTP.jl#382
57: Retry on `Base.IOError`s r=iamed2 a=rofinn HTTP.jl can often get `Base.IOError`s (e.g., broken pipe when closing a connection) which aren't caught and wrapped in an `HTTP.IOError`. JuliaWeb/HTTP.jl#382 Co-authored-by: rofinn <rory.finnegan@gmail.com>
|
So I've discovered a scenario in my own application that leads to these broken pipe errors: if a bunch (purposely a little vague w/ the amount here) of requests are all being attempted simultaneously (i.e. using I've also seen this in the context of AWS S3 uploading when the |
|
But I also appreciate the discussion and investigative work that's been done in this issue so far; I do believe there are some cleanups we can do in MbedTLS/HTTP w/ retry/close behavior here; they tend to be a bit of a red-herring to the actual error going on. |
|
Would it be possible to catch that case, warn and reschedule the request to run synchronously? Maybe using a lock to avoid context switching between opening the connection and writing the data? |
Part of the really core issue is that there doesn't exist such a "lock" to prevent context switching between I'm starting to mull over some refactoring ideas to try and simplify and clean up all the lower-level code that manages sockets/writing. A lot of that code has grown to be a bit messy over the years and is due for some updating. |
|
I'm going to close this as I believe the significant refactorings on master have probably resolved this, which touched directly on cleaning up our close/retry/error propagation and better handling of errors in various places. We can always revisit/open new issues once 1.0 is out and if people see related/new issues. |
Julia 1.0.3
HTTP.jl 0.8.0
MbedTLS.jl 0.6.6
We seem to be be randomly getting a broken pipe error when trying to fetch objects from S3. From what I'm gathering from the traceback the error is occuring when trying to close the SSL connection, so maybe this is issue should be filed against MbedTLS.jl instead?
The text was updated successfully, but these errors were encountered: