-
-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Large POST requests result in SSL3_WRITE_PENDING from urllib3.contrib.pyopenssl#sendall #855
Comments
I'm currently working on a PR to potentially fix this, but I am not sure where the best place to add a new test case is - should it be added to I ask because it looks like these specific API calls in pyopenssl.py are not being specifically called, but I can definitely be missing something. |
@strife25 |
That's what I just discovered as well! A more descriptive error would help tremendously here as I'm sure other people using elasticsearch-py may hit this issue (especially as people start using it on AWS over HTTPS) and it is not clear what is happening due to the many layers of libraries occurring. Thank you for the response and clarification. |
@strife25 I'll happily review/guide on a PR to add that more descriptive error if you're interested in doing the work! |
In this case, the bytes are being passed directly, but do we know whether Requests is encoding JSON it produces to handle this case? If it's not known already, I can dig into that later and open an |
I think requests does not handle that appropriately @haikuginger. |
Good to know @Lukasa; I'll have a look at that tonight. |
I worry that trying to enforce a specific type will hurt more than help, in case people are depending on scenarios where they pass in the wrong type that happens to coerce into the right format? We could intercept TypeErrors and annotate them with more info but I'm not sure that's worth the effort? There are a lot of places where the wrong type yields explosions and it would be hard to decide where to draw the line or where it's most useful. |
It's hard to say whether there is much value in being clearer on the exception is here. We could definitely log though, which would be helpful. |
for my case , the main reason is I have over 2000 tags for 90 tweets uploaded to elasticsearch with chunk_size default 500 , then error SSL3_WRITE_PENDING . if 1000 tags for 90 tweets being uploaded with chunk_size default 500 , no problem . I changed chunk_size to 30 when total tags count more than 1000 like below 👍
Above solved the errors of "SSL routines', 'SSL3_WRITE_PENDING', 'bad write retry" . Hope it will be helpful for other people . |
Hey there, I'm still experiencing this, although it looks like related issues were solved long ago. I'm also using elasticsearch to upload a bunch of documents; decreasing the number of documents in the batch definitely improves the error rate. Is there anything I can do to help debug this?
|
Are you experiencing it using Requests or urllib3? |
@Lukasa I'm using Requests through the elasticsearch library. Here's the full stacktrace:
|
Your Requests version is probably too old to have this fix. Can you try updating it? |
@Lukasa thanks for looking into this. I've also tried with the latest Requests, but the stacktrace only changes the line numbers slightly:
|
Hrm, this should be fine. Can you use pdb to catch this and check what the type of |
If this is the same issue as before, I believe the fix we did in Requests was only for JSON generated by Requests; if the ES lib is generating its own Unicode body, then we weren't doing anything about it. Let's start throwing an appropriate warning when receiving a Unicode body and think about killing this footgun altogether in v2. |
|
Here's what I got from the debugger
|
Ok, yeah, this seems like an elasticsearch problem. You should be getting warnings from PyOpenSSL, why aren't you? |
edit: nevermind, found it above in the thread 👍 hmm… I'm unsure about what's wrong with the unicode data. What warning should I be getting? @haikuginger could you please point me to the fix? I can try to fix this from within the ES library |
Sending data of type |
There's something weird going on here. After inspecting the ES library, I see they are correctly encoding the body before performing the request: https://github.com/elastic/elasticsearch-py/blob/5.0/elasticsearch/transport.py#L310
At this point in the stacktrace, body is indeed
Following the same trace, I see where the document is being converted to unicode again:
This |
You are probably providing a |
you're definitely right @Lukasa; the auth headers set by psf/requests#3573 I'll take the issue there. Thanks so much for your support! |
Otherwise we can get: "Error: [('SSL routines', 'ssl3_write_pending', 'bad write retry')]" errors see urllib3/urllib3#855
pyOpenSSL is deprecated and will be removed in future release version 2.x (#2691). |
I currently have a use case where I am sending a large set of data over https via a POST request. Specifically, I am trying to set a POST request to Elasticsearch with an HTTPS URL that has a body of ~2MB.
When sending these types of requests, I get the following stack trace:
At first, I thought this issue was a result of #717, but that issue has been resolved in the version of requests I am using. After further research, I landed on this comment in an SO answer
Using this clue, I added some print statements to
urllib3.contrib.pyopenssl#_send_until_done
andOpenSSL.SSL#send
:urllib3.contrib.pyopenssl#sendall
OpenSSL.SSL#send
After running this code, I realized that what was occurring was that a specific buffered subset of my data was being retried, BUT, the memory address of the buffered data was changing because of the call to
buf = _text_to_bytes_and_warn("buf", buf)
inOpenSSL.SSL#send
. Here is the output I received:As you can see in that output, the buffer of data in
urllib3.contrib.pyopenssl#_send_until_done
is correctly retried when a failure occurred and it has the correct memory address (0x10a2bba20
), however, on the second attempt, the memory address of the buffer inOpenSSL.SSL#send
is different between the two attempts (0x7fdd6585f400
and0x7fdd638f3000
).Considering this, it looks like the use of
OpenSSL.SSL#send
is the incorrect API to use inurllib3.contrib.pyopenssl#sendall
. A potential solution is to instead useOpenSSL.SSL#sendall
, which seems to account for large requests by performing the chunking of data after_text_to_bytes_and_warn
has been executed.Relevant versions in use:
Python 2.7.10
requests 2.10.0
pyOpenSSL 0.15.1
urllib3 1.15.1
The text was updated successfully, but these errors were encountered: