Make sure libcrypto threadsafety callbacks are properly set up #290
Conversation
Multithreaded programs using libcrypto (part of OpenSSL) need to set up callbacks to ensure safe execution. Both Python and libpq set up those callbacks, which might lead to a conflict. To avoid leaving dangling function pointers when being unloaded, libpq sets up and removes the callbacks every time a SSL connection it opened and closed. If another Python thread is performing unrelated SSL operations (like connecting to a HTTPS server), this might lead to deadlocks, as described in http://www.postgresql.org/message-id/871tlzrlkq.fsf@wulczer.org Even if the problem will be remediated in libpq, it's still useful to have it fixed in psycopg2. The solution is to use Python's own libcrypto callbacks and completely disable handling them in libpq.
|
Here's a script to reproduce deadlock when one Python thread is doing libpq over SSL and another is doing unrelated SSL: https://gist.github.com/wulczer/82b5b52a420ac7387694 Apart from the discussion in the linked pg-hackers thread these threads are relevant: http://www.postgresql.org/message-id/48620925.6070806@pws.com.au |
|
Patch seems fine to me. I don't have an environment to run the tests but I'll do that and merge tomorrow unless @dvarrazzo beats me to it. :D |
|
I'm totally fine with the patch (thank you Jan!) but I prefer to apply it to my branch, where I can make the tests run, and then merge it to master. |
|
Sure thing! Feel free to close the PR and apply the patch via different means :) |
|
It's fine as it is: I don't really like the way github makes the merge (with an extra commit) but it detects automatically if the changeset is pushed "manually" and closes the requests itself. We can leave everything as it is now ;-) |
|
Hi! Just checking in, where are we on this? |
|
Right, I forgot to include this in the last release :\ sorry, I only got through the bug list and skipped the pull request. I'll try to release it soon in a 2.6.1. |
|
No problem at all, I actually submitted this after 2.6 got released. Just wanted to make sure it doesn't fall through the cracks, thanks! |
|
Absolutely no, thank you: it must have been a tricky nut to crack! |
|
Merged, thank you very much. |
|
Yay! Thank you! |
|
Is there any chance of getting this into a stable release soon? We get bitten by this relatively frequently. |
|
I was aiming for a release the past week but I couldn't. I'll try to have it out before Monday. |
|
Great! Thanks very much. |
|
Still have to write the release notes to -announce but 2.6.1 is already tagged on github and released on PyPI. |
|
Thanks! |
|
@wulczer I was happy as fuck when I saw this issue - debugging our app for two days, strange freezes in connect, thinking I am the only one who is expereincing it. Seeking for some wrong initialization of SSL and in the end, all I needed was to update the psycopg2. :) Thanks a lot, you just made my day. |
|
@KarelJakubec Yeah! I love the sound of bugs squashed in the morning 🤘 |
|
You are my savior, @wulczer |
|
Glad to help @danhnguyen0902 :) |
Multithreaded programs using libcrypto (part of OpenSSL) need to set up
callbacks to ensure safe execution. Both Python and libpq set up those
callbacks, which might lead to a conflict.
To avoid leaving dangling function pointers when being unloaded, libpq sets up
and removes the callbacks every time a SSL connection it opened and closed. If
another Python thread is performing unrelated SSL operations (like connecting
to a HTTPS server), this might lead to deadlocks, as described in
http://www.postgresql.org/message-id/871tlzrlkq.fsf@wulczer.org
Even if the problem will be remediated in libpq, it's still useful to have it
fixed in psycopg2. The solution is to use Python's own libcrypto callbacks and
completely disable handling them in libpq.