-
Notifications
You must be signed in to change notification settings - Fork 322
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Threading.current_thread does not change when using greenthreads? #172
Comments
Another example that has a similar issue: https://review.openstack.org/#/c/138217/8/oslo_concurrency/lockutils.py (the |
threading.get_ident() is patched, but not threading.current_thread(). |
I'm wrong, I looked at eventlet/green/thread.py (thread), but I see current_thread() function in eventlet/green/threading.py (threading). |
I may have found a super hacky fix to this issue. I want to emphasize that this is THOROUGHLY UNTESTED (and by that I mean it works for me but no guarantees to anyone else). The code you have to change is in Lines 102-104 read as follows: try:
t = active[id(g)]
except KeyError:
... Replace those lines (102-104) with the following code block: try:
if id(g) in active:
t = active[id(g)]
else:
all_threads = patcher.patch_function(__import__('threading').enumerate)()
all_idents = {th.ident: th for th in all_threads}
t = all_idents[id(g)]
except KeyError:
... You'll also need to comment out line 26 to be able to reuse The AGAIN, I would like to emphasize that this is IN NO WAY a permanent fix, but as far as my testing goes, it's seems to be working for my setup. |
@jaketesler thank you. |
It works for me, I tested it in an OpenStack project (OpenStack/Magnum) with the kubernetes client which uses multiprocessing. #147 |
@jaketesler @strigazi @harlowja Please, try this patch.
|
works for me :) with oslo.service, so I expect other openstack services to not be affected. |
@strigazi yes, sorry for delay, I'll merge it. |
@temoto You will do a release too? xD |
Also solved my issues just to have another confirmation! Way to go fixing a > 3-year-old issue ; ) Thanks. A release would be cool. |
@temoto Apologies for pinging again. Eventlet is a very important dependency for OpenStack projects but not all projects suffer from this bug. The project I'm working needs this fix and I want to "sell" the release bump well before the next openstack release. Can we have a release? :) |
Sure.
…On Fri, May 4, 2018, 14:56 Spyros Trigazis ***@***.***> wrote:
@temoto <https://github.com/temoto> Apologies for pinging again. Eventlet
is a very important dependency for OpenStack projects but not all projects
suffer from this bug. The project I'm working needs this fix and I want to
"sell" the release bump well before the next openstack release. Can we have
a release? :)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#172 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AABpsQLsiudbHhMPxjVr5p_EYuGXZbkNks5tvEHPgaJpZM4DEAB8>
.
|
It's 0.23 on pypi. |
Issue [1] got resolved/addressed in eventlet in 0.23 version and currently Openstack using eventlet's 0.24 version. So removing the workaround code. [1]: eventlet/eventlet#172
The fasteners lib in version 0.15.0 removed the threading.current_thread workaround for eventlet[1] because eventlet seemed to fixed the current_thread issues tracked in [2]. However the fix for [2] only fixed half of the problem. The threading.current_thread call works if it is called from thread created by eventlet.spawn. However if the thread is created with eventlet.spawn_n then threading.current_thread is still broken and returns the ID of the python native thread. The fasteners' ReaderWriterLock depends heavily on threading.current_thread to decide which thread holds a lock and to allow re-entry of that thread. This leads to the situation that multiple threads created from spawn_n could take the same ReaderWriterLock at the same time. The fair internal lock in oslo.concurrency uses ReaderWriterLock and as a result such lock is broken for threads created with spawn_n. Note that this issue was raised with eventlet in [3] when the nova team detected it via a direct usage of ReaderWriterLock in the nova test code. As [3] did not lead to a solution in eventlet nova implemented a nova local fix for the test code in [4]. However now we detected that oslo.concurrency is affected by this issue as well. This patch adds tests to show the scope of the problem. Note that the coverage tox target is changed to explicitly enable native threading otherwise it runs eventlet specific tests in a native environment. Also note that [5] was opened to reintroduce the workaround[1] in fasteners. [1] harlowja/fasteners@467ed75 [2] eventlet/eventlet#172 [3] eventlet/eventlet#731 [4] https://review.opendev.org/c/openstack/nova/+/813114 [5] harlowja/fasteners#96 Related-Bug: #1988311 Change-Id: Ibc193c855b49b95b46ebd2aac82ea89e33f885f0
* Update oslo.concurrency from branch 'master' to 796203c94846d44237d869e3b20a6cd8a3602e36 - Prove that spawn_n with fair lock is broken The fasteners lib in version 0.15.0 removed the threading.current_thread workaround for eventlet[1] because eventlet seemed to fixed the current_thread issues tracked in [2]. However the fix for [2] only fixed half of the problem. The threading.current_thread call works if it is called from thread created by eventlet.spawn. However if the thread is created with eventlet.spawn_n then threading.current_thread is still broken and returns the ID of the python native thread. The fasteners' ReaderWriterLock depends heavily on threading.current_thread to decide which thread holds a lock and to allow re-entry of that thread. This leads to the situation that multiple threads created from spawn_n could take the same ReaderWriterLock at the same time. The fair internal lock in oslo.concurrency uses ReaderWriterLock and as a result such lock is broken for threads created with spawn_n. Note that this issue was raised with eventlet in [3] when the nova team detected it via a direct usage of ReaderWriterLock in the nova test code. As [3] did not lead to a solution in eventlet nova implemented a nova local fix for the test code in [4]. However now we detected that oslo.concurrency is affected by this issue as well. This patch adds tests to show the scope of the problem. Note that the coverage tox target is changed to explicitly enable native threading otherwise it runs eventlet specific tests in a native environment. Also note that [5] was opened to reintroduce the workaround[1] in fasteners. [1] harlowja/fasteners@467ed75 [2] eventlet/eventlet#172 [3] eventlet/eventlet#731 [4] https://review.opendev.org/c/openstack/nova/+/813114 [5] harlowja/fasteners#96 Related-Bug: #1988311 Change-Id: Ibc193c855b49b95b46ebd2aac82ea89e33f885f0
* Update oslo.concurrency from branch 'master' to ee3f73a13379a79282325906787aae7da0f6ac27 - Fix fair internal lock used from eventlet.spawn_n The fasteners lib in version 0.15.0 removed the threading.current_thread workaround for eventlet[1] because eventlet seemed to fixed the current_thread issues tracked in [2]. However the fix for [2] only fixed half of the problem. The threading.current_thread call works if it is called from thread created by eventlet.spawn. However if the thread is created with eventlet.spawn_n then threading.current_thread is still broken and returns the ID of the python native thread. The fasteners' ReaderWriterLock depends heavily on threading.current_thread to decide which thread holds a lock and to allow re-entry of that thread. This leads to the situation that multiple threads created from spawn_n could take the same ReaderWriterLock at the same time. The fair internal lock in oslo.concurrency uses ReaderWriterLock and as a result such lock is broken for threads created with spawn_n. Note that this issue was raised with eventlet in [3] when the nova team detected it via a direct usage of ReaderWriterLock in the nova test code. As [3] did not lead to a solution in eventlet nova implemented a nova local fix for the test code in [4]. However now we detected that oslo.concurrency is affected by this issue as well. This patch restores the workaround that was removed by [1]. Note that a fasteners issue [5] also opened to restore the workaround[1]. [1] harlowja/fasteners@467ed75 [2] eventlet/eventlet#172 [3] eventlet/eventlet#731 [4] https://review.opendev.org/c/openstack/nova/+/813114 [5] harlowja/fasteners#96 Closes-Bug: #1988311 Change-Id: Ia873bcc6b07121c9bd0b94c593567d537b4c1112
The fasteners lib in version 0.15.0 removed the threading.current_thread workaround for eventlet[1] because eventlet seemed to fixed the current_thread issues tracked in [2]. However the fix for [2] only fixed half of the problem. The threading.current_thread call works if it is called from thread created by eventlet.spawn. However if the thread is created with eventlet.spawn_n then threading.current_thread is still broken and returns the ID of the python native thread. The fasteners' ReaderWriterLock depends heavily on threading.current_thread to decide which thread holds a lock and to allow re-entry of that thread. This leads to the situation that multiple threads created from spawn_n could take the same ReaderWriterLock at the same time. The fair internal lock in oslo.concurrency uses ReaderWriterLock and as a result such lock is broken for threads created with spawn_n. Note that this issue was raised with eventlet in [3] when the nova team detected it via a direct usage of ReaderWriterLock in the nova test code. As [3] did not lead to a solution in eventlet nova implemented a nova local fix for the test code in [4]. However now we detected that oslo.concurrency is affected by this issue as well. This patch restores the workaround that was removed by [1]. Note that a fasteners issue [5] also opened to restore the workaround[1]. [1] harlowja/fasteners@467ed75 [2] eventlet/eventlet#172 [3] eventlet/eventlet#731 [4] https://review.opendev.org/c/openstack/nova/+/813114 [5] harlowja/fasteners#96 Closes-Bug: #1988311 Change-Id: Ia873bcc6b07121c9bd0b94c593567d537b4c1112
In 0.15.0 version, removed the workaround code for issue [1] openstack/masakari need these fixes to lock the progress details information during host recovery after waitall method of GreenPool thread is called [2]. [1]: eventlet/eventlet#172 [2]: https://opendev.org/openstack/masakari/src/branch/master/masakari/engine/drivers/taskflow/host_failure.py#L346 Change-Id: Ic6bab5d860d0b44262605bdeb41bc458dbe99dcf
It appears that usage of threading.current_thread() does not change when the current greenthread is different. Was this intentional?
An example that creates a simple latch that uses the current_thread() information that doesn't work when the threading module is used due to this:
When ran using
threading.current_thread()
the following (or similar) is output due to this current_thread update not happening...This is obviously not right (since those are supposed to be different threads).
If instead you comment out the following two lines then this will work as expected:
The text was updated successfully, but these errors were encountered: