Skip to content

Fix threading monkey patching for Python 3.13 fork handling#1044

Closed
SeanMooney wants to merge 1 commit into
eventlet:masterfrom
SeanMooney:bug/1030
Closed

Fix threading monkey patching for Python 3.13 fork handling#1044
SeanMooney wants to merge 1 commit into
eventlet:masterfrom
SeanMooney:bug/1030

Conversation

@SeanMooney

@SeanMooney SeanMooney commented Jun 7, 2025

Copy link
Copy Markdown

Explanation of the fix:

  1. The issue occurred because Python 3.13 changed how threads are reinitialized
    after forking
  2. When threading._after_fork() attempts to rebuild thread objects, it calls
    _make_thread_handle()
  3. The previous implementation contained a strict assertion ident ==
    get_ident(greenthread)
  4. This assertion would fail because after forking:
    • The thread identities change
    • Green thread mapping isn't preserved across forks
    • The main thread recreation requires custom handling
  5. The fix modifies the conditional to:
    • Only perform identity verification when NOT in a fork scenario
    • Return a safe "done" thread handle during fork situations
    • Handle cases where no current greenlet exists
  6. This maintains the integrity of eventlet's thread emulation while accounting for
    Python 3.13's revised fork behavior

Note i have created this patch using aider, a terminal based AI coding tool
with my personal open-router key to access the deepseak-r1 and Gemini
flash LLMs.

deepseak drove the analysis of the the trackback and code base and
gemini applied the proposed changes to the files.

Closes #1030

This resolves an assertion failure in threading._after_fork() when using
monkey-patched threads on Python 3.13. The root cause was the stricter
thread identity verification in _make_thread_handle() that didn't account
for how Python 3.13 reinitializes threads after forking.

Key changes:
1. Replaced the problematic identity assertion with a conditional check
2. Return a safe "done" thread handle during fork detection
3. Add graceful handling for cases with no current greenlet

The new implementation:
- Maintains thread safety while preventing crashes during forking
- Works with Python 3.13's revised thread initialization after fork
- Preserves backwards compatibility with older Python versions
- Allows stdlib's _after_fork() to properly recreate thread objects

Closes eventlet#1030
Assisted-By: aider (openrouter/deepseek/deepseek-r1-0528:free) <noreply@aider.chat>
Assisted-By: aider (openrouter/google/gemini-2.0-flash-exp:free) <noreply@aider.chat>
Signed-off-by: Sean Mooney <work@seanmooney.info>
@SeanMooney

Copy link
Copy Markdown
Author

not i have only tested this by running eventlet's unit tests and the reproducer under python 3.13
both pass but that by itself does not prove this is a complete and correct fix.

Comment thread eventlet/green/thread.py
return _ThreadHandle()
except greenlet.error:
# Handle cases with no current greenlet
return _ThreadHandle()

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thinking about this more. this will fix the reproducer script but im not convinced its correct. with that said i am hoping to find time to actully deploy with debian 13 or ubuntu 25.04 to try and repoduce the poblems in the next week or so.

this might work but without actully testing it with openstack and seing if it fixes the porblem in a real end to end usecase it s hard to determin if this really is the direction to take.

the repoducer uses the _at_fork hook to demonstrate teh probelm but that may have biased the soltuion so we may need other repoducers too to be able to fully determin why this failes the way it id.

@itamarst

itamarst commented Jun 9, 2025

Copy link
Copy Markdown
Contributor

So here is the thing: fork() without exec() is fundamentally broken. See e.g. writeup I made about multiprocessing: https://pythonspeed.com/articles/python-multiprocessing/ - I have successfully convinced CPython devs to switch multiprocessing to not use fork() backend as of Python 3.14.

If you are seeing problems with fork() you should stop using fork()`. Even if we "fix" eventlet there will be other mysterious intermittent bugs.

So separate from this PR I would treat any use of fork() on your side as something that needs to be ripped out ASAP. It's just broken. There is no way to make it not broken.

@itamarst

itamarst commented Jun 9, 2025

Copy link
Copy Markdown
Contributor

I am inclined to say we shouldn't accept anything that is intended to fix fork() without a lot of thought and understanding of impacts.

Given this was generated by an LLM I am also inclined to reject it without even spending anytime thinking about it. Eventlet is a very brittle system and changes we've made have had unexpected consequences, I would like to actually have code written by an entity capable of having some understanding of state.

@SeanMooney

Copy link
Copy Markdown
Author

the repoducer is form #1030 is

import eventlet

eventlet.monkey_patch()

import threading

def background_task():
    threading._after_fork()

t = threading.Thread(target=background_task)
t.start()
t.join()

that is not actully forking it just show that the way we are doing the tread identity is broken

i think that repoducer may have partly lead my ai analasys in the wrong direction and we need more examples of the failrue that do not invovle calling _after_fork().

with that said oslo service is callifn fork in teh eventlet mode when there are multiple workers

https://github.com/openstack/oslo.service/blob/226fe87b907bfa5ec6a20fbff821c89313e61649/oslo_service/backend/eventlet/service.py#L476

so until the eventlet backend of oslo.service is updated to use spawn this is still a problem we need to adress.

the right fix is undobable to update how the eventlest backend of oslo service work to not use fork, but dont discount this as jsut a llm generated patch.

i spent 3-4 hours working with several different tools to try and isolate a potential cause for the issue. and i explicitly called out that while this passes all of eventlets test, i was hoping to repoduce the actual issue in a real deployment to confirm if this actully adress the probelm before we go any future. with the patch

@itamarst

itamarst commented Jun 10, 2025

Copy link
Copy Markdown
Contributor

I am discounting this because it was written by an LLM, specifically because LLMs aren't by their nature going to be good at modeling massively stateful systems, especially ones with hideous hidden state like this one. They can look at patterns in text. In some situations that's good enough.

Here's the hideous mess we're dealing with here, though:

  1. There are three copies of threading.py imported. The LLM has no notion of this.
  2. Each one registers a fork post handler.
  3. The fork() then doubles this so there are six copies of threading.py in memory!
  4. The whole purpose of the threading.py post-fork handler is to remove threads, because threads don't survive fork.
  5. In this case we're monkeypatching some copies of threading.py (another thing that the pattern recognition in the LLM can't really deal with). The monkeypatching means that threads are actually greenlets, which do survive fork.
  6. That means the post-fork threading.py stdlib thing is wrong and unnecessary for monkeypatched modules, but not other modules.

A plausible solution is therefore to disable the stdlib threading.py post-fork handler in some limited cases. I have this working in a branch, so will close this and submit once I've written a bunch of tests.

@itamarst itamarst closed this Jun 10, 2025
@SeanMooney

Copy link
Copy Markdown
Author

thanks for taking the time to explain this and how you intend to proceed.
i was partyly creating this to better understand the nature of the problem and to try and discover some of the context you provided.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

os.fork() doesn't work with Eventlet on Python 3.13

2 participants