Skip to content

Close write_pipe after forking - #707

Merged
synweap15 merged 1 commit into
Bogdanp:masterfrom
dansimko:close-write-pipe
May 12, 2025
Merged

Close write_pipe after forking#707
synweap15 merged 1 commit into
Bogdanp:masterfrom
dansimko:close-write-pipe

Conversation

@dansimko

@dansimko dansimko commented May 12, 2025

Copy link
Copy Markdown
Contributor

The write_pipe never goes out of scope and so the last process's write_pipe is never closed by the watcher process, which results in a hang on read() from this pipe in log_watcher when the worker process is killed, and blocking all worker processes when their respective write pipe buffers are exhausted.

The write_pipe never goes out of scope and so the last process's
write_pipe is never closed by the watcher process, which results in a
hang on read() from this pipe is log_watcher when the worker process is
killed, blocking all worker processes when their respective write pipe
buffers are exhausted.
@thrix

thrix commented May 12, 2025

Copy link
Copy Markdown

Looks like a common issue with Python multiprocessing:

Before Fix:
Parent: [read_pipe, write_pipe]   ❌ <- still holds write_pipe
Child:              [write_pipe]
EOF never happens!

After Fix:
Parent: [read_pipe] ✅
Child:              [write_pipe]
→ EOF is detected correctly when child closes or exits.

Interestingly, we do not hit it always, but this was hunting our production for quite some time now :) nice find @dansimko

@synweap15

Copy link
Copy Markdown
Collaborator

@dansimko great find. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants