Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

process.spawn: Use multiprocessing.Process for returnproc #1249

Merged

Conversation

zmedico
Copy link
Member

@zmedico zmedico commented Feb 3, 2024

Use multiprocessing.Process for returnproc, so that fork will stop being used when python makes "spawn" the default multiprocessing start method.

Continue to use _start_fork when returnproc is not enabled, for backward compatibility. Ultimately, it can be removed at the same time as the returnpid parameter.

Bug: https://bugs.gentoo.org/916566

@zmedico zmedico force-pushed the bug_916566_spawn_multiprocessing_Process branch 3 times, most recently from 3478c19 to 45a3416 Compare February 3, 2024 03:37
@zmedico zmedico marked this pull request as draft February 3, 2024 04:39
lib/portage/process.py Outdated Show resolved Hide resolved
@zmedico zmedico force-pushed the bug_916566_spawn_multiprocessing_Process branch from 45a3416 to d756ce9 Compare February 3, 2024 07:04
@zmedico zmedico marked this pull request as ready for review February 3, 2024 07:04
@zmedico zmedico force-pushed the bug_916566_spawn_multiprocessing_Process branch 4 times, most recently from 56dfffa to d65a1c2 Compare February 3, 2024 19:36
In order to allow callers to manage the lifecycle of fd_pipes
file descriptors, create duplicates for _send_fd_pipes to
close when it has finished sending them.

This fixes bug 916601 in a nice way, allowing commit
3b1234b to be reverted.

Bug: https://bugs.gentoo.org/916601
Signed-off-by: Zac Medico <zmedico@gentoo.org>
This reverts commit 3b1234b,
since ForkProcess now solves the problem by creating temporary
duplicate file descriptors.

Bug: https://bugs.gentoo.org/916601
Signed-off-by: Zac Medico <zmedico@gentoo.org>
@zmedico zmedico force-pushed the bug_916566_spawn_multiprocessing_Process branch 2 times, most recently from fb98250 to a85b175 Compare February 3, 2024 20:48
Use multiprocessing.Process for returnproc, so that
fork will stop being used when python makes "spawn"
the default multiprocessing start method.

Continue to use _start_fork when returnproc is not
enabled, for backward compatibility. Ultimately,
it can be removed at the same time as the returnpid
parameter.

Bug: https://bugs.gentoo.org/916566
Signed-off-by: Zac Medico <zmedico@gentoo.org>
@zmedico zmedico force-pushed the bug_916566_spawn_multiprocessing_Process branch from a85b175 to 305612d Compare February 3, 2024 21:28
@thesamesam
Copy link
Member

btw it might be worth us adding a single impl (3.12 maybe) to CI w/ a hack to use spawn

@zmedico
Copy link
Member Author

zmedico commented Feb 3, 2024

btw it might be worth us adding a single impl (3.12 maybe) to CI w/ a hack to use spawn

Yeah, that's a necessity. I suppose we could use an environment variable to trigger multiprocessing.set_start_method("spawn", force=True) at the beginning of all portage programs. Then there's this little script I use to launch pytest:

import multiprocessing
import sys
import pytest

if __name__ == "__main__":
    multiprocessing.set_start_method("spawn", force=True)
    sys.exit(pytest.console_main())

Maybe it's easier to use as a one-liner:

python -c 'import multiprocessing, sys, pytest; multiprocessing.set_start_method("spawn", force=True); sys.exit(pytest.console_main())'

@gentoo-bot gentoo-bot merged commit 305612d into gentoo:master Feb 3, 2024
11 checks passed
@zmedico zmedico deleted the bug_916566_spawn_multiprocessing_Process branch February 3, 2024 23:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants