Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pytest-xdist --forked can break on darwin #194290

Closed
tjni opened this issue Oct 3, 2022 · 1 comment
Closed

pytest-xdist --forked can break on darwin #194290

tjni opened this issue Oct 3, 2022 · 1 comment
Labels
0.kind: build failure 6.topic: python
Projects
22.11 Blockers

Comments

@tjni
Copy link
Contributor

tjni commented Oct 3, 2022

Steps To Reproduce

nix build .#python310Packages.poetry

Build log

Segmentation fault when running tests on Darwin (at least, on aarch64-darwin):

objc[22085]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.

with stack trace:

Current thread 0x00000001007a4580 (most recent call first):
  File "/nix/store/w4agbx5b17d57jij2i4klv67r8qnl6g3-python3-3.10.7/lib/python3.10/urllib/request.py", line 2625 in proxy_bypass_macosx_sysconf
  File "/nix/store/w4agbx5b17d57jij2i4klv67r8qnl6g3-python3-3.10.7/lib/python3.10/urllib/request.py", line 2649 in proxy_bypass
  File "/nix/store/xbw8prirwh78v84fi8i5yfrg970l4a97-python3.10-requests-2.28.1/lib/python3.10/site-packages/requests/utils.py", line 809 in shou
ld_bypass_proxies
  File "/nix/store/xbw8prirwh78v84fi8i5yfrg970l4a97-python3.10-requests-2.28.1/lib/python3.10/site-packages/requests/utils.py", line 825 in get_
environ_proxies
  File "/nix/store/xbw8prirwh78v84fi8i5yfrg970l4a97-python3.10-requests-2.28.1/lib/python3.10/site-packages/requests/sessions.py", line 759 in m
erge_environment_settings
  File "/nix/store/pq57z1i58f5hi23rk61lcrgbds5hh324-python3.10-poetry-1.2.0/lib/python3.10/site-packages/poetry/utils/authenticator.py", line 20
9 in request
  File "/nix/store/pq57z1i58f5hi23rk61lcrgbds5hh324-python3.10-poetry-1.2.0/lib/python3.10/site-packages/poetry/utils/authenticator.py", line 24
6 in get
  File "/nix/store/pq57z1i58f5hi23rk61lcrgbds5hh324-python3.10-poetry-1.2.0/lib/python3.10/site-packages/poetry/repositories/http.py", line 267
in _get_response
  File "/nix/store/pq57z1i58f5hi23rk61lcrgbds5hh324-python3.10-poetry-1.2.0/lib/python3.10/site-packages/poetry/repositories/legacy_repository.p
y", line 137 in _get_page
  File "/nix/store/pq57z1i58f5hi23rk61lcrgbds5hh324-python3.10-poetry-1.2.0/lib/python3.10/site-packages/poetry/repositories/legacy_repository.p
y", line 63 in find_links_for_package
  File "/nix/store/pq57z1i58f5hi23rk61lcrgbds5hh324-python3.10-poetry-1.2.0/lib/python3.10/site-packages/poetry/installation/chooser.py", line 1
19 in _get_links
  File "/nix/store/pq57z1i58f5hi23rk61lcrgbds5hh324-python3.10-poetry-1.2.0/lib/python3.10/site-packages/poetry/installation/chooser.py", line 7
7 in choose_for
  File "/private/tmp/nix-build-python3.10-poetry-1.2.0.drv-0/source/tests/installation/test_chooser.py", line 184 in test_chooser_no_binary_poli
cy

Additional context

From OSX 10.12+, functions that access ObjC libraries cannot be called after forking. In this case, the poetry tests are running into https://bugs.python.org/issue30385#msg293958. This was enough of a problem that Python 3.8 switched to using "spawn" instead of "fork" when using the multiprocess library (python/cpython#77906), but this doesn't impact pytest-forked since it explicitly calls os.fork.

Suggestion: don't use "--forked" (at least on Darwin)

Notify maintainers

@Artturin @dotlambda @mweinelt
@mweinelt mweinelt added this to To do in 22.11 Blockers via automation Oct 3, 2022
@mweinelt
Copy link
Member

mweinelt commented Oct 4, 2022 •
edited
Loading

Ah, splendid. I remember this issue from our python setup at work.

The --forked argument is configured through the pytest-xdist setup hook, which was introduced in march 2022 (50402f8).

pytestXdistHook() {
    pytestFlagsArray+=(
        "--numprocesses=$NIX_BUILD_CORES"
        "--forked"
    )
}

[...]

The question is whether we just allow unsafe forking for the pytest-xdist setup hook, or find a way to not pass --forked on darwin. I don't think we're willing to give it up altogether. Wondering if matching on $OSTYPE would be appropriate.

@mweinelt mweinelt mentioned this issue Oct 4, 2022
Merged
13 tasks
mweinelt added a commit to mweinelt/nixpkgs that referenced this issue Oct 4, 2022
@mweinelt
python3Packages.pytest-xdist: Disable fork safety on darwin
2ba40b5 
We run pytest with `--forked` in nixpkgs, to reduce side effects that
can occur when multiple tests mutate their environment in incompatible
ways.

Forking on macOS 10.13 and later is unsafe when an application does work
between calls to fork() and its followup exec(). This may lead to
crashes when calls into the Objective-C runtime are issued, which will
in turn coredump the Python interpreter.

One good reproducer for this scenario is when the urllib module tries
to lookup proxy configurations in `urllib.request.getproxies()` through
`get_proxies_macos_sysconf` into the native `_scproxy` module.

This is a class of issues that is of course not limited to the urllib
module. The general recommendation is to use `spawn` instead of `fork`,
but we don't have any influence on upstream developers to do one or the
other.

One often cited workaround would be to disable fork safety entirely on
calls to `initialize()`, which is probably a better solution than
running without multithreading (slow) or without the `--forked` (prone
to side effects) mode.

This currently happens on aarch64-linux only, where we use more recent
11.0 SDK version, while x86_64-darwin has been stuck on 10.12 for a
while now.

python/cpython#77906 (comment)
http://www.sealiesoftware.com/blog/archive/2017/6/5/Objective-C_and_fork_in_macOS_1013.html

Closes: NixOS#194290
@tjni tjni closed this as completed Oct 5, 2022
22.11 Blockers automation moved this from To do to Done Oct 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
0.kind: build failure 6.topic: python
Projects
No open projects
22.11 Blockers
Done
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mweinelt @tjni