rc.exe randomly not found (race condition?) #2212
Comments
|
I'm afraid I don't have a lot to offer. You've already performed quite a bit of analysis deeper than I could. I did look at the failing kivy build log, and I don't see anything there that strikes me as abnormal. I do observe that There are some recent builds of Setuptools that are attempting to adopt distutils (>=48.x,<49.1). If you want to try building with those, you can see if they're any better (as they'll incorporate many updates to distutils for Python 3.7-3.9). Soon I hope, Setuptools will adopt distutils in the latest release and we can add some debugging code. Let me know if there's more I can do to help you investigate. |
|
Thanks for the info, that helped me get a starting point. For a first step, I copied over I then tested replacing that So, to figure out where there threads are coming from given that there's no obvious thread usage, I added code to detect if spawn is called from multiple threads (https://gist.github.com/matham/a0da3beae05834a3607a8fdc9a6aacd4#file-_msvccompiler-py-L513). Turns out there are almost always two simultaneous calls to spawn from two threads. Following is the stack trace from one example where it detected that another thread concurrently called and e.g. Could this be the issue? If so, is there anything we could do for this, short for replacing |
|
Nice work. Thanks for confirming my suspicion and tracing the threads. I wish I knew what was creating those threads, but at least knowing that there are more than one is confirmation enough.
That's actually my best recommendation -- avoid mutating global state and thread-unsafe operations. In fde70b1, I've patched distutils to avoid this condition. If you install setuptools from this link, that's a preview of the aforementioned commit, you should be able to test this potential fix. Keep in mind that the distutils override feature is disabled by default in 49.1.2, so you'll also want to also set Can you try that in your environment and see if it corrects the issue? |
|
Thanks for fixing this so quickly. I tried with the provided setuptools and there hasn't been any failures after 10 CI runs. Previously it would fail after a few runs, so this seems to indicate that the issue is fixed! |
|
The latest release of Setuptools, v49.1.3 should have this fix (still the |
…0 to version 49.2.0
Alex Henrie (1):
Change exec_module to load_module
Hai Shi (2):
bpo-40275: Use new test.support helper submodules in tests (GH-21151)
bpo-40275: Use new test.support helper submodules in tests (GH-21317)
Hugo (1):
Test Python 3.9-dev
Hugo van Kemenade (1):
Test Python 3.9-dev
Jason R. Coombs (26):
Amend changelog for 48.0 to include more detail about usage expectations. Ref #2230.
Use lowercase 't' for consistency in branding.
Rename logo assets to remove project name and 'logo', which are implied by the context.
Render logo in the readme.
Add banner to main docs page
Remove stale description of packaging.
Add test for spawn when exe is missing. Ref pypa/distutils#3.
Replace OSError with DistutilsExecError. Fixes pypa/distutils#3 and pypa/setuptools#2228 and bpo-41207.
Update changelog. Ref #2228.
Bump version: 49.0.0 → 49.0.1
Move assert outside the context so it actually has its effect.
Remove py2_warn, no longer needed as a SyntaxError is encountered before the warning can be issueed.
bpo-41207 In distutils.spawn, rewrite FileNotFound (GH-21359)
Update changelog.
Bump version: 49.1.0 → 49.1.1
Provide escape hatch for distutils adoption.
Allow opt-in and opt-out of distutils adoption at run time with SETUPTOOLS_USE_DISTUTILS environment variable.
Bump version: 49.1.1 → 49.1.2
Add a simple blank issue so it gets a green button.
Update changelog.
Allow spawn to accept environment. Avoid monkey-patching global state. Closes pypa/setuptools#2212 and closes pypa/distutils#5.
Update changelog.
Bump version: 49.1.2 → 49.1.3
Warn the user when distutils is present to discourage this usage and direct users to the recommended usage. Closes #2230.
Programmatically disable coverage when running on PyPy.
Bump version: 49.1.3 → 49.2.0
Serhiy Storchaka (1):
bpo-41043: Escape literal part of the path for glob(). (GH-20994)
Victor Stinner (1):
bpo-41003: Fix test_copyreg when numpy is installed (GH-20935)
cajhne (1):
Add logo resources
…on 49.2.0
Alex Henrie (1):
Change exec_module to load_module
Hugo (1):
Test Python 3.9-dev
Hugo van Kemenade (1):
Test Python 3.9-dev
Jason R. Coombs (11):
Provide escape hatch for distutils adoption.
Allow opt-in and opt-out of distutils adoption at run time with SETUPTOOLS_USE_DISTUTILS environment variable.
Bump version: 49.1.1 → 49.1.2
Add a simple blank issue so it gets a green button.
Update changelog.
Allow spawn to accept environment. Avoid monkey-patching global state. Closes pypa/setuptools#2212 and closes pypa/distutils#5.
Update changelog.
Bump version: 49.1.2 → 49.1.3
Warn the user when distutils is present to discourage this usage and direct users to the recommended usage. Closes #2230.
Programmatically disable coverage when running on PyPy.
Bump version: 49.1.3 → 49.2.0
I've run into a issue on GitHub Actions where in like a small percent of the VMs spinned up, pip will fail with the error
LINK : fatal error LNK1158: cannot run 'rc.exe':Simply re-running the actions normally solves the issue. Here's an example of a good run https://github.com/kivy/kivy/runs/791883878?check_suite_focus=true, and a bad run https://github.com/kivy/kivy/runs/791983410?check_suite_focus=true.
My first thought was that this a GitHub Action environment issue, so I opened a bug report there: actions/runner-images#1087. However, after some investigation, it seemed this is more likely a bug in setuptools. The reason is that using process monitor to check all instantiations of
link.exewe looked at the environmentalPATHvariable of the process when it started. And in the failing examples, python would calllink.exemultiple times for each cython generated c-file, but some of these would be missing all the visual studio compiler paths from itsPATH.E.g. process monitor shows this during the compile step:
.
But the first one's (6396) PATH is:
while the second one's (6520) is much shorter and is missing 18 paths (Doesn't contain C:\Program Files (x86)\Windows Kits\10\bin\10.0.19041.0\x86 with rc.exe):
You can inspect a pared down process monitor log here (you need to open it in process monitor):
logfile.zip or get the full 100+MB file from this action: https://github.com/kivy/kivy/actions/runs/145338811.
procmon-3.8-x64is the good log,procmon-3.7-x64is the bad log.As far as I'm aware, the only path type stuff we manipulate is with the following when some package is imported. But I fail to see how this could cause it:
The question is, why randomly within one
python -m pip install -e .command does somelink.exeget created without the proper visual studio paths in its environment? It seems like it must be some kind of race condition where in some cases, the compiler gets called with the wrong env?The text was updated successfully, but these errors were encountered: