[BUG] Setuptools passes too long a command to link.exe, causing [WinError 206] The filename or extension is too long #4177
Comments
Mause
added
bug
Needs Triage
|
Doing a diff on the commands generated by CPython's distutils and the one included with setuptools, it looks like the difference might be caused by the name of the build folder - I guess the question here then is, do we need to migrate away from setuptools? Or is there some way we can split up the link calls? |
|
Hi @Mause thank you very much for reporting this issue. Is this something that can be solved by changing the # powershell
New-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1 -PropertyType DWORD -Forcehttps://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry If not, I belive the best would be adding the workaround with the shortening function directly to the |
This is already enabled - without it, the limit is 320 characters I believe
I'm happy to do this, but arguably this just delays the issue, rather than solving it - is there a way to pass link.exe arguments with a file perhaps? |
The microsoft docs do mention something like "link command files", so in theory yes that would be possible (but I don't know if that would be a solution or the behaviour is just to use the files as if they were directly typed in the terminal... that probably would have to be tested). Other possibilities include I believe that the folks in the pypa/distutils repository would be the best to advise on this. /cc @zooba (for advice) @Mause, one quick question just to make sure we are looking into the correct bug: |
|
The compiler and linker don't support long paths, unfortunately. If the problem is that the overall command line is too long, then a command file is a good option (also helps avoid quoting issues, though as we've mostly got those sorted it probably isn't a big deal anymore). But if the problem is that an individual path exceeds 260 then I don't think there's a way around it yet. I'll prod the team about this again. Most likely they'll suggest moving the build directory to a shorter path (again)... |
|
Thank you very much @zooba for the advice. @Mause, this would mean that the "longest" term option we have available is to use the shortening path workaround as you described. Regarding the path added by setuptools:
But it might be worth checking if there is no other bug causing unnecessary nesting first. |
In my particular case that doesn't matter too much, as the longest path in our codebase is only 113 characters:
I've only eyeballed the paths in question (see my original post for the very long link.exe invocation in question), but I can't see any extra nesting beyond what was there prior in older versions of distutils. It sounds to me like the link command files are a pretty viable option, and can be combined with the short path names to work around the fact that the compiler and linker don't support long paths as yet (though again, this second part doesn't affect us too much, as individually our paths aren't all that long). |
I think the best would be discussing the problem in
This is interesting... so the file paths are actually not getting beyond the limit of 32766 that you get when you change the registry option... Do you think it is because the command line string is too big? But then I think distutils calls the executables using |
Can do
Yes, that's what this issue is. Under the hood on windows, Popen calls CreateProcessW. From the Windows docs:
|
abravalheri
added
upstream
and removed
Needs Triage
setuptools version
setuptools==69.0.3
Python version
Python 3.12
OS
Windows
Additional environment information
This is occuring on Windows build agents on Github Actions, when we updated to start building for Python 3.12 (and hence had to switch to using setuptools' distutils fork). Previously we had SETUPTOOLS_USE_DISTUTILS=stdlib set due to an old numpy issue.
This issue is related in the error code to #3591, but the issue there doesn't seem to be the same
Description
When we switch from SETUPTOOLS_USE_DISTUTILS=stdlib to the default of
local, we started seeing[WinError 206] The filename or extension is too long(though we had to patch distutils to stop it from swallowing this exception and only returningfailed: None)After further investigation, we found that the call to
link.exewas longer than the 32766 characters permitted by Windows, hence causing the above error.For now, we've worked around this by patching distutils to call GetShortPathNameW, which manages to shorten the call by around 12000 characters
Expected behavior
The call to link.exe should be shorter than the maximum allowed by Windows, and compilation should succeed, as it did with the version of distutils includes in < Python 3.12
How to Reproduce
pip install -e .intools/pythonpkgon Windows 10+Output
Command in question:
Details
When patched to dump the stack trace:
The text was updated successfully, but these errors were encountered: