pip install generates file paths > 260 which break on Windows #11231

truekonrads · 2022-07-06T04:57:23Z

Description

NB: This is a cross-bug report with pypa/pipenv#5160. As per @matteius advice I am putting this on pip backlog.

pip can generate file paths during build which are more than 260 characters which breaks on Windows as described in this issue by netleibi/fastchunking#6 (comment) by @netleibi

There are no suitable OS workarounds and therefore I propose that this is a pip bug.

Expected behavior

It should at least warn that long file paths are there and that users should apply a workaround to trim build path prefix using TMPDIR=C:\temp && TEMP=C:\temp or similar
It should generate shorter paths.

pip version

22.1.2

Python version

3.9.7

OS

Windows 10

How to Reproduce

pip install fastchunking

Output

$ pip install fastchunking
Collecting fastchunking
  Downloading fastchunking-0.0.3.tar.gz (24 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting pybindgen
  Using cached PyBindGen-0.22.1-py2.py3-none-any.whl (152 kB)
Using legacy 'setup.py install' for fastchunking, since package 'wheel' is not installed.
Installing collected packages: pybindgen, fastchunking
  Running setup.py install for fastchunking: started
  Running setup.py install for fastchunking: finished with status 'error'
  error: subprocess-exited-with-error

  Running setup.py install for fastchunking did not run successfully.
  exit code: 1

  [50 lines of output]
  Generating file C:\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\build\rabinkarprh.cpp
  running install
  running build
  running build_py
  creating build\lib.win-amd64-3.10
  creating build\lib.win-amd64-3.10\fastchunking
  copying fastchunking\benchmark.py -> build\lib.win-amd64-3.10\fastchunking
  copying fastchunking\test.py -> build\lib.win-amd64-3.10\fastchunking
  copying fastchunking\__init__.py -> build\lib.win-amd64-3.10\fastchunking
  creating build\lib.win-amd64-3.10\lib
  copying lib\rabinkarpgen.py -> build\lib.win-amd64-3.10\lib
  copying lib\rabinkarp_gen.py -> build\lib.win-amd64-3.10\lib
  copying lib\__init__.py -> build\lib.win-amd64-3.10\lib
  running build_ext
  building 'fastchunking._rabinkarprh' extension
  creating build\temp.win-amd64-3.10
  creating build\temp.win-amd64-3.10\Release
  creating build\temp.win-amd64-3.10\Release\Users
  creating build\temp.win-amd64-3.10\Release\Users\matte
  creating build\temp.win-amd64-3.10\Release\Users\matte\AppData
  creating build\temp.win-amd64-3.10\Release\Users\matte\AppData\Local
  creating build\temp.win-amd64-3.10\Release\Users\matte\AppData\Local\Temp
  creating build\temp.win-amd64-3.10\Release\Users\matte\AppData\Local\Temp\pip-install-ii196agq
  creating build\temp.win-amd64-3.10\Release\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7
  creating build\temp.win-amd64-3.10\Release\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\build
  creating build\temp.win-amd64-3.10\Release\lib
  C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Ilib -IC:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\include -IC:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\Include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\include -IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt /EHsc /TpC:\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\build\rabinkarprh.cpp /Fobuild\temp.win-amd64-3.10\Release\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\build\rabinkarprh.obj
  rabinkarprh.cpp
  lib\rabinkarp.h(116): warning C4267: 'initializing': conversion from 'size_t' to 'unsigned int', possible loss of data
  lib\rabinkarp.h(171): warning C4267: 'initializing': conversion from 'size_t' to 'int', possible loss of data
  lib\rabinkarp.h(214): warning C4267: 'initializing': conversion from 'size_t' to 'unsigned int', possible loss of data
  C:\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\lib\characterhash.h(62): warning C4293: '>>': shift count negative or too big, undefined behavior
  C:\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\lib\characterhash.h(55): note: while compiling class template member function 'CharacterHash<uint32,unsigned char>::CharacterHash(hashvaluetype,uint32)'
          with
          [
              hashvaluetype=uint32
          ]
  lib\rabinkarp.h(34): note: see reference to function template instantiation 'CharacterHash<uint32,unsigned char>::CharacterHash(hashvaluetype,uint32)' being compiled
          with
          [
              hashvaluetype=uint32
          ]
  lib\rabinkarp.h(81): note: see reference to class template instantiation 'CharacterHash<uint32,unsigned char>' being compiled
  C:\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\lib\characterhash.h(64): warning C4293: '>>': shift count negative or too big, undefined behavior
  C:\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\lib\characterhash.h(67): warning C4293: '<<': shift count negative or too big, undefined behavior
  C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Ilib -IC:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\include -IC:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\Include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\include -IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt /EHsc /Tplib/rabinkarp.cpp /Fobuild\temp.win-amd64-3.10\Release\lib/rabinkarp.obj
  rabinkarp.cpp
  C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\libs /LIBPATH:C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\PCbuild\amd64 /LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\lib\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\ucrt\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\um\x64 /EXPORT:PyInit__rabinkarprh build\temp.win-amd64-3.10\Release\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\build\rabinkarprh.obj build\temp.win-amd64-3.10\Release\lib/rabinkarp.obj /OUT:build\lib.win-amd64-3.10\fastchunking\_rabinkarprh.cp310-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.10\Release\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\build\_rabinkarprh.cp310-win_amd64.lib
  LINK : fatal error LNK1104: cannot open file 'build\temp.win-amd64-3.10\Release\Users\matte\AppData\Local\Temp\pip-install-ii196agq\fastchunking_0899c9e99e74493f853fd48e3bdd72a7\build\_rabinkarprh.cp310-win_amd64.exp'
  error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX86\\x64\\link.exe' failed with exit code 1104
  [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

Encountered error while trying to install package.

fastchunking

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

Code of Conduct

I agree to follow the PSF Code of Conduct.

matteius · 2022-07-06T05:00:35Z

I imagine the pip developers may not be interested in improving the default behavior for Windows because of this option: https://docs.python.org/3/using/windows.html#removing-the-max-path-limitation

However I note also that because fastchunking invokes C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\bin\HostX86\x64\link.exe it would appear that the max-path limitation cannot be changed and the install continues to fail even with modification of this registry value.

The only way to get pip install fastchunking to work on Windows today is to override the pip cache temp directory to be much shorter than the default.

uranusjr · 2022-07-06T06:09:59Z

Running setup.py install for fastchunking: started

pip does not have control over setup.py install since it’s run inside a subprocess. I’m keeping this open since we should still add some checks during wheel installtion, but that will not resolve this particular issue, which can only be resolved by setuptools.

truekonrads · 2022-07-06T06:15:26Z

@uranusjr Should an issue be raised with setuptools then as well?

uranusjr · 2022-07-06T08:11:14Z

Yes please!

pfmoore · 2022-07-06T09:39:59Z

Presumably this isn't just a Windows issue, Unix also has MAX_PATH and we should handle the two platforms the same. It's just that on Unix (I assume) MAX_PATH is typically bigger, and $TEMP is shorter, so the problem isn't typically triggered there.

notatallshaw · 2022-07-07T16:04:31Z

Presumably this isn't just a Windows issue, Unix also has MAX_PATH and we should handle the two platforms the same. It's just that on Unix (I assume) MAX_PATH is typically bigger, and $TEMP is shorter, so the problem isn't typically triggered there.

POSIX compliant OSes have a PATH_MAX defined in limits.h, the default for Linux systems is 4096 bytes but it does allow for a minimum value of 256 bytes.

Though it should be noted that PATH_MAX on Linux and MAX_PATH on Windows are implemented quite differently and have different edge cases, some further reading if anyone is interested: https://insanecoding.blogspot.com/2007/11/pathmax-simply-isnt.html

pfmoore · 2022-07-07T16:18:59Z

Thanks for the reference (which I haven't read yet, but I will do so at some point). I'll also note as another example of weirdness, that the maximum length of a path is almost certainly filesystem dependent, not OS-dependent. (And the "support long paths" setting on Windows imples that it's might not even be filesystem dependent, but can depend on user configuration as well...)

Anyway, my main point is that we should "fix" this issue in a generic way, using whatever means is available to detect the maximum supported path length, and warning based on that, not based on assumptions about the platform.

notatallshaw · 2022-07-07T16:28:12Z

So fun fact, PATH_MAX and MAX_PATH are both OS level limitations, most popular filesystems have no limitation on the length of the path except for NTFS which is 32,767 Unicode characters.

pfmoore · 2022-07-07T16:39:00Z

Hmm, so on Linux we could check PATH_MAX but on Windows we can't tell (see this SO question which suggests that the setting is per-process and can't be queried via documented APIs).

Given that I'm a strong -1 on producing a warning on paths over 260 characters for Windows systems with long paths enabled, this feels like it's too hard to get right for too little benefit.

I'm willing to be convinced otherwise by a working PR (that doesn't report false positives) though.

matteius · 2022-07-07T16:44:28Z

The other thing to consider on Windows is if the thing you are installing is not a wheel, then it needs to be built and that python process may have long paths enabled but -- if that is going to invoke the Visual Studio linker as some things do, then it is going to fail on the < 260 limit. This bit me last night with Cython, unrelated to the OP, and I squeaked past it by setting my env variable for TMPDIR to /c/tmp/ to work around it, but it was probably a close call. Pip generates a uuid type dir prefix, and then setuptools takes its turn doing similar inside that temp dir, and then if the build assets are too long it may just bust. I personally think Microsoft needs to address that about their linker, but it is problematic.

pfmoore · 2022-07-07T16:51:52Z

Yes, I think this is the point where it likely needs to be handled by setuptools, as they will know the process failed and can potentially report on why. Although of course they may also simply say that it's the linker's issue to give a better error message, as well...

As a strawman example, a linker on any platform that failed on paths over 100 characters long, for no better reason than they hard coded path buffers to that length, would behave in basically the same way as we're seeing here. At some point, we have to assume that the tools we're invoking work properly 🤷

The problem is that the further up the chain you go, the worse the risk of mistakenly guessing that it's a "path too long" issue when it isn't.

truekonrads added S: needs triage Issues/PRs that need to be triaged type: bug A confirmed bug or unintended behavior labels Jul 6, 2022

uranusjr changed the title ~~pip install generates file paths > 260 which break on Windows #5160~~ pip install generates file paths > 260 which break on Windows Jul 6, 2022

matteius mentioned this issue Jul 6, 2022

MSBuild should handle paths longer than MAX_PATH dotnet/msbuild#53

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pip install generates file paths > 260 which break on Windows #11231

pip install generates file paths > 260 which break on Windows #11231

truekonrads commented Jul 6, 2022 •

edited by uranusjr

Loading

matteius commented Jul 6, 2022

uranusjr commented Jul 6, 2022

truekonrads commented Jul 6, 2022

uranusjr commented Jul 6, 2022

pfmoore commented Jul 6, 2022

notatallshaw commented Jul 7, 2022 •

edited

Loading

pfmoore commented Jul 7, 2022

notatallshaw commented Jul 7, 2022 •

edited

Loading

pfmoore commented Jul 7, 2022

matteius commented Jul 7, 2022 •

edited

Loading

pfmoore commented Jul 7, 2022

pip install generates file paths > 260 which break on Windows #11231

pip install generates file paths > 260 which break on Windows #11231

Comments

truekonrads commented Jul 6, 2022 • edited by uranusjr Loading

Description

Expected behavior

pip version

Python version

OS

How to Reproduce

Output

Code of Conduct

matteius commented Jul 6, 2022

uranusjr commented Jul 6, 2022

truekonrads commented Jul 6, 2022

uranusjr commented Jul 6, 2022

pfmoore commented Jul 6, 2022

notatallshaw commented Jul 7, 2022 • edited Loading

pfmoore commented Jul 7, 2022

notatallshaw commented Jul 7, 2022 • edited Loading

pfmoore commented Jul 7, 2022

matteius commented Jul 7, 2022 • edited Loading

pfmoore commented Jul 7, 2022

truekonrads commented Jul 6, 2022 •

edited by uranusjr

Loading

notatallshaw commented Jul 7, 2022 •

edited

Loading

notatallshaw commented Jul 7, 2022 •

edited

Loading

matteius commented Jul 7, 2022 •

edited

Loading