Unable to install package on Windows: "EnvironmentError: No such file or directory" when a node_modules exists in a subdirectory

[ErikBjare]

Environment

• pip version: 19.0.2
• Python version: Python 3.5/3.6
• OS: Windows

You can see the issue being raised at the end of this Travis build: https://travis-ci.org/ActivityWatch/activitywatch/jobs/495639762#L493

Description

I'm working on trying to get my Python project aw-server (a component of ActivityWatch) to build on Windows using Travis. In doing so, I stumbled upon an old issue we've had before when building on Windows using Appveyor which required us to downgrade pip to 9.0.3 (see this old issue: ActivityWatch/activitywatch#195).

I'm basically trying to install a Python package that has a subfolder (git submodule) containing a node_modules directory, which seems to contain some files that make pip blow up when trying to install. The previous workaround we used (when using Appveyor) was a simple pip downgrade (also mentioned in ActivityWatch/activitywatch#195). Now I'd rather not downgrade pip any longer and have been unable to find anyone else having a similar issue, so I'm asking if anybody here knows what could be going wrong.

The PR with the changes, for those interested, can be found here: ActivityWatch/activitywatch#274

Expected behavior

I expected the package to install correctly, as it does when using pip 9.0.3.

How to Reproduce

Try to install aw-server on Windows (with the aw-webui files built underneath) following the same process as when Travis CI tries to build aw-server.

Output

Copied from the Travis CI build linked above. Formatted for readability.

pip3 install . -r requirements.txt --upgrade
Processing c:\users\travis\build\activitywatch\activitywatch\aw-server
Could not install packages due to an EnvironmentError: [
    ('C:\\Users\\travis\\build\\ActivityWatch\\activitywatch\\aw-server\\aw-webui\\node_modules\\.cache\\uglifyjs-webpack-plugin\\content-v2\\sha512\\13\\a5\\5ec76b20a6485047f8bea1d98ae54f57eabd74c5a4b36b62343dc3e9a43299f0d8c2c084f79a3c4f3e3c4799eb2f4b3652eb240ebe0acf432f28a32fa3be', 
     'C:\\Users\\travis\\AppData\\Local\\Temp\\pip-req-build-pe12a3vh\\aw-webui\\node_modules\\.cache\\uglifyjs-webpack-plugin\\content-v2\\sha512\\13\\a5\\5ec76b20a6485047f8bea1d98ae54f57eabd74c5a4b36b62343dc3e9a43299f0d8c2c084f79a3c4f3e3c4799eb2f4b3652eb240ebe0acf432f28a32fa3be', 
     "[Errno 2] No such file or directory: 'C:\\\\Users\\\\travis\\\\build\\\\ActivityWatch\\\\activitywatch\\\\aw-server\\\\aw-webui\\\\node_modules\\\\.cache\\\\uglifyjs-webpack-plugin\\\\content-v2\\\\sha512\\\\13\\\\a5\\\\5ec76b20a6485047f8bea1d98ae54f57eabd74c5a4b36b62343dc3e9a43299f0d8c2c084f79a3c4f3e3c4799eb2f4b3652eb240ebe0acf432f28a32fa3be'"), 
    ('C:\\Users\\travis\\build\\ActivityWatch\\activitywatch\\aw-server\\aw-webui\\node_modules\\.cache\\uglifyjs-webpack-plugin\\content-v2\\sha512\\56\\b2\\9d1eb73bce24568f732784705e0fe0ac9b26d9f9775a9078bf688cb5996970f3c565520e1da220d3df7b19d2822f80e50225e2865b25879a9346151436cf', 
     'C:\\Users\\travis\\AppData\\Local\\Temp\\pip-req-build-pe12a3vh\\aw-webui\\node_modules\\.cache\\uglifyjs-webpack-plugin\\content-v2\\sha512\\56\\b2\\9d1eb73bce24568f732784705e0fe0ac9b26d9f9775a9078bf688cb5996970f3c565520e1da220d3df7b19d2822f80e50225e2865b25879a9346151436cf', 
     "[Errno 2] No such file or directory: 'C:\\\\Users\\\\travis\\\\build\\\\ActivityWatch\\\\activitywatch\\\\aw-server\\\\aw-webui\\\\node_modules\\\\.cache\\\\uglifyjs-webpack-plugin\\\\content-v2\\\\sha512\\\\56\\\\b2\\\\9d1eb73bce24568f732784705e0fe0ac9b26d9f9775a9078bf688cb5996970f3c565520e1da220d3df7b19d2822f80e50225e2865b25879a9346151436cf'")
]

Additional comments

I'd be happy to find a workaround other than downgrading pip, maybe there is a way to tell pip to ignore the problematic files?

[ErikBjare]

This issue is no longer of interest to me as I found a satisfactory workaround (removing the files as a step in our CI). Still surprised that no one has seen this before though.

[demianzhang]

Hi @ErikBjare , I met the same problem, how do you fix it now? Removing the files or change the pip version? Thanks

[ErikBjare]

@demianzhang I removed the files before running pip. You could downgrade pip to 9.0.3 but I wouldn't recommend it.

[demianzhang]

Thanks for your reply.

[ErikBjare]

This keeps happening to me, really annoying.

[chrahunt]

This looks like a case of the files exceeding the 260 character max path limit and is probably not related to node_modules specifically. There are a few options:

1. You can see here for a workaround to reduce the length of the base directory used for the build in Travis.
2. Enable long paths:
   1. Only works for Python 3.6 and above (see bpo-27731)
   2. Only works for supported versions of Windows (see Microsoft and Travis page - the Windows Server version is higher, but honestly I don't know if they are comparable values).
   3. Requires Long path support to be enabled (i.e. set HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem@LongPathsEnabled to 1)

I checked and it looks like the long path support also works for the dynamically-created entrypoints (e.g. pip.exe) since internally just just invoke python.exe - so if you follow through and are able to configure your Travis environment accordingly, then you should just be able to pip install directly.

[chrahunt]

I hope the above helped. I'll close this now since it seems like it isn't an issue with pip, per se. Please let us know if you're still experiencing any issues. Thanks!