Cannot install packages with long filenames

[minrk]

I'm submitting a...

• bug report
• feature request

Current Behavior

Cannot install nodejs on Windows due to long paths.

nodejs produces extremely deeply nested node_modules/pkg/node_modules/pkg/..., making it pretty easy to readily exceeding the Windows max path limit. nodejs itself doesn't seem have any issues with these long paths due to using modern Windows APIs, but Python seems to have issues.

This is related to #2452, but there seem to be other places (install) where UNC path prefixes or similar patches may be needed.

Reported on the nodejs conda package: conda-forge/nodejs-feedstock#12

(C:\Users\sarathod\AppData\Local\Continuum\anaconda3) C:\Users\sarathod>conda install -c conda-forge nodejs
Fetching package metadata ...............
Solving package specifications: .

Package plan for installation in environment C:\Users\sarathod\AppData\Local\Continuum\anaconda3:

The following NEW packages will be INSTALLED:

    nodejs: 9.9.0-0 conda-forge

Proceed ([y]/n)? y
CondaError: FileNotFoundError(2, 'The filename or extension is too long')
CondaError: FileNotFoundError(2, 'The filename or extension is too long')
CondaError: FileNotFoundError(2, 'The filename or extension is too long')

Steps to Reproduce

On Windows:

conda install -c conda-forge nodejs==9.9

Expected Behavior

package should install without errors

[mingwandroid]

Closing, please fill in the details asked for in the issue template.

---

EDIT: I (@kalefranz) am overtaking this comment to pull out action items discussed below.

• Include long path enabling in conda init.
• Double bonus if it could be in the error message from conda itself when install fails due to long path errors. It'd be great if Conda could detect this particular problem and give a better error message, e.g. "The system has hit the maximum file-path length for this version of Windows".
• Better long paths FAQ, documentation on limitations, etc.

[jakirkham]

Not following. Seems like all of the info was filled out, no? Maybe a more specific request would help.

[kalefranz]

One part in particular that’s missing is conda info. I had initial questions that would have answered.

[jakirkham]

This report ( conda-forge/nodejs-feedstock#33 ) does supply conda info with conda version 4.5.1 and they are running into the same issue.

[lebedov]

I reported conda-forge/nodejs-feedstock#33; the problem arose while trying to install nodejs 9.11.0-0 from conda-forge using conda 4.5.1 and python 3.6.5 on Win7. Output of conda info follows:

     active environment : None
       user config file : C:\Users\lebedov\DOCUME~1\MOBAXT~1\home\.condarc
 populated config files : C:\Users\lebedov\DOCUME~1\MOBAXT~1\home\.condarc
          conda version : 4.5.1
    conda-build version : 3.8.1
         python version : 3.6.5.final.0
       base environment : C:\Users\lebedov\AppData\Local\Continuum\anaconda3  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/win-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/win-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/pro/win-64
                          https://repo.anaconda.com/pkgs/pro/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\lebedov\AppData\Local\Continuum\anaconda3\pkgs
                          C:\Users\lebedov\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\lebedov\AppData\Local\Continuum\anaconda3\envs
                          C:\Users\lebedov\AppData\Local\conda\conda\envs
                          C:\Users\lebedov\DOCUME~1\MOBAXT~1\home\.conda\envs
               platform : win-64
             user-agent : conda/4.5.1 requests/2.18.4 CPython/3.6.5 Windows/7 Windows/6.1.7601
          administrator : True
             netrc file : None
           offline mode : False

[minrk]

Can we get this reopened? All requested info has been provided.

[kalefranz]

I’m ok re-opening. However fixing this is a non-trivial project that will have to be stacked against other priorities. In the meantime, users encountering this problem can add something like

pkgs_dirs:
  - c:\conda-pkgs

to their condarc file.

[mingwandroid]

Sure but I should note that fixing it requires asking the end user to set a registery key and that's going to be a lot of work.

Also allowing conda to install to paths requiring >260 characters will break a whole load of software with inbuilt assumptions of a maximum requirement of 260 characters, some of which testing would catch allowing us to fix but some of it not.

These unfixed ones can in turn lead to stack buffer overflows so it's not something that can be just done and forgotten about (any time a function assumes that the calling function had already checked the 260 limit which would no longer always be the case for example).

Regardless, a registery key with this option could be implemented into constructor and PRs to do that would be welcomed.

[kalefranz]

I could also add the registry key to ‘conda init’.

…

Sent from my iPhone

On May 20, 2018, at 3:08 PM, Ray Donnelly ***@***.***> wrote: Sure but I should not that fixing it requires asking the end user to set a registery key and that's going to be a lot of work. Also allowing conda to install to paths requiring >260 characters will break a whole load of software with inbuilt assumptions of a maximum requirement of 260 characters, some of which testing would catch allowing us to fix but some of it not. These unfixed ones can in turn lead to stack buffer overflows so it's not something that can be just done and forgotten about. Regardless, a registery key with this option could be implemented into constructor and PRs to do that would be welcomed. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or mute the thread.

[mingwandroid]

I think it needs to be done at the Administrator level. Official CPython binary releases do this in their install process on Windows.

[jakirkham]

Thanks all.

FWIW 3 issues were filed against the nodejs feedstock on this exact same issue with at least 8 people claiming to be affected by this problem. 2 of these issues were filed in the past 2 months. The older one (filed late last year) has had activity within the past week. Expect this is a small portion of the actual number of people affected by this problem. Most of whom likely switched to using Node.js some other way and we will likely never know. As this is a common path users are going to take when working with JavaScript in Conda, it seems worthwhile to solve.

Guessing by registry key change, you are referring to something like this, correct?

[mingwandroid]

Indeed. And as you can see it's win 10 only so NSIS UI would be to be done for that unless conda init will be presenting UIs around UAC and the like?

But let's not rush into another noarch: python scenario here where a feature gets promoted despite considerable issues in the wider ecosystem in this case regarding security concerns being resolved first.

And I also believe this is being or had been fixed in newer node releases and we're mostly editing for an LTS release to fix it anyway?

[jakirkham]

Are there solutions for older versions of Windows?

Not sure I follow the noarch: python comment. In any event, only trying to point out this is a significant pain point that is in need of a solution.

Recent reports include Node.js 9.11.0, which was released nearly two months ago.

[mingwandroid]

No solutions I'm aware of no. We tried ages ago to use UNC paths in conda but Windows file locking broke badly.

The noarch: python comment is a reference to the complications that adding a seemingly 'simple' feature to conda caused a great amount of strife in the ecosystem around constructor, testing and differences between how conda-forge and defaults build their packages and I'm keen for this not to happen again when people start using deep install paths and their software doesn't work in there while we're busy advertising that 'long paths work now!' or some other simplification of reality.

[minrk]

My understanding of this being fixed in node is that I believe node and npm themselves no longer have a problem with extremely deep paths on Windows, and continue to routinely make paths greater than 260 characters and that won't change. There was an improvement in path length when npm started deduping packages by default, resulting in a slightly reduced path length, but then shortly after it also increased its dependencies essentially negating that improvement. That was always a partial fix that would only work for some packages.

It's AOK for me if this starts out as a documentation I can point to, even if it's a manual fix requiring admin permissions for some folks. Of course, true support for long paths would be ideal, but a clear "You need to do this..." link would be a great start. Double bonus if it could be in the error message from conda itself when install fails due to long path errors.

[kalefranz]

For binary file prefix replacement, we have to be careful that we're not trying to shove in a path that's longer than the binary placeholder. It's been a while since I looked at this logic, so I created #7324 as a reminder. Prefix replacement in text files isn't as much of an issue.

[SeaDude]

@kalefranz ,

Host: Win10
Guest: VirtualBox of Win10
Goal: To get past the Jupyter Lab 404 error by installing nodejs
Steps:

• Was stuck at nodejs download 77% with a fail of The filename or extension is too long
• Added your recommended:

pkgs_dirs:
  - c:\conda-pkgs

to my condarc file

• This got me past the 77% download BUT download still failed with:

Preparing transaction: done
Verifying transaction: done
Executing transaction: failed
ERROR conda.core.link:_execute(502): An error occurred while installing package 'conda-forge::nodejs-9.11.1-0'.
FileNotFoundError(2, 'The filename or extension is too long')
Attempting to roll back.

Rolling back transaction: done

FileNotFoundError(2, 'The filename or extension is too long')

Not sure where to go from here.

[SeaDude]

Hacked a way through...

• Launched anaconda-navigator
• Clicked "Environments"
• Selected "Not Installed" from le dropdown
• Entered "nodejs" in search box
• Selected it to install (NOTE: It is still installing nodejs 8.9
• After installation, clicked "Jupyter Lab" from Navigator and ran jupyter lab from cli without issue.

May this help someone else too 👍 !

[kalefranz]

@SeaDude

After that configuration change, try

conda create -p c:\node-env -c conda-forge nodejs

[SeaDude]

Seems to "work" though I have no idea what I just did!

(base) C:\Users\XXXX>conda create -p c:\node-env -c conda-forge nodejs
Solving environment: done

## Package Plan ##

  environment location: c:\node-env

  added / updated specs:
    - nodejs


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    vs2015_runtime-14.0.25420  |                0         1.9 MB  conda-forge

The following NEW packages will be INSTALLED:

    nodejs:         9.11.1-0     conda-forge
    vs2015_runtime: 14.0.25420-0 conda-forge

Proceed ([y]/n)? y


Downloading and Extracting Packages
vs2015_runtime-14.0. |  1.9 MB | ############################################################################## | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate c:\node-env
#
# To deactivate an active environment, use
#
#     $ conda deactivate

anaconda-navigator still shows a nodejs version of 8.9

[kalefranz]

You successfully created an environment located at c:\node-env containing nodejs. To use it, run

conda activate c:\node-env

in the Anaconda Prompt.

[ghost]

I fix it to enable Win32 long paths. As windows10 pro, you can open Local Group Policy Editor, and open item Computer Configuration -> Administrative Templates -> System -> Filesystem -> Enable Win32 long paths and click it to make it enable. But I have got the information it's just support windows10, I don't know it will or will not work in Windows8 or old.

[pgunn]

I recently ran into this issue, which is much worse in default configurations of most CI systems because they want to work out of the checkout directory, which can already have a fairly long path. The error message wasn't particularly helpful ; it just gave me a

FileNotFoundError(2, 'No such file or directory')

during an environment build (at the "Executing transaction" stage). Fortunately, I've seen this kinda thing before, but if possible it'd be great if Conda could detect this particular problem and give a better error message, e.g. "The system has hit the maximum file-path length for this version of Windows". This would help people who are not aware of these limits on Windows as they have a longer and more fragile path to understanding.

[Proteusiq]

I can confirm that setting LongPathsEnabled from 0 to 1 solved this issue on Windows 10 64bits
cmd -> regedit -> HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem -> LongPathsEnabled -> 1

[github-actions]

Hi there, thank you for your contribution to Conda!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

1. Verify that you can still reproduce the issue in the latest version of Conda

2. Comment that the issue is still reproducible and include:

• What version of Conda you reproduced the issue on
• What OS and version you reproduced the issue on
• What steps you followed to reproduce the issue

3. It would also be helpful to have the output of the following commands available:

• conda info
• conda config --show-sources
• conda list --show-channel-urls

NOTE: If this issue was closed prematurely, please leave a comment and we will gladly reopen the issue.

In case this issue was originally about a project that is covered by the Anaconda issue tracker (e.g. Anaconda, Miniconda, packages built by Anaconda, Inc. like Anaconda Navigator etc), please reopen the issue there again.

Thanks!

[jakirkham]

IIRC Node changed behavior to avoid this issue.

That said, we still see this issue crop up periodically.

[cleebp]

There are still reports of the max path length limit being hit when installing tensorflow builds in this stale issue here: #10767 This doesn't seem to be getting fixed from the tensorflow side as they still nest files deep into Lib\site-packages\tensorflow\include\external\cudnn_frontend_archive\_virtual_includes\cudnn_frontend\third_party\cudnn_frontend\include\contrib\nlohmann\json\

Setting the registry key to enable long paths enables existing tensorflow conda packages to be installed and environments to clone; however, building the tensorflow package I still hit a CondaVerificationError error even with this registry key enabled. To get around this I've setup a shortened link off C:\ to get a CONDA_PREFIX of C:\c to build successfully, but it's strange the registry key doesn't resolve this build issue.

[D-Ogi]

Execute both as admin:
Set-ItemProperty -Path HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem -Name LongPathsEnabled -Value 1;
git config --system core.longpaths true

The second command specifically solves the issue related to building with pip.

[cleebp]

Execute both as admin: Set-ItemProperty -Path HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem -Name LongPathsEnabled -Value 1;

This is how we tell our windows users to setup their systems to install tensorflow builds when built with long path names enabled. It works but some users can't get their admins to enable long paths which makes the package uninstallable. Right now we are documenting this as a known issue for deep learning using conda with tensorflow.

For our daily Windows CI test machine we set the following at the beginning of machine setup in order to install deep learning packages with long path names:

REG ADD "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem" /v LongPathsEnabled /t REG_DWORD /d 1 /f

[asmeurer]

Since I'm gathering from this that most people are solving this issue on the package side, is there a de facto path length limit that packages are expected to adhere to to avoid seeing this problem, at least for most users?

[scdub]

@asmeurer For most users, the filename in the package cache will be the limiting step, as that includes the full package name, and package names with their suffixes and hashes tend to be longer than the environment names themselves. Here's a worked example showing how long the username of a Windows user could be for a pkg_dir set to one of the default locations, %LOCALAPPDATA%\conda\conda\pkgs:

max_path = 260
user_base = "C:\\Users\\"

pkgs_subdir = "\\AppData\\Local\\conda\\conda\\pkgs"

package_name = "tensorflow-base-2.7.0-py39_cuda11.1_cudnn8.1_7"
max_package_path = r"/Lib/site-packages/tensorflow/include/external/cudnn_frontend_archive/_virtual_includes/cudnn_frontend/third_party/cudnn_frontend/include/cudnn_frontend_EngineConfigGenerator.h"

# what's the maximum length of a username allowed below MAX_PATH for this case?
max_path - sum([len(s) for s in [user_base, pkgs_subdir, package_name, max_package_path]])

In this example, the maximum username length would be -2, so invalid paths will be written and the installation will fail unless long paths are enabled. For a package with a shorter package name and shorter lengths, this typically isn't an issue. As a general guideline, I'd say something like 130 characters for the package path should be OK. In the failure above the max package path length is 176 characters, and in the nodejs issue originally posted it is 190 characters.

[asmeurer]

Thanks for the info @scdub. As it were, I'm not really asking as a packager, but as someone trying to figure out how long of a prefix path I can get away with. That's why I asked if there's a de facto standard. But knowing that almost all installations use this particular directory structure is very helpful.