Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot install packages with long filenames #7203

Open
minrk opened this issue Apr 23, 2018 · 28 comments

Comments

@minrk
Copy link
Contributor

commented Apr 23, 2018

I'm submitting a...

  • bug report
  • feature request

Current Behavior

Cannot install nodejs on Windows due to long paths.

nodejs produces extremely deeply nested node_modules/pkg/node_modules/pkg/..., making it pretty easy to readily exceeding the Windows max path limit. nodejs itself doesn't seem have any issues with these long paths due to using modern Windows APIs, but Python seems to have issues.

This is related to #2452, but there seem to be other places (install) where UNC path prefixes or similar patches may be needed.

Reported on the nodejs conda package: conda-forge/nodejs-feedstock#12

(C:\Users\sarathod\AppData\Local\Continuum\anaconda3) C:\Users\sarathod>conda install -c conda-forge nodejs
Fetching package metadata ...............
Solving package specifications: .

Package plan for installation in environment C:\Users\sarathod\AppData\Local\Continuum\anaconda3:

The following NEW packages will be INSTALLED:

    nodejs: 9.9.0-0 conda-forge

Proceed ([y]/n)? y
CondaError: FileNotFoundError(2, 'The filename or extension is too long')
CondaError: FileNotFoundError(2, 'The filename or extension is too long')
CondaError: FileNotFoundError(2, 'The filename or extension is too long')

Steps to Reproduce

On Windows:

conda install -c conda-forge nodejs==9.9

Expected Behavior

package should install without errors

@mingwandroid

This comment has been minimized.

Copy link
Contributor

commented Apr 23, 2018

Closing, please fill in the details asked for in the issue template.


EDIT: I (@kalefranz) am overtaking this comment to pull out action items discussed below.

  • Include long path enabling in conda init.
  • Double bonus if it could be in the error message from conda itself when install fails due to long path errors. It'd be great if Conda could detect this particular problem and give a better error message, e.g. "The system has hit the maximum file-path length for this version of Windows".
  • Better long paths FAQ, documentation on limitations, etc.
@jakirkham

This comment has been minimized.

Copy link
Contributor

commented May 7, 2018

Not following. Seems like all of the info was filled out, no? Maybe a more specific request would help.

@kalefranz

This comment has been minimized.

Copy link
Member

commented May 7, 2018

One part in particular that’s missing is conda info. I had initial questions that would have answered.

@jakirkham

This comment has been minimized.

Copy link
Contributor

commented May 7, 2018

This report ( conda-forge/nodejs-feedstock#33 ) does supply conda info with conda version 4.5.1 and they are running into the same issue.

@lebedov

This comment has been minimized.

Copy link

commented May 7, 2018

I reported conda-forge/nodejs-feedstock#33; the problem arose while trying to install nodejs 9.11.0-0 from conda-forge using conda 4.5.1 and python 3.6.5 on Win7. Output of conda info follows:

     active environment : None
       user config file : C:\Users\lebedov\DOCUME~1\MOBAXT~1\home\.condarc
 populated config files : C:\Users\lebedov\DOCUME~1\MOBAXT~1\home\.condarc
          conda version : 4.5.1
    conda-build version : 3.8.1
         python version : 3.6.5.final.0
       base environment : C:\Users\lebedov\AppData\Local\Continuum\anaconda3  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/win-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/win-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/pro/win-64
                          https://repo.anaconda.com/pkgs/pro/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\lebedov\AppData\Local\Continuum\anaconda3\pkgs
                          C:\Users\lebedov\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\lebedov\AppData\Local\Continuum\anaconda3\envs
                          C:\Users\lebedov\AppData\Local\conda\conda\envs
                          C:\Users\lebedov\DOCUME~1\MOBAXT~1\home\.conda\envs
               platform : win-64
             user-agent : conda/4.5.1 requests/2.18.4 CPython/3.6.5 Windows/7 Windows/6.1.7601
          administrator : True
             netrc file : None
           offline mode : False
@minrk

This comment has been minimized.

Copy link
Contributor Author

commented May 20, 2018

Can we get this reopened? All requested info has been provided.

@kalefranz

This comment has been minimized.

Copy link
Member

commented May 20, 2018

I’m ok re-opening. However fixing this is a non-trivial project that will have to be stacked against other priorities. In the meantime, users encountering this problem can add something like

pkgs_dirs:
  - c:\conda-pkgs

to their condarc file.

@kalefranz kalefranz reopened this May 20, 2018

@mingwandroid

This comment has been minimized.

Copy link
Contributor

commented May 20, 2018

Sure but I should note that fixing it requires asking the end user to set a registery key and that's going to be a lot of work.

Also allowing conda to install to paths requiring >260 characters will break a whole load of software with inbuilt assumptions of a maximum requirement of 260 characters, some of which testing would catch allowing us to fix but some of it not.

These unfixed ones can in turn lead to stack buffer overflows so it's not something that can be just done and forgotten about (any time a function assumes that the calling function had already checked the 260 limit which would no longer always be the case for example).

Regardless, a registery key with this option could be implemented into constructor and PRs to do that would be welcomed.

@kalefranz

This comment has been minimized.

Copy link
Member

commented May 20, 2018

@mingwandroid

This comment has been minimized.

Copy link
Contributor

commented May 20, 2018

I think it needs to be done at the Administrator level. Official CPython binary releases do this in their install process on Windows.

@jakirkham

This comment has been minimized.

Copy link
Contributor

commented May 20, 2018

Thanks all.

FWIW 3 issues were filed against the nodejs feedstock on this exact same issue with at least 8 people claiming to be affected by this problem. 2 of these issues were filed in the past 2 months. The older one (filed late last year) has had activity within the past week. Expect this is a small portion of the actual number of people affected by this problem. Most of whom likely switched to using Node.js some other way and we will likely never know. As this is a common path users are going to take when working with JavaScript in Conda, it seems worthwhile to solve.

Guessing by registry key change, you are referring to something like this, correct?

@mingwandroid

This comment has been minimized.

Copy link
Contributor

commented May 20, 2018

Indeed. And as you can see it's win 10 only so NSIS UI would be to be done for that unless conda init will be presenting UIs around UAC and the like?

But let's not rush into another noarch: python scenario here where a feature gets promoted despite considerable issues in the wider ecosystem in this case regarding security concerns being resolved first.

And I also believe this is being or had been fixed in newer node releases and we're mostly editing for an LTS release to fix it anyway?

@jakirkham

This comment has been minimized.

Copy link
Contributor

commented May 20, 2018

Are there solutions for older versions of Windows?

Not sure I follow the noarch: python comment. In any event, only trying to point out this is a significant pain point that is in need of a solution.

Recent reports include Node.js 9.11.0, which was released nearly two months ago.

@mingwandroid

This comment has been minimized.

Copy link
Contributor

commented May 20, 2018

No solutions I'm aware of no. We tried ages ago to use UNC paths in conda but Windows file locking broke badly.

The noarch: python comment is a reference to the complications that adding a seemingly 'simple' feature to conda caused a great amount of strife in the ecosystem around constructor, testing and differences between how conda-forge and defaults build their packages and I'm keen for this not to happen again when people start using deep install paths and their software doesn't work in there while we're busy advertising that 'long paths work now!' or some other simplification of reality.

@minrk

This comment has been minimized.

Copy link
Contributor Author

commented May 21, 2018

My understanding of this being fixed in node is that I believe node and npm themselves no longer have a problem with extremely deep paths on Windows, and continue to routinely make paths greater than 260 characters and that won't change. There was an improvement in path length when npm started deduping packages by default, resulting in a slightly reduced path length, but then shortly after it also increased its dependencies essentially negating that improvement. That was always a partial fix that would only work for some packages.

It's AOK for me if this starts out as a documentation I can point to, even if it's a manual fix requiring admin permissions for some folks. Of course, true support for long paths would be ideal, but a clear "You need to do this..." link would be a great start. Double bonus if it could be in the error message from conda itself when install fails due to long path errors.

@kalefranz

This comment has been minimized.

Copy link
Member

commented May 21, 2018

For binary file prefix replacement, we have to be careful that we're not trying to shove in a path that's longer than the binary placeholder. It's been a while since I looked at this logic, so I created #7324 as a reminder. Prefix replacement in text files isn't as much of an issue.

@SeaDude

This comment has been minimized.

Copy link

commented May 21, 2018

@kalefranz ,

Host: Win10
Guest: VirtualBox of Win10
Goal: To get past the Jupyter Lab 404 error by installing nodejs
Steps:

  • Was stuck at nodejs download 77% with a fail of The filename or extension is too long
  • Added your recommended:
pkgs_dirs:
  - c:\conda-pkgs

to my condarc file

  • This got me past the 77% download BUT download still failed with:
Preparing transaction: done
Verifying transaction: done
Executing transaction: failed
ERROR conda.core.link:_execute(502): An error occurred while installing package 'conda-forge::nodejs-9.11.1-0'.
FileNotFoundError(2, 'The filename or extension is too long')
Attempting to roll back.

Rolling back transaction: done

FileNotFoundError(2, 'The filename or extension is too long')

Not sure where to go from here.

@SeaDude

This comment has been minimized.

Copy link

commented May 22, 2018

Hacked a way through...

  • Launched anaconda-navigator
  • Clicked "Environments"
  • Selected "Not Installed" from le dropdown
  • Entered "nodejs" in search box
  • Selected it to install (NOTE: It is still installing nodejs 8.9
  • After installation, clicked "Jupyter Lab" from Navigator and ran jupyter lab from cli without issue.

May this help someone else too 👍 !

@kalefranz

This comment has been minimized.

Copy link
Member

commented May 22, 2018

@SeaDude

After that configuration change, try

conda create -p c:\node-env -c conda-forge nodejs
@SeaDude

This comment has been minimized.

Copy link

commented May 22, 2018

Seems to "work" though I have no idea what I just did!

(base) C:\Users\XXXX>conda create -p c:\node-env -c conda-forge nodejs
Solving environment: done

## Package Plan ##

  environment location: c:\node-env

  added / updated specs:
    - nodejs


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    vs2015_runtime-14.0.25420  |                0         1.9 MB  conda-forge

The following NEW packages will be INSTALLED:

    nodejs:         9.11.1-0     conda-forge
    vs2015_runtime: 14.0.25420-0 conda-forge

Proceed ([y]/n)? y


Downloading and Extracting Packages
vs2015_runtime-14.0. |  1.9 MB | ############################################################################## | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate c:\node-env
#
# To deactivate an active environment, use
#
#     $ conda deactivate

anaconda-navigator still shows a nodejs version of 8.9

@kalefranz

This comment has been minimized.

Copy link
Member

commented May 22, 2018

You successfully created an environment located at c:\node-env containing nodejs. To use it, run

conda activate c:\node-env

in the Anaconda Prompt.

@WenqiangXieIgg

This comment has been minimized.

Copy link

commented Jun 26, 2018

I fix it to enable Win32 long paths. As windows10 pro, you can open Local Group Policy Editor, and open item Computer Configuration -> Administrative Templates -> System -> Filesystem -> Enable Win32 long paths and click it to make it enable. But I have got the information it's just support windows10, I don't know it will or will not work in Windows8 or old.

@pgunn

This comment has been minimized.

Copy link

commented Aug 2, 2018

I recently ran into this issue, which is much worse in default configurations of most CI systems because they want to work out of the checkout directory, which can already have a fairly long path. The error message wasn't particularly helpful ; it just gave me a

FileNotFoundError(2, 'No such file or directory')

during an environment build (at the "Executing transaction" stage). Fortunately, I've seen this kinda thing before, but if possible it'd be great if Conda could detect this particular problem and give a better error message, e.g. "The system has hit the maximum file-path length for this version of Windows". This would help people who are not aware of these limits on Windows as they have a longer and more fragile path to understanding.

pgunn added a commit to flatironinstitute/CaImAn that referenced this issue Aug 2, 2018
Deal with MAXPATHLENGTH issue on Windows in CI
Windows has a maximum path length that it can reason about. A recent refactoring of a crypto package pushed us just over this limit, but packages with long descriptions could too.

This just pushes the problem back a bit. If we run into this again (will only hit windows), we may need to move the package install dir out from the checkout working dir, which would make for management issues but would otherwise pretty solidly solve the issue.

See conda/conda#7203 for more info.
@dhirschfeld

This comment has been minimized.

Copy link
Contributor

commented Aug 3, 2018

I use the below in my docker images:

Set-ItemProperty -Path HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem -Name LongPathsEnabled -Value 1;

https://docs.python.org/3/using/windows.html#removing-the-max-path-limitation

Hope that helps, but I'm not 100% sure it's the cause of the problem described

@mingwandroid

This comment has been minimized.

Copy link
Contributor

commented Aug 3, 2018

How is docker in Windows working out for you wrt conda these days @dhirschfeld?

It's been in my to-do list for ever to see if we can use it for our buildsn somehow.

@dhirschfeld

This comment has been minimized.

Copy link
Contributor

commented Aug 3, 2018

Containers are working out great. I've got a full Python & R analytics environment deployed in a Windows container image using conda.

The main issue I'm facing is trying to mix and match defaults and conda-forge which can sometimes cause problems - e.g. conda-forge/openmp-feedstock#8

I'm trying to sort these out one-by-one so that hopefully going forwards they will play nicely together.

In that vein I was hoping to get you to take a look at AnacondaRecipes/pyzmq-feedstock#2! 😉

R is more difficult - you pretty much can't mix channels but I understand it's up to conda-forge to implement the _r-mutex scheme. Currently I'm installing both R and Python in the same environment so you can call one from the other but I'm wondering if it wouldn't be better to install into separate environments and stack them. It's on my TODO list to test that.

The windowsservercore miniconda base image is available at https://hub.docker.com/r/dhirschfeld/miniconda-core/. It's slightly out of date there but it's easy enough to build your own.

Also on my TODO is to try and setup a docker image for building our conda packages and to hook it into our TFS CI.

Refs:

@kalefranz

This comment has been minimized.

Copy link
Member

commented Aug 3, 2018

And I also believe this is being or had been fixed in newer node releases and we're mostly editing for an LTS release to fix it anyway?

It's my understanding that npm changed its strategy for vendoring packages. The original intent of vendoring was to let each package use whatever version of a library it wanted, including libraries using libraries. Definitely an understandable goal. The iteration of npm that fixed this in many cases still lets multiple versions of packages exist in an "environment" (npm equivalent), but also tries to flatten dependencies as best as possible back to whatever the root level is when it can. I'm probably using different words than npm uses to describe this. It's been a while since I read about it.

Edit: @minrk already had basically the same response, but his is more complete and accurate.

@Proteusiq

This comment has been minimized.

Copy link

commented Jan 16, 2019

I can confirm that setting LongPathsEnabled from 0 to 1 solved this issue on Windows 10 64bits
cmd -> regedit -> HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem -> LongPathsEnabled -> 1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
10 participants
You can’t perform that action at this time.