Windows: Getting ERROR: Could not install packages due to an EnvironmentError

[Ram-N]

All:
I am getting an error, that is very similar to #1046

On my Windows-10 machine, when I am trying to run pre-commit, I am getting a could not install packages (EnvironmentError).

Things I have tried:

1. Ran it from Anaconda prompt, same error.
2. pre-commit clean and then running - did not help
3. Based on @asottile suggestion in the other issue, I checked if import asyncio works, it does.
4. I was running Python 3.7.3, I downgraded to 3.7.1, but I'm still unable to shake off the error.
   (I am new to Pre-commit, but any help is much appreciated.)

Details

$ cat .pre-commit-config.yaml 
repos:
- repo: https://github.com/ambv/black
  rev: stable
  hooks:
  - id: black
    language_version: python3.7

$ pre-commit --version
pre-commit 1.18.1

Here's the error:

$ pre-commit
[INFO] Installing environment for https://github.com/ambv/black.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: Command: ('C:\\Users\\ramna\\.cache\\pre-commit\\repobylwb6ah\\py_env-python3.7\\Scripts\\pip.exe', 'install', '.')
Return code: 1
Expected return code: 0
Output: 
    Processing c:\users\ramna\.cache\pre-commit\repobylwb6ah
    
Errors: 
    ERROR: Could not install packages due to an EnvironmentError: [('C:\\Users\\ramna\\.cache\\pre-commit\\repobylwb6ah\\docs\\authors.md', 'C:\\Users\\ramna\\AppData\\Local\\Temp\\pip-req-build-dpix3821\\docs\\authors.md', 'symbolic link privilege not held'), ('C:\\Users\\ramna\\.cache\\pre-commit\\repobylwb6ah\\docs\\blackd.md', 'C:\\Users\\ramna\\AppData\\Local\\Temp\\pip-req-build-dpix3821\\docs\\blackd.md', 'symbolic link privilege not held'),

python -c "import sys; print(sys.version)"
3.7.1 | packaged by conda-forge | (default, Mar 13 2019, 13:32:59) [MSC v.1900 64 bit (AMD64)]

And for completeness, I went to the .cache directory and did an ls

ramna@Lenovo-Yoga MINGW64 ~/.cache/pre-commit/repobylwb6ah ((026c81b...))
$ ls -alth
total 321K
drwxr-xr-x 1 ramna 197609    0 Oct 24 08:19 .
drwxr-xr-x 1 ramna 197609    0 Oct 23 23:56 ..
drwxr-xr-x 1 ramna 197609    0 Oct 23 23:56 .git
drwxr-xr-x 1 ramna 197609    0 Oct 23 23:56 tests
-rw-r--r-- 1 ramna 197609 2.3K Oct 23 23:56 setup.py
-rw-r--r-- 1 ramna 197609   72 Oct 23 23:56 setup.cfg
-rw-r--r-- 1 ramna 197609 2.3K Oct 23 23:56 pyproject.toml
-rw-r--r-- 1 ramna 197609   90 Oct 23 23:56 readthedocs.yml
-rw-r--r-- 1 ramna 197609  33K Oct 23 23:56 pyproject.lock
drwxr-xr-x 1 ramna 197609    0 Oct 23 23:56 profiling
drwxr-xr-x 1 ramna 197609    0 Oct 23 23:56 plugin
-rw-r--r-- 1 ramna 197609  817 Oct 23 23:56 mypy.ini
drwxr-xr-x 1 ramna 197609    0 Oct 23 23:56 docs
drwxr-xr-x 1 ramna 197609    0 Oct 23 23:56 blib2to3
-rw-r--r-- 1 ramna 197609 5.7K Oct 23 23:56 blackd.py
-rw-r--r-- 1 ramna 197609 127K Oct 23 23:56 black.py
-rw-r--r-- 1 ramna 197609  52K Oct 23 23:56 README.md
-rw-r--r-- 1 ramna 197609  32K Oct 23 23:56 Pipfile.lock
-rw-r--r-- 1 ramna 197609  511 Oct 23 23:56 Pipfile
-rw-r--r-- 1 ramna 197609  143 Oct 23 23:56 MANIFEST.in
-rw-r--r-- 1 ramna 197609 1.1K Oct 23 23:56 LICENSE
-rw-r--r-- 1 ramna 197609 1.9K Oct 23 23:56 .travis.yml
-rw-r--r-- 1 ramna 197609 2.0K Oct 23 23:56 CONTRIBUTING.md
-rw-r--r-- 1 ramna 197609  218 Oct 23 23:56 .pre-commit-hooks.yaml
-rw-r--r-- 1 ramna 197609  540 Oct 23 23:56 .pre-commit-config.yaml
-rw-r--r-- 1 ramna 197609  131 Oct 23 23:56 .gitignore
drwxr-xr-x 1 ramna 197609    0 Oct 23 23:56 .github
-rw-r--r-- 1 ramna 197609  255 Oct 23 23:56 .flake8
-rw-r--r-- 1 ramna 197609   53 Oct 23 23:56 .coveragerc
-rw-r--r-- 1 ramna 197609  675 Oct 23 23:56 .appveyor.yml

Any suggestions for things to change or try are welcome.

[asottile]

that's pretty weird -- it looks as if you once held the symbolic link privilege and then it was removed?

what appens if you ls -alth inside the docs directory (that's the one that's failing) -- presumably also pre-commit clean should blow away the old symlinks and then a new run would succeed 🤔

the other option is to give yourself the symbolic link privilege

[Ram-N]

I tried pre-commit clean but the problem persists. Here's what's inside the docs directory.

ramna@Lenovo-Yoga MINGW64 ~/.cache/pre-commit/repobylwb6ah/docs ((026c81b...))
$ ls -alth
total 39K
drwxr-xr-x 1 ramna 197609    0 Oct 24 08:19 ..
drwxr-xr-x 1 ramna 197609    0 Oct 23 23:56 .
lrwxrwxrwx 1 ramna 197609   40 Oct 23 23:56 the_black_code_style.md -> _build/generated/the_black_code_style.md
lrwxrwxrwx 1 ramna 197609   47 Oct 23 23:56 version_control_integration.md -> _build/generated/version_control_integration.md       
lrwxrwxrwx 1 ramna 197609   32 Oct 23 23:56 testimonials.md -> _build/generated/testimonials.md
-rw-r--r-- 1 ramna 197609   36 Oct 23 23:56 requirements.txt
lrwxrwxrwx 1 ramna 197609   35 Oct 23 23:56 show_your_style.md -> _build/generated/show_your_style.md
drwxr-xr-x 1 ramna 197609    0 Oct 23 23:56 reference
lrwxrwxrwx 1 ramna 197609   34 Oct 23 23:56 pyproject_toml.md -> _build/generated/pyproject_toml.md
-rw-r--r-- 1 ramna 197609  809 Oct 23 23:56 make.bat
lrwxrwxrwx 1 ramna 197609   42 Oct 23 23:56 installation_and_usage.md -> _build/generated/installation_and_usage.md
lrwxrwxrwx 1 ramna 197609   27 Oct 23 23:56 license.md -> _build/generated/license.md
-rw-r--r-- 1 ramna 197609 2.0K Oct 23 23:56 index.rst
lrwxrwxrwx 1 ramna 197609   45 Oct 23 23:56 ignoring_unmodified_files.md -> _build/generated/ignoring_unmodified_files.md
lrwxrwxrwx 1 ramna 197609   38 Oct 23 23:56 editor_integration.md -> _build/generated/editor_integration.md
-rw-r--r-- 1 ramna 197609  173 Oct 23 23:56 environment.yml
lrwxrwxrwx 1 ramna 197609   41 Oct 23 23:56 contributing_to_black.md -> _build/generated/contributing_to_black.md
-rw-r--r-- 1 ramna 197609 8.5K Oct 23 23:56 conf.py
lrwxrwxrwx 1 ramna 197609   18 Oct 23 23:56 contributing.md -> ../CONTRIBUTING.md
lrwxrwxrwx 1 ramna 197609   30 Oct 23 23:56 change_log.md -> _build/generated/change_log.md
lrwxrwxrwx 1 ramna 197609   27 Oct 23 23:56 authors.md -> _build/generated/authors.md
lrwxrwxrwx 1 ramna 197609   26 Oct 23 23:56 blackd.md -> _build/generated/blackd.md
drwxr-xr-x 1 ramna 197609    0 Oct 23 23:56 _static
-rw-r--r-- 1 ramna 197609  621 Oct 23 23:56 Makefile

[asottile]

interesting, pre-commit clean should be blowing that directory away (and then it should delete all the symlinks) -- is it still the same path the second time? can you show the output of pre-commit clean?

[Ram-N]

Yes, I am blowing away that directory. Here's the output as requested.
I haven't yet tried to give myself Symlink permissions. Maybe therein lies the problem?

ramna@Lenovo-Yoga MINGW64 ~/PyLibrary/abc-classroom (docfix)
$ ls ~/.cache/pre-commit/
db.db  pre-commit.log  README  repofsty8xaa
(abc-dev)
ramna@Lenovo-Yoga MINGW64 ~/PyLibrary/abc-classroom (docfix)
$ pre-commit clean
Cleaned C:\Users\ramna/.cache\pre-commit.
(abc-dev)
ramna@Lenovo-Yoga MINGW64 ~/PyLibrary/abc-classroom (docfix)
$ ls ~/.cache/pre-commit/
ls: cannot access '/c/Users/ramna/.cache/pre-commit/': No such file or directory
(abc-dev)
ramna@Lenovo-Yoga MINGW64 ~/PyLibrary/abc-classroom (docfix)
$ pre-commit
[INFO] Initializing environment for https://github.com/ambv/black.
[INFO] Installing environment for https://github.com/ambv/black.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: Command: ('C:\\Users\\ramna\\.cache\\pre-commit\\repojrhkb6rg\\py_env-python3.7\\Scripts\\pip.exe', 'install', '.')
Return code: 1
Expected return code: 0
Output: 
    Processing c:\users\ramna\.cache\pre-commit\repojrhkb6rg
    
Errors: 
    ERROR: Could not install packages due to an EnvironmentError: [('C:\\Users\\ramna\\.cache\\pre-commit\\repojrhkb6rg\\docs\\authors.md', 'C:\\Users\\ramna\\AppData\\Local\\Temp\\pip-req-build-4nxog5tk\\docs\\authors.md', 'symbolic link privilege not held'), ('C:\\Users\\ramna\\.cache\\pre-commit\\repojrhkb6rg\\docs\\blackd.md', 'C:\\Users\\ramna\\AppData\\Local\\Temp\\pip-req-build-4nxog5tk\\docs\\blackd.md', 'symbolic link privilege not held'), ('C:\\Users\\ramna\\.cache\\pre-commit\\repojrhkb6rg\\docs\\change_log.md', 'C:\\Users\\ramna\\AppData\\Local\\Temp\\pip-req-build-4nxog5tk\\docs\\change_log.md', 'symbolic link privilege not held'), ('C:\\Users\\ramna\\.cache\\pre-commit\\repojrhkb6rg\\docs\\contributing.md', 'C:\\Users\\ramna\\AppData\\Local\\Temp\\pip-req-build-4nxog5tk\\docs\\contributing.md', 'symbolic link privilege not held'), ('C:\\Users\\ramna\\.cache\\pre-commit\\repojrhkb6rg\\docs\\contributing_to_black.md', 'C:\\Users\\ramna\\AppData\\Local\\Temp\\pip-req-build-4nxog5tk\\docs\\contributing_to_black.md', 'symbolic link privilege not held'), ('C:\\Users\\ramna\\.cache\\pre-commit\\repojrhkb6rg\\docs\\editor_integration.md', 'C:\\Users\\ramna\\AppData\\Local\\Temp\\pip-req-build-4nxog5tk\\docs\\editor_integration.md', 'symbolic link privilege not held'), ('C:\\Users\\ramna\\.cache\\pre-commit\\repojrhkb6rg\\docs\\ignoring_unmodified_files.md', 'C:\\Users\\ramna\\AppData\\Local\\Temp\\pip-req-build-4nxog5tk\\docs\\ignoring_unmodified_files.md', 'symbolic link privilege not held'), ('C:\\Users\\ramna\\.cache\\pre-commit\\repojrhkb6rg\\docs\\installation_and_usage.md', 'C:\\Users\\ramna\\AppData\\Local\\Temp\\pip-req-build-4nxog5tk\\docs\\installation_and_usage.md', 'symbolic link privilege not held'), ('C:\\Users\\ramna\\.cache\\pre-commit\\repojrhkb6rg\\docs\\license.md', 'C:\\Users\\ramna\\AppData\\Local\\Temp\\pip-req-build-4nxog5tk\\docs\\license.md', 'symbolic link privilege not held'), ('C:\\Users\\ramna\\.cache\\pre-commit\\repojrhkb6rg\\docs\\pyproject_toml.md', 'C:\\Users\\ramna\\AppData\\Local\\Temp\\pip-req-build-4nxog5tk\\docs\\pyproject_toml.md', 'symbolic link privilege not held'), ('C:\\Users\\ramna\\.cache\\pre-commit\\repojrhkb6rg\\docs\\show_your_style.md', 'C:\\Users\\ramna\\AppData\\Local\\Temp\\pip-req-build-4nxog5tk\\docs\\show_your_style.md', 'symbolic link privilege not held'), ('C:\\Users\\ramna\\.cache\\pre-commit\\repojrhkb6rg\\docs\\testimonials.md', 'C:\\Users\\ramna\\AppData\\Local\\Temp\\pip-req-build-4nxog5tk\\docs\\testimonials.md', 'symbolic link privilege not held'), ('C:\\Users\\ramna\\.cache\\pre-commit\\repojrhkb6rg\\docs\\the_black_code_style.md', 'C:\\Users\\ramna\\AppData\\Local\\Temp\\pip-req-build-4nxog5tk\\docs\\the_black_code_style.md', 'symbolic link privilege not held'), ('C:\\Users\\ramna\\.cache\\pre-commit\\repojrhkb6rg\\docs\\version_control_integration.md', 'C:\\Users\\ramna\\AppData\\Local\\Temp\\pip-req-build-4nxog5tk\\docs\\version_control_integration.md', 'symbolic link privilege not held')]



Check the log at C:\Users\ramna/.cache\pre-commit\pre-commit.log

[asottile]

what does python -c 'import struct; print(struct.calcsize("P" * 8))' give you? I wonder if this is 32 bit python being unable to symlink but 64bit git having no problems?

it appears that git is able to create symlinks just fine but python isn't (at least that's what I see from the output)

[Ram-N]

$ python -c 'import struct; print(struct.calcsize("P" * 8))'
64

[Ram-N]

@asottile I am hesitant to impose on your valuable time, but if a zoom or hangout call will speed up things, I'd be glad to share my screen and we can try things.

[asottile]

I'm at work right now but could maybe do a quick call on the weekend maybe. The easiest short fix would probably be enabling symbolic links privilege for your user?

[Ram-N]

Sure, thank you. In the meantime, I will try enabling symbolic links privilege for myself.

[Ram-N]

I am able to get pre-commit working now! It was the Symlink permission issue.

It took me hours of searching and trying things. I'm adding this in case others come here with the same issue. What worked for me:

1. @asottile correctly pointed to the right place. I had to give myself permission to Symlinks. (Note that git was able to do create symbolic links, but Windows was not able to.) The relevant link for that is: https://superuser.com/questions/124679/how-do-i-create-a-link-in-windows-7-home-premium-as-a-regular-user
2. Now, on my personal laptop, I only have Windows 10 home edition. This, unfortunately, means that I don't have secpol.msc at all.
3. This YouTube video helped me out: https://www.youtube.com/watch?v=_FHgfioupLo Specifically, there is an all-important .bat script in the notes for this video.
4. I downloaded that script, right-clicked and did a "Run as Administrator." After a few minutes, I had secpol.msc available. I followed the instructions in 1. above.
5. Finally, after giving myself permission, I was still getting the error. But after I shut down and restarted my machine, the Symlink permissions kicked in, and I was able to get pre-commit running.

This issue can be closed. Thanks again, @asottile for all your help.

[asottile]

cheers! thanks for the thorough follow-up as well! this will surely be helpful for the next person

[s-weigand]

@Ram-N Thanks for the guide 😃
Additional info: On Win10 Pro x64 with active Developer Mode, secpol.msc did work out of the box.

[j9ac9k]

Hi @Ram-N thank you so much for detailed steps toward your solution. I ran into this same issue with my Windows 10 Home installation, and was initially really confused, until I saw the last step of a reboot; and now pre-commit is running as expected.