Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NotWritableError: The current user does not have write permissions to a required path. #7267

Closed
marak8 opened this issue May 8, 2018 · 98 comments

Comments

@marak8
Copy link

marak8 commented May 8, 2018

Conda is seriously confused about permissions. Right after I installed Anaconda from Anaconda3-5.1.0-Linux-x86_64.sh into my home (/home/marek/anaconda3) and added

. /home/marek/anaconda3/etc/profile.d/conda.sh

into my ~/.zshrc and started new shell, I tried to create new environment:

% conda create --name condatest python=3
Solving environment: failed

NotWritableError: The current user does not have write permissions to a required path.
  path: /home/marek/.conda/pkgs/urls.txt
  uid: 1009
  gid: 1009

If you feel that permissions on this path are set incorrectly, you can manually
change them by executing

  $ sudo chown 1009:1009 /home/marek/.conda/pkgs/urls.txt

In general, it's not advisable to use 'sudo conda'.
  • My user and group ids are 1009.
  • ~/.conda directory didn't exist before I run the command above
  • ~/.conda directory DID exist after I run the command above, though it was empty
  • % touch ~/.conda/foo works

This happens on latest ArchLinux, latest Ubuntu (bionic) and with both bash and zsh as shell...

@marak8
Copy link
Author

marak8 commented May 8, 2018

TEMPORARY WORKAROUND

Add to your ~/.zshrc/~/.bashrc:

unset SUDO_UID SUDO_GID SUDO_USER

(Note that conda checks only the SUDO_UID.) This isn't the solution as it breaks programs that rely correctly on these variables.


OK. I narrowed it down. It's caused by mkdir_p_sudo_safe. I forgot to mention, that I have users for various tasks, into which I login from main user with

sudo -u marek -i

This opens new login shell under the specified user. I've seen many people use this practice.

I don't really understand what business does Anaconda have with sudo but I suggest to remove this behavior as it breaks legitimate user setups. Anaconda probably expects that sudo is only used to become root, which is far from truth.

    if isdir(path): 
        return 
    base_dir = dirname(path) 
    if not isdir(base_dir): 
        mkdir_p_sudo_safe(base_dir) 
    log.trace('making directory %s', path) 
    mkdir(path) 
    if not on_win and os.environ.get('SUDO_UID') is not None: 
        uid = int(os.environ['SUDO_UID']) 
        gid = int(os.environ.get('SUDO_GID', -1)) 
        log.trace("chowning %s:%s %s", uid, gid, path) 
        os.chown(path, uid, gid) 

Obviously, the user to which I login via sudo doesn't have permissions to chown() files under previous account credentials.

@argideritzalpea
Copy link

argideritzalpea commented May 8, 2018

I am getting the same errors:

NotWritableError: The current user does not have write permissions to a required path.
  path: //anaconda/pkgs/conda-4.5.2-py27_0/info/repodata_record.json
  uid: 502
  gid: 20

If you feel that permissions on this path are set incorrectly, you can manually
change them by executing

  $ sudo chown 502:20 //anaconda/pkgs/conda-4.5.2-py27_0/info/repodata_record.json

In general, it's not advisable to use 'sudo conda'.

I tried deleting my ~/USER/anaconda folder and reinstalling, to no avail.

Can anyone help me fix my conda environment? I am not experienced with this type of thing.

@genome1
Copy link

genome1 commented May 15, 2018

I am getting the same error message although the target path is different.

The conda installation is initiated by bcbio_nextgen (https://github.com/bcbio/bcbio-nextgen) but the error comes from conda itself. My job script is running on a server that only permits writing to a temporary work directory, so writing onto my home folder is not allowed. Changing the environment settings with CONDA_ENVS_PATH and CONDA_PKGS_DIRS to a writable directory does not solve the problem. Is there any workaround to avoid this error?

NotWritableError: The current user does not have write permissions to a required path.
path: /home/myaccount/.conda/environments.txt
uid: 2026
gid: 11181

If you feel that permissions on this path are set incorrectly, you can manually
change them by executing

$ sudo chown 2026:11181 /home/myaccount/.conda/environments.txt

In general, it's not advisable to use 'sudo conda'.

@WeatherGod
Copy link

Same problem here when done from within a Dockerfile. For consistency with a similarly built EC2 image, I have conda installed as the "ec2-user". All docker commands are issued as root, so I have to do sudo -u ec2-user <install commands> to have conda installed as the ec2-user. This used to work until very recently.

@charlesreid1
Copy link

charlesreid1 commented May 16, 2018

This is also a problem when root runs a script containing "conda install" commands as a regular user via sudo -H -u username my_script.sh. OP nailed the core issue with this comment:

Anaconda probably expects that sudo is only used to become root, which is far from truth.

EDIT: another unpleasant workaround for folks to try... Calling my_script.sh (which contains conda install commands) directly will fail, but calling it through Python using subprocess.call() and passing shell=True apparently convinces conda it is not being run via the sudo command:

sudo -H -u username my_script.sh   # fails
sudo -H -u username my_script_via_subprocess.py  # works

The contents of my_script_via_subprocess.py is basically:

import subprocess
subprocess.call(['./my_script.sh'], shell=True)

@WeatherGod
Copy link

Another use-case that is now broken... I have production and test machines where we are supposed to log in using our user accounts, but then deploy software sudo'ed as a common user (this allows for multiple developers to be able to maintain the server, regardless of who deployed the original code). This is completely broken!

I am now trying to figure out when was the last version of conda where this worked so that I can continue doing deployments. It seems like it is 4.3.31 is the latest conda installer on repo.continuum.io that will leave you in a usable state.

@kalefranz
Copy link
Contributor

Rather than just sudo conda ..., try sudo su USERNAME -c ‘conda ...’.

I’m on my phone right now and can’t tryout the exact commands. Hopefully I’m giving enough hints for you to figure out what I mean though.

There’s a subtle difference between sudo and su.

Getting all this sudo usage exactly right with conda internals is still a work in progress. I think we are making progress, but we’re not done yet. There are multiple related open issues. Will try to swing back later and link them in here for reference.

@marak8
Copy link
Author

marak8 commented May 25, 2018

I don't understand why is this discussion this long. There's a full description of the problem, workaround, and hint for the EASY fix in conda in the second post.

Workaround: add unset SUDO_UID SUDO_GID SUDO_USER to ~/.bashrc

Fix: Just completely ignore sudo in conda. There's no need to get sudo right. Sudo IS right. You are wrong by thinking you have some business with sudo and by this you're breaking people's setup. Sudo is end user's tool, not your API.

@WeatherGod
Copy link

WeatherGod commented May 25, 2018 via email

@ghost
Copy link

ghost commented Jul 24, 2018

FWIW I have no problem in redhat/centos if I have a user "miniconda2", intall to /opt/miniconda2/ and give access to another user by making them a member of the miniconda2 group. Trying the same thing on debian fails completely for me.

@sveinugu
Copy link

sveinugu commented Sep 11, 2018

I have the exact same setup as @WeatherGod. It used to work, but no longer. Please fix this issue!

@keithj
Copy link

keithj commented Sep 12, 2018

Thanks for your work on Conda - it's made our builds and deployments easier and more pleasant.
On this issue specifically though, we have the same pain point as @sveinugu and @WeatherGod

We have a service user (which we sudo to) in order to deploy software. We now use a workaround with an Ansible playbook to preempt Conda by creating the directories it wants, ahead of invoking it. Ironically, the user we sudo to has significantly reduced permissions on the system e.g. no permission to modify production data which might be on mounted volumes.

@kalefranz
Copy link
Contributor

Based in part on the feedback in this issue, the sudo checks should have already been removed in master.

The master branch is currently tagged as 4.6.0beta0. We’ll be following our canary processs of the next several weeks to graduate this feature release into defaults.

Definitely encourage use and feedback on the current beta, as it relates to this issue or any other. (Please file unrelated bugs as new issues; not tacked onto this one.)

To opt in to the canary releases, execute

conda config --add channels conda-canary
conda update -n base conda

@Aisuko
Copy link

Aisuko commented Oct 13, 2018

The same issue with you, there is my answer with the same issue, maybe work for you Solving environment: failed" after conda update conda #7950

@WeatherGod
Copy link

@kalefranz just realized a problem with testing the canary. This error happens when installing packages or creating environments for the first time. So, if one has this problem, then I don't think they can properly test the canary channel. And, if they are able to update their install of conda, then they already have the files made.

One thing I noticed today is that repeated attempts at installing packages gets me closer. On the fourth attempt, everything worked. Each time, it complained about a different file. So, the file is still getting created, it is the chmod'ing that never happens I guess (but, in my case, the permissions are fine as-is).

The files that are complained about are, in this order:
.conda/pkgs/urls.txt (this is an immediate failure)
.conda/pkgs/urls.txt (yes, twice)
.conda/environments.txt (fails after figuring out the package plan and downloading)

@milljm
Copy link

milljm commented Jan 29, 2019

We too run into this issue. Long story short, we use Singularity throughout our continuous integration system (a bit like Docker). In order to keep the 'containers' small, we clear out the /pkgs/* directory. Once everything is set up, this directory (and the entire container) is read-only / sand-boxed.

So with pkgs/* gone, we end up deleting the urls.txt file with it. Conda can not re-create this file in a read-only filesystem, and thus we receive the same error os the original poster. I have found, if you touch this file, you can continue to create new environments without errors. Even if this file is empty and read-only.

touch <install prefix>/pkgs/urls.txt

I realize my workaround probably does not apply to most folks here. Took a while to nail this down, and thought I would mention what worked for us.

@utaiota
Copy link

utaiota commented Feb 11, 2019

Based in part on the feedback in this issue, the sudo checks should have already been removed in master.

The master branch is currently tagged as 4.6.0beta0. We’ll be following our canary processs of the next several weeks to graduate this feature release into defaults.

Definitely encourage use and feedback on the current beta, as it relates to this issue or any other. (Please file unrelated bugs as new issues; not tacked onto this one.)

To opt in to the canary releases, execute

conda config --add channels conda-canary
conda update -n base conda

It worked for me!
;)

@whamblen
Copy link

whamblen commented Mar 4, 2019

I have a comment that I think is related to what @milljm reported on January 29.

We had a read-only conda environment where pkgs/urls and pkgs/urls.txt were somehow group writeable even though the rest of pkgs/ were not - especially not pkgs/cache. Commands like "conda create" and "conda search" failed (NotWriteableError) for people in the group because conda was trying open for write the .json files in pkgs/cache. Simply removing the group write bit on pkgs/urls[.txt] fixed the issue because the same commands now went to pkgs/cache in the user's home .conda directory where they do have write.

In other words, it seems that conda was checking permissions for pkgs/urls in our read-only environment to determine if it could open the associated pkgs/cache/blah.json for write. And in this case it could not.

I'm not going to call this an actual conda bug since I cannot explain how pkgs/urls[.txt] became writeable except to say that it happened when I installed Miniconda to a specific location. But it does seem like conda might be able to work around it.

This is easily reproducible in our environment and I am happy to create a separate bug report with more details if someone wants to tell me where a Miniconda installer bug should go.

@milljm
Copy link

milljm commented Apr 9, 2019

Do not do the above! 👍 🎉 ❤️ 🚀
Please, everyone coming here finding and cheering on the above solution... That was posted by someone that does not understand octal permissions. There should almost never be a reason to run sudo chmod 777 -R. Doing so creates a tremendous security risk on your part. Proceed further down to @alexdauenhauer solution of using chown instead.
[end edit]

If you needed to use sudo to chmod something in your own home directory, that might represent the actual issue. Did you install anaconda using sudo at any point? Shouldn't be necessary.

@WeatherGod
Copy link

Why hasn't there been a miniconda installer released for the 4.6.x series? This bug makes the 4.4.x and 4.5.x series unusable for me.

@msarahan
Copy link
Contributor

There are test installers going through QA now. You can test them yourself at https://repo.anaconda.com/pkgs/misc/previews/miniconda/4.6.12/

There has been a rebuild of conda 4.6.12 since those were created. If you use windows with git bash or msys2, you will probably want to immediately update to 4.6.12 build 1 using the cmd.exe prompt (not bash), and then use bash once you've updated.

@WeatherGod
Copy link

WeatherGod commented Apr 11, 2019

Thank you. I just gave one of them a try and it seemed to have worked (although it did warn me about an unwritable path:

  environment location: /home/ec2-user/.conda/environments.txt

But it kept going and it seemed to have successfully finish the job.

@mfn
Copy link

mfn commented Apr 17, 2019

Have/had the same problem, it would abort with this permission error although everything is fine.

This is because I use sudo -i -u <the user> and then tried the conda installation.

unset SUDO_UID fixed it for me.

The installer mentioned in #7267 (comment) did not fix it for me!

@WeatherGod
Copy link

I wonder if the difference is that in the case of my installer, I was doing it from a docker container where I was originally "root" and used sudo -i -u <the user> to do the install as that user? This is the case that broke for me last year. I haven't had the time to check if sudo-ing over to that user from a non-root account would look any different.

@KseniaErshova
Copy link

KseniaErshova commented Apr 18, 2019

Based in part on the feedback in this issue, the sudo checks should have already been removed in master.

The master branch is currently tagged as 4.6.0beta0. We’ll be following our canary processs of the next several weeks to graduate this feature release into defaults.

Definitely encourage use and feedback on the current beta, as it relates to this issue or any other. (Please file unrelated bugs as new issues; not tacked onto this one.)

To opt in to the canary releases, execute

conda config --add channels conda-canary
conda update -n base conda

I did what you said but the error still persists:
Downloading and Extracting Packages
conda 4.5.2: ####################################################################################################################################################################################### | 100%
Preparing transaction: done
Verifying transaction: failed

NotWritableError: The current user does not have write permissions to a required path.
path: /Users/user/anaconda3/envs/catalog.json
uid: 502
gid: 20

If you feel that permissions on this path are set incorrectly, you can manually
change them by executing

$ sudo chmod 502:20 /Users/user/anaconda3/envs/catalog.json

How do I solve it?

P.S. Of course, sudo chmod 502:20 /Users/user/anaconda3/envs/catalog.json doesn't work, saying chmod: Invalid file mode: 502:20

@mfn
Copy link

mfn commented Apr 18, 2019

To elaborate my case from #7267 (comment)

  • I'm user userA
  • using sudo -i -u <conda_user>

The environment SUDO_* variables point to my initial userA.

Fun fact, even in some bash scripts I 've in conda_user I do in fact make use of some SUDO_* variables to detect this and perform certain actions.

However, at the end of the day, technically there is no real permission error. It seems like some part of conda installer is acting prematurely here.

@msaharia
Copy link

msaharia commented Apr 18, 2019

Hi - I am facing the same issue. And I am on a supercomputer, which means I don't have sudo access to anything. Would appreciate any help!

#8553 (Issue)

@hillaag
Copy link

hillaag commented Mar 30, 2020

You do not have to reinstall with your own user.. instead, you can just create an environment:

conda create -y -n conda-env
source activate conda-env

and re-run your conda install...
This solution worked for me.

@cervantes-loves-ai
Copy link

sudo chmod 777 -R /home/marek/anaconda3
worked with me

worked for me too!!!

also worked for me

@nhenscheid
Copy link

I had the same problem on Ubuntu because I installed Anaconda as a superuser:
sudo bash Anaconda3-2019.10-Linux-x86_64.sh

[solution Ubuntu] remove Anaconda and install it without the superuser privileges:

sudo rm -rf ~/anaconda3
bash Anaconda3-2019.10-Linux-x86_64.sh

This fixed the problem.

This worked for me.

@schw1804
Copy link

schw1804 commented May 9, 2020

I had the same problem on Ubuntu because I installed Anaconda as a superuser:
sudo bash Anaconda3-2019.10-Linux-x86_64.sh

[solution Ubuntu] remove Anaconda and install it without the superuser privileges:

sudo rm -rf ~/anaconda3
bash Anaconda3-2019.10-Linux-x86_64.sh

This fixed the problem.

Same problem. Same solution. Now, why does this happen?

@dwchen-tech
Copy link

I had the same problem on Ubuntu because I installed Anaconda as a superuser:
sudo bash Anaconda3-2019.10-Linux-x86_64.sh
[solution Ubuntu] remove Anaconda and install it without the superuser privileges:

sudo rm -rf ~/anaconda3
bash Anaconda3-2019.10-Linux-x86_64.sh

This fixed the problem.

Same problem. Same solution. Now, why does this happen?

Two ways to fix the problem.
1.
sudo bash Anaconda3-2019.10-Linux-x86_64.sh
sudo chmod 777 -R /home/*/anaconda3
2.
bash Anaconda3-2019.10-Linux-x86_64.sh
2 is the wise way!!!

@jobic10
Copy link

jobic10 commented May 26, 2020

Based in part on the feedback in this issue, the sudo checks should have already been removed in master.

The master branch is currently tagged as 4.6.0beta0. We’ll be following our canary processs of the next several weeks to graduate this feature release into defaults.

Definitely encourage use and feedback on the current beta, as it relates to this issue or any other. (Please file unrelated bugs as new issues; not tacked onto this one.)

To opt in to the canary releases, execute

conda config --add channels conda-canary
conda update -n base conda

I did
sudo chown -R $USER:$USER /home/unknown/anaconda3
alongside with the tagged tip and it worked for me.
Ubuntu 18.04

@caioltfo
Copy link

caioltfo commented Jul 5, 2020

If you are getting this error "NotWritableError: The current user does not have write permissions to a required path." means the conda directory doesn't have enough permissions to download and save new libraries.

Just run these commands in the terminal and you're ready:

sudo chown -R username /home/username/path-to-conda-folder/

sudo chmod -R +x /home/username/path-to-conda-folder/

Change the user name and path to the conda folder according to your PC.

Hope this helps! :)

Translated with www.DeepL.com/Translator (free version)

nudles added a commit to nudles/singa that referenced this issue Sep 15, 2020
Add the CI to run on MacOS, including building, testing pysinga and
uploading the package to anaconda if the token is available.

Fix the permission issue conda/conda#7267
and the sdk issue.
The sdk is fixed to the latest version (in conda_build_config.yaml).
The python is fixed to 3.6 to be compatible with colab.
@then4p
Copy link

then4p commented Sep 16, 2020

sudo chmod -R +x /home/username/path-to-conda-folder/

If I execute this command I get ' chown: invalid user: ‘+x’ ' However, just running the first command fixed the issue for me.

@paniash
Copy link

paniash commented Sep 27, 2020

I installed the conda package manager through my distribution's package manager. How do I know the path-to-conda folder?

@RoelantStegmann
Copy link

RoelantStegmann commented Oct 6, 2020

I have this issue when a I try to make a conda environment (in a CI pipeline, for testing a specific package). The container has been build via the standard anaconda docker

FROM continuumio/anaconda3:latest

and later

which conda
conda --version
conda create --yes --quiet --name test python=3.6.12

which returns

/opt/conda/bin/conda
conda 4.8.3
NotWritableError: The current user does not have write permissions to a required path.
  path: /opt/conda/pkgs/urls.txt

it feels weird that this does not work when we started from the official conda docker? :)

@morgansobol
Copy link

@caioltfo Is there another work around when you do not have permission to use sudo?

@SwapnilJain28
Copy link

Hello All,

The problem can be fixed by changing the permissions to 777.
sudo chmod 777 $PathOfDirectoryWhereAnacondaIsInstalled.
This worked for me.

@WeatherGod
Copy link

The problem can be fixed by changing the permissions to 777.
sudo chmod 777 $PathOfDirectoryWhereAnacondaIsInstalled.

Please do not do this. It opens you up to anyone on the system being able to make changes to your anaconda install. Chances are, in your case, the problem was that you couldn't list, read, and/or execute the contents of the directory where anaconda was installed. Fixing that does not require enabling write access for everyone.

@rogelioprieto
Copy link

The solution is assign permissions to the user.
I solved this issue using this (1. create a group. 2. add the user to the group 3. change permissions):

MYGROUP=miniconda3
sudo groupadd $MYGROUP
sudo usermod -a -G $MYGROUP myuser
sudo chown -R root:$MYGROUP $APPSFOLDER/miniconda3/
sudo chmod -R 775 $APPSFOLDER/miniconda3/

NOTE:
MYGROUP can be any descriptive name: anaconda, miniconda2 or miniconda3.
APPSFOLDER=/usr/local/bin

As my APPFOLDER is a system folder and is necessary sudo permissions.
MAYBE! if your APPSFOLDER is in your userfolder (example: /home/myuser/miniconda3) you can execute the chown command without sudo.

@gebob19
Copy link

gebob19 commented Aug 1, 2021

Nothing above worked for me but, bioconda/bioconda-recipes#7950 (comment) worked for me and doesn't require any sudo

@bmxbmx3
Copy link

bmxbmx3 commented Aug 31, 2021

I had the same problem on Ubuntu because I installed Anaconda as a superuser:
sudo bash Anaconda3-2019.10-Linux-x86_64.sh

[solution Ubuntu] remove Anaconda and install it without the superuser privileges:

sudo rm -rf ~/anaconda3
bash Anaconda3-2019.10-Linux-x86_64.sh

This fixed the problem.

this worked for me,thx!

@kingdavid72
Copy link

#8553

basically
you can try to edit .condarc file
then add pkg folder to your home

@franva
Copy link

franva commented Mar 21, 2022

FWIW I discovered that when I installed anaconda, I did so with sudo so the owner of the folder was root. This worked for me to fix this.

sudo chown -R username /path/to/anaconda3

obviously replace "username" with your username and the correct path to anaconda3

oh man, you brightened my day. I have been stuck by this issues for days!!

@whatdhack
Copy link

The summary looks like problem happens when the base conda environment (e.g. source ~/miniconda3/bin/activate ) was setup using someone else's base conda .

@lpasselin
Copy link

lpasselin commented May 31, 2022

We are suddently having this problem when switching from conda 4.9 to 4.11 (in a docker container)

btw, the proper way for multiple users to reuse the same conda installation on linux is detailed in the docs here: https://docs.anaconda.com/anaconda/install/multi-user/?highlight=chgrp#multi-user-anaconda-installation-on-linux

@jaanli
Copy link

jaanli commented Jun 7, 2022

I'm also getting this error on a new Mac. Can you reopen?

The error: NotWritableError: The current user does not have write permissions to a required path.

All I did was run:

brew install anaconda
conda install scikit-learn-intelex

Could you please reopen this issue?

@b-y-f
Copy link

b-y-f commented Sep 29, 2022

conda update -n base conda

This solved my problem!
Im using miniforge

@hams12khan
Copy link

start your command prompt with run as administrator

Using this method can solve your problem

@milljm
Copy link

milljm commented May 15, 2023

start your command prompt with run as administrator

Using this method can solve your problem

It can solve the immediate issue, I agree. But further usage of Conda may permanently require elevated permissions moving forward. Generally speaking it is best to start over, and figure out why elevated permissions were required at all.

While the rest of my post probably doesn't apply to you (sounds like you may be working with a Windows install), I write the following in hopes of aiding the non-windows folks:

Conda should be installed to/in a users home directory, by the user. Conda only needs write access to where you plan on installing it (and we all have write access to our own home directories). Example:

bash Mambaforge-Linux-x86_64.sh -b -p ~/mambaforge3

If you are not installing Conda to your home directory, then it is best to first create the directory where you are wanting to install it (using sudo), chowning that directory (using sudo), and then install Conda there (without using sudo). Example:

sudo mkdir -p /some/path
sudo chown -R me /some/path
bash Mambaforge-Linux-x86_64.sh -b -p /some/path/mambaforge3

@tianlinxu312
Copy link

tianlinxu312 commented Aug 1, 2023

I had the same error. My problem is due to having 2 version of condas installed on the laptop (when M1 chip just came out, I had to install two versions for tensorflow and pytorch). So the directories in the error message:

/home/user/.conda/pkgs

and

/home/user/.conda/envs

aren't really the directories where I installed my conda.

You can generate a .condarc file:

sudo chown your_username:your_username .condarc

and edit the paths in this file:

envs_dirs:
  - /correct_path_to_your_env_folder/envs
pkgs_dirs:
  - /correct_path_to_your_pkgs_folder/pkgs

Then it worked for me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests