Write permission error with a shared package cache folder

[emi80]

Current Behavior

I would like to share the package cache folder of a miniconda install with users in a linux group. I set group write permission for the folder and also set the guid bit. With a fresh install, the first user has no problem but the subsequent users get a permission error.

I looked within currently open issues but I could not find anything related to my problem. Please point me to anything relevant if I am wrong.

Thanks a lot.

Steps to Reproduce

# main install in /software/miniconda
$ chmod -R 02775 /software/miniconda/pkgs

# first user, download a simple package
$ conda install --download-only zip 
# OK

# second user
$ conda create -n test
Solving environment: done

NotWritableError: The current user does not have write permissions to a required path.
  path: /software/miniconda/pkgs/cache/47929eba.json
  uid: 32061
  gid: 32000

If you feel that permissions on this path are set incorrectly, you can manually
change them by executing

  $ sudo chown 32061:32000 /software/miniconda/pkgs/cache/47929eba.json

In general, it's not advisable to use 'sudo conda'.

Expected Behavior

There should maybe be a way to specify the umask when creating the files or to flag the package cache folder as shared in some way.

Environment Information

`conda info`

     active environment : None
            shell level : 0
       user config file : /home/user/.condarc
 populated config files : /software/miniconda/.condarc
          conda version : 4.5.1
    conda-build version : not installed
         python version : 2.7.14.final.0
       base environment : /software/miniconda  (read only)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/linux-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/pro/linux-64
                          https://repo.anaconda.com/pkgs/pro/noarch
          package cache : /software/miniconda/pkgs
       envs directories : /home/user/.conda/envs
                          /software/miniconda/envs
               platform : linux-64
             user-agent : conda/4.5.1 requests/2.18.4 CPython/2.7.14 Linux/3.10.0-693.5.2.el7.x86_64 debian/9 glibc/2.24
                UID:GID : 32061:32000
             netrc file : None
           offline mode : False

`conda config --show-sources`

==> /software/miniconda/.condarc <==
pkgs_dirs:
  - /software/miniconda/pkgs

`conda list --show-channel-urls`

# packages in environment at /software/miniconda:
#
# Name                    Version                   Build  Channel
asn1crypto                0.24.0                   py27_0    defaults
ca-certificates           2018.03.07                    0    defaults
certifi                   2018.4.16                py27_0    defaults
cffi                      1.11.4           py27h9745a5d_0    defaults
chardet                   3.0.4            py27hfa10054_1    defaults
conda                     4.5.1                    py27_0    defaults
conda-env                 2.6.0                h36134e3_1    defaults
cryptography              2.1.4            py27h6697b16_0    defaults
enum34                    1.1.6            py27h99a27e9_1    defaults
futures                   3.2.0            py27h7b459c0_0    defaults
idna                      2.6              py27h5722d68_1    defaults
ipaddress                 1.0.19                   py27_0    defaults
libedit                   3.1                  heed3624_0    defaults
libffi                    3.2.1                hd88cf55_4    defaults
libgcc-ng                 7.2.0                h7cc24e2_2    defaults
libstdcxx-ng              7.2.0                h7a57d05_2    defaults
ncurses                   6.0                  h9df7e31_2    defaults
openssl                   1.0.2o               h20670df_0    defaults
pip                       9.0.1            py27ha730c48_4    defaults
pycosat                   0.6.3            py27ha4109ae_0    defaults
pycparser                 2.18             py27hefa08c5_1    defaults
pyopenssl                 17.5.0           py27hcee3be0_0    defaults
pysocks                   1.6.7            py27he2db6d2_1    defaults
python                    2.7.14              h1571d57_29    defaults
readline                  7.0                  ha6073c6_4    defaults
requests                  2.18.4           py27hc5b0589_1    defaults
ruamel_yaml               0.15.35          py27h14c3975_1    defaults
setuptools                38.4.0                   py27_0    defaults
six                       1.11.0           py27h5f960f1_1    defaults
sqlite                    3.22.0               h1bed415_0    defaults
tk                        8.6.7                hc745277_3    defaults
urllib3                   1.22             py27ha55213b_0    defaults
wheel                     0.30.0           py27h2bc6bb2_1    defaults
yaml                      0.1.7                had09818_2    defaults
zlib                      1.2.11               ha838bed_2    defaults

[daveuu]

I attempted to do exactly the same thing and encountered the same problem, so I second this request. Sharing a pkg cache and environments seems useful to me.

Probably a duplicate of this #6953

My conda info:

    active environment : None
            shell level : 0
...
          conda version : 4.6.14
    conda-build version : not installed
         python version : 3.7.3.final.0
       base environment : /opt/conda  (read only)
...
          package cache : /mnt/conda/pkgs
       envs directories : /mnt/conda/envs
                          /home/.../.conda/envs
                          /opt/conda/envs
               platform : linux-64
             user-agent : conda/4.6.14 requests/2.21.0 CPython/3.7.3 Linux/4.15.0-55-generic ubuntu/18.04.2 glibc/2.27
                UID:GID : 876090092:876090092
             netrc file : None
           offline mode : False

[daveuu]

FWIW I think #6378 is also the same issue which was closed, on the assumption that the root cause was sudo-related. I think this older resolved sudo-related issue is not the same problem: #6576.

[daveuu]

So I came across a robust work around (or perhaps 'correct configuration') by using setuid on the pkgs directory as mentioned here for the similar issue mentioned above.

Assuming you have a group called condausers and users who want to use conda have been added, set up alternative paths for shared pkgs and envs. Set these two paths to the same filesystem so hard links can be used between pkg cache and environments.

sudo mkdir /mnt/conda
sudo mkdir /mnt/conda/{pkgs,envs}
sudo chgrp -R "condausers" /mnt/conda
# the +s makes files and directories created within inherit the group
sudo chmod -R g+ws /mnt/conda
# Allow conda to find these new locations for all users
echo "echo 'CONDA_PKGS_DIRS=/mnt/conda/pkgs' >> /etc/environment" | sudo bash
echo "echo 'CONDA_ENVS_PATH=/mnt/conda/envs' >> /etc/environment" | sudo bash

[adriendelsalle]

Hi @daveuu
I'm facing the same issue and your workaround didn't completely worked for me.

• you can use the gid bit to make the sub directories or files to inherit that specific group
• I also tried to use facl to force permissions on sub dirs/files created
• but when archives are expended from tarball, directories and files stay with the permissions they had before packing.. and if it's something x00, other users will have no rights on it

The only workaround I've found is to run chmod -R 777 mycondacache &>/dev/null to fix messy permissions silently.
Unfortunately it should be done at each conda create, conda install, conda upgrade etc. or maybe it is possible to introduce a hook after packages unpacking step (that is surely the same for each commands) to allow this fix.
And if I dream a bit more, add a shared-cache boolean option (or a shared-cache-perm to set octal permission value if declared) that could be used in the .condarc to run automatically a fix-permission script?

@kalefranz There are a lot of issues around this, do you think one of those propositions could do the job?
Thanks!

[xguse]

Still experiencing this on our private HPC. Has there been any progress? Updated recommended actions?

[adriendelsalle]

Hi! Never heard about a feature to help on this.
Anyone could help?

[trallnag]

I have the same problem. I have also tried using FACL but to no avail. Maybe I did something wrong?

Anyway: Another way to circumvent the issue is to delete the cache or to run conda clean -a before installing a package as a user.

[adriendelsalle]

Hi @trallnag, I don't understand "to no avail.", Could you please detail a bit more?

Your workaround for the permission error also lose the interest of a shared package cache. Doing that way you will download again and again the same packages instead of creating hardlinks against the same files located in the package cache directory.

Anyone in the conda team that could help on that?

Regards!

[daveuu]

So a solution could be to have a CLI argument to conda and or an option in .condarc specifying a permission mask to apply to packages when installed. That information would have to be plumbed through to extract_tarball (AFAIU) and applied with a block like this one. This would be close to tar's own:

--no-same-permissions
    Apply the user's umask when extracting permissions from the archive (default for ordinary users).

If I can find more time I could attempt a pull request (but I'm not familiar with the conda code base so no guarantees I'll have the time soon 😢 )

[oscarkey]

Hi @trallnag
Could you explain a little more about what you attempted using FACL?

[adriendelsalle]

Hi!
Is there a maintainer around to give us hope to see this issue solved?
Has a PR a chance to be merged if submitted?
Thanks

[gbbio]

Same issue here, and no real solid solution.

[Madave94]

I ran conda clean -a. Afterwards it worked.

Edit: So I didn't read trallnag's answer, but at least I can confirm that this way works.

[rainwoodman]

I am coming here hoping if there is a way to unshare the package cache folder.

Even though conda info shows two cache locations, one readonly for the root, one writeable under my home, somehow I am still getting permission errors. Should conda prefer to use the writeable cache directory instead?

Admittedly I am using a rather old version of conda (the version shipped by Fedora) -- were there recent improvements on this front?

     active environment : bccp
    active env location : /home/yfeng1/.conda/envs/bccp
            shell level : 1
       user config file : /home/yfeng1/.condarc
 populated config files : /usr/share/conda/condarc.d/defaults.yaml
                          /usr/share/conda/condarc.d/usr-local-conda-env.yaml
          conda version : 4.8.2
    conda-build version : not installed
         python version : 3.8.5.final.0
       virtual packages : __glibc=2.31
       base environment : /usr  (read only)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /var/cache/conda/pkgs
                          /home/yfeng1/.conda/pkgs
       envs directories : /usr/local/conda-envs
                          /home/yfeng1/.conda/envs
                          /usr/envs
               platform : linux-64
             user-agent : conda/4.8.2 requests/2.22.0 CPython/3.8.5 Linux/5.8.10-200.fc32.x86_64 fedora/32 glibc/2.31
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

[rainwoodman]

My solution to this is to disable the shared cache. I addec a condarc file (On Fedora it is /usr/share/conda/condarc.d/zz-local.yaml) that flipped the ordering of the per user cache directory and the root (readonly) cache directory.

pkgs_dirs:
 - ~/.conda/pkgs
 - /var/cache/conda/pkgs

envs_dirs:
  - /usr/local/conda-envs

allow_softlinks: False

I am not sure to what extent allow_softlinks helped; just added to be safe that nothing in the per user directory is a symlink to the readonly directory. I have also run conda clean --all as root to purge the root cache.

[nick-youngblut]

~/.conda/pkgs works, but it does have the downside of placing conda files outside of the conda install directory. I was hoping of creating user-specific pkgs directories inside the conda base dir via expansion of environmental variables; however, when I try /path/to/miniconda/install/${USERNAME}/pkgs, the new path created by conda is literally /path/to/miniconda/install/${USERNAME}/pkgs instead of changing ${USERNAME} to the user's username. I guess the conda docs are not accurate in that:

Conda expands environment variables in a subset of configuration settings. These are:

• envs_dirs
• pkgs_dirs

[tntmatthews]

Same issue here on our shared HPC environment - its a bummer that there is no clean solution (only the workaround of everyone having a local pkgs_dir).

[gqqnbig]

I got an impression that a local cache server may be used for the caching purpose.

[rainwoodman]

Like, a LAN mirror of interested channels, and then disable conda's cache?

[gqqnbig]

@rainwoodman yes. I guess that's the reason that the officials are hesitant in participating this discussion.

[daveuu]

Is that officially recommended in the documentation anywhere? Because, as described in this issue, the docs specifically recommend pointing different users to the same local path:
https://docs.anaconda.com/anaconda/user-guide/tasks/shared-pkg-cache/
Maybe that page in the documentation is out of date and should be removed or replaced with "do not do this" - but I'd prefer this issue to be fixed directly! I think a permissions solution for a local path is much simpler than setting up a conda repo mirror on the LAN.

[tdhopper]

I'm seeing this in a shared Jupyterhub instance as of Conda 4.10.3.

[gcastillo-ucsd]

Seeing this behavior in conda 4.10.3. It's causing problems in a shared research environment.

Our workaround thus far has been to set a default group ACL so that newly created objects are group read/write.

[Hoeze]

@chenghlee @jezdez Could you please give this issue some priority?
It's a really old issue, causes headaches and wastes tons of disk space.

[hoedt]

when I try /path/to/miniconda/install/${USERNAME}/pkgs, the new path created by conda is literally /path/to/miniconda/install/${USERNAME}/pkgs instead of changing ${USERNAME} to the user's username.

Seems like conda does not support brace expansion. Something like /path/to/miniconda/pkgs-$USER works just fine for me.

This being said, I think the problem is that the files in cache do not have group write permissions. The same holds for urls and urls.txt, but this is pretty easy to resolve by setting the permissions after first creation. The root for this problem seems to stem from the fact that these files are created with the default umask (see e.g., the implementation of touch, which is used in create_package_cache_directory). I did not manage to figure out where the files in the cache directory are created, but I assume this would be the same problem.

On UNIX systems, the default umask is 022, which means that files are only writeable by the user who created the file.
In python, this umask can be set and retrieved by means of os.umask(0o022) (not sure how this works on Windows).
So, by setting os.umask(0o002), it should be possible to create group-writeable files. For more control about the permissions, os.open could be used (inspiration by this SO answer).

By changing the source code from something like

with open(file, 'a'):
    pass

to

old_umask = os.umask(0o002)
os.close(os.open(file, os.O_CREAT, mode=0o664))
os.umask(old_umask)

I have this feeling that a lot of the problems w.r.t. permissions could be resolved. For the problems in this thread this would have to be combined with setting GUID on the pkgs directory and proper group management of course.

Assuming GUID has been set on the shared pkg directory, a temporary solution (on UNIX) might be to include something like umask 002 in your .bashrc or create an alias for conda that does something along the lines of alias conda='umask 002 && conda'. It would probably be cleaner to wrap this into a command that executes everything in a sub-shell, but you get the idea. Also note that these modifications would need to be applied for everyone who uses the shared pkg cache.

[simonrmonk]

Any updates on this? This would solve a lot of headaches for our team if this bug was fixed.

[cjops]

I was also misled by the documentation into thinking that a shared package cache on our HPC cluster was feasible. A fix would be very useful.

[Hoeze]

@jezdez Are there any plans on your side to investigate this issue?
Currently the docs are indeed misleading:
https://docs.anaconda.com/anaconda/user-guide/tasks/shared-pkg-cache/

I just did some checks how large the "$USER/.conda/pkgs" directories on our system are.
Turns out we waste ~950GB of storage for 28 users because of this bug.

[stanleyjs]

Just piling in - have a docker container that is built as root and then changed to $USER as an environment variable at runtime. The result is that the $USER has to chown -R the entire /opt/conda, even if they have group permissions. chowning a large environment takes forever
conda clean -a didn't help

[aersoares81]

Any news on this? I'm trying to install conda-libmamba-solver and keep bumping into this permissions problem.

[JohannesWiesner]

Same problem here, we could solve it by:

adding ACLs to the folder csp-tools/anaconda3/pkgs/cache. The ACL set the permission rw for the group domänen-benutzer as default on all files and directories inside that folder.

Our IT is however not really happy with this solution.

[awwad]

conda normally preserves file permissions in packages it installs. Sometimes, this includes world-writable files (which as far as we know should never be created by a package manager).

Some packages include such world-writable files (usually in the info/ directory). Multiple solutions are required: while these permissions should not be set in package building, it is unlikely that repository maintainers will go back and alter existing, published build artifacts, and it's necessary -- whether folks make build changes or not -- to prevent conda from producing these permissions on end user systems.

Cheng, Preston, and Seb think this is a moderately high priority; conda should never do this. It causes issues for users, and it should be fixed by, e.g., having conda respect umask, which by default will not permit o+w permissions.

Prior tickets include: #7057, #8200. There might be some others.

[LtDan33]

I'm moving this into the sprint candidates for the Anaconda team so we can move this closer to a solution. Per Seb's comment above its not something that conda should be doing and creates potential security issues along with general headaches.

[hoedt]

Why is this issue being hijacked? This was originally about multi-user installs. I fail to see how the file permission problem is even related to the original issue.

[dholth]

When conda-package-handling >= 2.0 is installed, conda uses the standard library to extract files instead of libarchive. This can affect permissions in some cases; so far we have noticed it in github codespaces, which may be using extended or sticky directory permissions. Is the new behavior better?

[pgramme]

I agree with @hoedt : the new issue title and description is now completely different from the original issue. It was about too strict permissions on JSON cache files. Now it became too loose permissions on unzipped pkgs files

@awwad, could you please roll back the change and create a new issue instead (related but different)? The related issues, priority, etc were assigned way before changing it

[awwad]

I'll assume this got confused with another issue somehow.... Sorry for that. Sure, reverted.

At a guess, I'd say that this is likely because a umask-checking solution might address both.

[g24swint]

December 2023, and I'm still seeing this issue with a relatively recent conda version 23.11.0. The multi-user install instructions for Linux should probably include some version of sticky bits getting set on the cache directory or at least instructions on how to turn off the shared cache

[suckrowPierre]

Jan 2024 still having the issue on our server. Only thing seeming to work is deleting the cache for creating a new env :(

[suckrowPierre]

when I try /path/to/miniconda/install/${USERNAME}/pkgs, the new path created by conda is literally /path/to/miniconda/install/${USERNAME}/pkgs instead of changing ${USERNAME} to the user's username.

Seems like conda does not support brace expansion. Something like /path/to/miniconda/pkgs-$USER works just fine for me.

This being said, I think the problem is that the files in cache do not have group write permissions. The same holds for urls and urls.txt, but this is pretty easy to resolve by setting the permissions after first creation. The root for this problem seems to stem from the fact that these files are created with the default umask (see e.g., the implementation of touch, which is used in create_package_cache_directory). I did not manage to figure out where the files in the cache directory are created, but I assume this would be the same problem.

On UNIX systems, the default umask is 022, which means that files are only writeable by the user who created the file. In python, this umask can be set and retrieved by means of os.umask(0o022) (not sure how this works on Windows). So, by setting os.umask(0o002), it should be possible to create group-writeable files. For more control about the permissions, os.open could be used (inspiration by this SO answer).

By changing the source code from something like

with open(file, 'a'):
    pass

to

old_umask = os.umask(0o002)
os.close(os.open(file, os.O_CREAT, mode=0o664))
os.umask(old_umask)

I have this feeling that a lot of the problems w.r.t. permissions could be resolved. For the problems in this thread this would have to be combined with setting GUID on the pkgs directory and proper group management of course.

Assuming GUID has been set on the shared pkg directory, a temporary solution (on UNIX) might be to include something like umask 002 in your .bashrc or create an alias for conda that does something along the lines of alias conda='umask 002 && conda'. It would probably be cleaner to wrap this into a command that executes everything in a sub-shell, but you get the idea. Also note that these modifications would need to be applied for everyone who uses the shared pkg cache.

Is this the location where the json are written to the cache?

conda/conda/gateways/disk/create.py

Line 111 in 72fe69d

def write_as_json_to_file(file_path, obj):

[dholth]

IIUC the Unix-way of setting umask is to have everyone who wants to use a shared folder, set umask in bashrc. Awfully error-prone. Is there a way to set a directory permission or extended permission?