PermissionError when creating data_dir when user doesn't have permissions

[leducvin]

Description

Version 0.16.2 worked, but 0.17.1 tries to create a directory in whatever appdirs thinks is an appropriate cache directory.

For my flask app's service, the user is a no-login user, and somehow this resolves to /var/www/.
The user doesn't have permissions to create a directory there, so the flask app crashes with a PermissionError raised from skimage.

I specified SKIMAGE_DATADIR=/tmp/.skimage_cache, and ended up seeing #4660, as my flask app starts multiple processes.

Way to reproduce

import appdirs
# Monkey-patch appdirs.user_cache_dir to an unwritable location
appdirs.user_cache_dir = lambda _: '/var/log'
import skimage

Cannot create data cache folder '/var/log/0.17.1'. Will not be able to download remote data files. Use environment variable 'SKIMAGE_DATADIR' to specify another directory.
Traceback (most recent call last):
  File "skimage_issue.py", line 4, in <module>
    import skimage
  File "/home/vleduc/python/virtualenvs/py37/lib/python3.7/site-packages/skimage/__init__.py", line 133, in <module>
    from .data import data_dir
  File "/home/vleduc/python/virtualenvs/py37/lib/python3.7/site-packages/skimage/data/__init__.py", line 90, in <module>
    os.makedirs(data_dir, exist_ok=True)
  File "/home/vleduc/python/virtualenvs/py37/lib/python3.7/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/home/vleduc/python/virtualenvs/py37/lib/python3.7/os.py", line 223, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/var/log/0.17.1'

Version information

# Paste the output of the following python commands
from __future__ import print_function
import sys; print(sys.version)
import platform; print(platform.platform())
import skimage; print("scikit-image version: {}".format(skimage.__version__))
import numpy; print("numpy version: {}".format(numpy.__version__))

3.7.5 (default, Nov  7 2019, 10:50:52) 
[GCC 8.3.0]
Linux-4.15.0-99-generic-x86_64-with-Ubuntu-18.04-bionic
scikit-image version: 0.17.1
numpy version: 1.18.4

[hmaarrfk]

Can you tell me if you start multiple processes before or after you import sxikit image?

[leouieda]

@hmaarrfk it seems that skimage is trying to create the data dir at import time in https://github.com/scikit-image/scikit-image/blob/master/skimage/data/__init__.py#L90 Is that necessary? Could it be delayed to the _fetch function instead (with a check to see if it already exists)?

On the Pooch side, if we don't raise an exception with a permission error (only issue a warning as seen above). So it wouldn't cause any problems unless the user wants to load sample data.

[hmaarrfk]

Should we open an issue in appdirs regarding using /tmp if the home dir is not writable?

Opened one ActiveState/appdirs#146

[sciunto]

In a more generic way, tempfile.gettempdir()

[hmaarrfk]

This is definitely an issue we should workaround today.

I'm just waiting until we merge in an other pull request relating to this new feature of ours.

[leducvin]

@hmaarrfk Thanks for looking into it! I think multiple processes are started after skimage is imported. Although the problem occurs at import time.

Yeah I'm not sure if this should be an issue in appdirs, pooch or scikit-image. Although from my point of view as a user of scikit-image, I feel the library might be trying to do much for me without asking. The error states:

Will not be able to download remote data files. Use environment variable 'SKIMAGE_DATADIR' to specify another directory.

, but I don't even want to use this functionality. In my use case, I don't need it.

[rfezzani]

What about simply wrap the directory creation in a try ... except ... bloc?

[hmaarrfk]

@leducvin you have been heard loud and clear, and we are working toward a solution that will work well with python 3.6 (though that isn't clear how we would do it).

Unfortunately, for now, this isn't an issue that we can solve right way.

We have a few contradictory requirements:

1. Shipping more examples to our users.
2. Enablign builds in applications such as your own.

For now, I would suggest you skip

• 0.17.1 (hard dependency on pooch)
• 0.17.2 (soft dependency on pooch, but if pooch is installed, the same problem will happen)

You may you use !=0.17.1,!=0.17.2

Rest assured that we are working to resolve this breaking change for your application.

[leducvin]

No worries. I already started using !=0.17.1, and that is working fine. I'll update to !=0.17.1,!=0.17.2. Thanks for the heads up.

As mentioned in ActiveState/appdirs#146, setting a $HOME env variable to a writable dir should also be a workaround, although I haven't tried it at this time.

[hmaarrfk]

I agree that this should not cause failure.

0.17.2 doesn't have the pooch requirement, but pooch may be installed by an other of your dependencies by accident.

We just need to balance a few other things to get it right. In my mind, you shouldn't have to use any configuration to get going in your usecase.

The only time you should set that environment variable is if you want to have a shared store of our future GB large datasets.

[sklipnoty]

When using a AWS Lambda, still is also a problem.

[mimranfaruqi]

@hmaarrfk Thanks for looking into it! I think multiple processes are started after skimage is imported. Although the problem occurs at import time.

Yeah I'm not sure if this should be an issue in appdirs, pooch or scikit-image. Although from my point of view as a user of scikit-image, I feel the library might be trying to do much for me without asking. The error states:

Will not be able to download remote data files. Use environment variable 'SKIMAGE_DATADIR' to specify another directory.

, but I don't even want to use this functionality. In my use case, I don't need it.

I was using this in my docker container. Setting up the environment variable solved my problem!

Thanks!

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/why-does-init-py-attempt-to-download-sample-data/79665/4

[lagru]

Reproducable with SKIMAGE_DATADIR=/etc/skimage python -c 'import skimage.data'. Assuming that /etc/skimage) isn't writable this will produce give

Traceback (most recent call last):
File "<string>", line 1, in <module>
File "~/Res/scikit-image/skimage/__init__.py", line 141, in <module>
from .data import data_dir
File "<frozen importlib._bootstrap>", line 1231, in _handle_fromlist
File "~/.local/lib/venv/skimagedev/lib/python3.11/site-packages/lazy_loader/__init__.py", line76, in __getattr__
submod = importlib.import_module(submod_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "~/.local/lib/micromamba/envs/calc/lib/python3.11/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "~/Res/scikit-image/skimage/data/_fetchers.py", line 263, in <module>
_init_pooch()
File "~/Res/scikit-image/skimage/data/_fetchers.py", line 240, in _init_pooch
os.makedirs(data_dir, exist_ok=True)
File "<frozen os>", line 215, in makedirs
File "<frozen os>", line 215, in makedirs
File "<frozen os>", line 225, in makedirs
PermissionError: [Errno 13] Permission denied: '/etc/skimage'

[lagru]

I'll look into fixing this.