--symlink-app-data: causes pip / wheel / setuptools to link to random tmp dir #2000

asottile · 2020-10-28T23:00:38Z

Issue

I expect --symlink-app-data to work when creating a virtualenv

Environment

I've got a docker image for you :)

FROM ubuntu:focal
RUN : \
    && apt-get update \
    && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
        python3 \
        python3-distutils \
        curl \
        ca-certificates \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

ENV PATH=/venv/bin:$PATH
RUN : \
    && curl --location --silent --output /virtualenv.pyz https://bootstrap.pypa.io/virtualenv.pyz \
    && python3 /virtualenv.pyz /venv \
    && pip install virtualenv \
    && rm /virtualenv.pyz

ENV \
    VIRTUALENV_OVERRIDE_APP_DATA=/opt/virtualenvdata \
    VIRTUALENV_SYMLINK_APP_DATA=1

# prepopulate appdata cache
RUN virtualenv /wat --symlink-app-data && rm -rf /wat

USER 1000
RUN : \
    && virtualenv /tmp/test -vvv --with-traceback \
    && ls -al /tmp/test/lib/python3.8/site-packages \
    && /tmp/test/bin/pip install astpretty

Output of the virtual environment creation

Make sure to run the creation with -vvv --with-traceback:

$ docker build -t test .
STEP 1: FROM ubuntu:focal
STEP 2: RUN :     && apt-get update     && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends         python3         python3-distutils         curl         ca-certificates     && apt-get clean     && rm -rf /var/lib/apt/lists/*
--> Using cache cd34708b98bedf0d512db31127ae88edd1288a868a640550a23e3fc125d9c2b1
--> cd34708b98b
STEP 3: ENV PATH=/venv/bin:$PATH
--> Using cache d237cc02ce7a5abed4e67086eb0c245fa4e6406a000a95cf0af1d9209751d162
--> d237cc02ce7
STEP 4: RUN :     && curl --location --silent --output /virtualenv.pyz https://bootstrap.pypa.io/virtualenv.pyz     && python3 /virtualenv.pyz /venv     && pip install virtualenv     && rm /virtualenv.pyz
--> Using cache f0afa26da50e854119e318071290ababf95bae97337a870a04a279e2ce8cdb00
--> f0afa26da50
STEP 5: ENV     VIRTUALENV_OVERRIDE_APP_DATA=/opt/virtualenvdata     VIRTUALENV_SYMLINK_APP_DATA=1
--> Using cache 4489d1eb9aa51f8a463dffef6cf761258ae5e4ab8667e978fa32af2e70618d6a
--> 4489d1eb9aa
STEP 6: RUN virtualenv /wat --symlink-app-data && rm -rf /wat
--> Using cache 7b1d35bfcc84bc716caa075c655cf566bc0e8f62a9f18f8f0e5e88f2179d2502
--> 7b1d35bfcc8
STEP 7: USER 1000
--> Using cache 6d3a71bdb4a0dad209fbbac17565525e02d99457908a041f8352e72050b91376
--> 6d3a71bdb4a
STEP 8: RUN :     && virtualenv /tmp/test -vvv --with-traceback     && ls -al /tmp/test/lib/python3.8/site-packages     && /tmp/test/bin/pip install astpretty
137 setup logging to NOTSET [DEBUG report:43]
137 app data folder /opt/virtualenvdata has no write access [DEBUG __init__:37]
139 created temporary app data folder /tmp/tmpr8o4y3qh [DEBUG via_tempdir:14]
153 find interpreter for spec PythonSpec(path=/venv/bin/python) [INFO builtin:51]
153 discover exe for PythonInfo(spec=CPython3.8.5.final.0-64, exe=/venv/bin/python, platform=linux, version='3.8.5 (default, Jul 28 2020, 12:59:40) \n[GCC 9.3.0]', encoding_fs_io=utf-8-utf-8) in /usr [DEBUG py_info:371]
154 filesystem is case-sensitive [DEBUG info:28]
159 Attempting to acquire lock 140572762775952 on /tmp/tmpr8o4y3qh/py_info/1/df0893f56f349688326838aaeea0de204df53a132722cbd565e54b24a8fec5f6.lock [DEBUG filelock:270]
159 Lock 140572762775952 acquired on /tmp/tmpr8o4y3qh/py_info/1/df0893f56f349688326838aaeea0de204df53a132722cbd565e54b24a8fec5f6.lock [INFO filelock:274]
160 get interpreter info via cmd: /usr/bin/python3.8 /venv/lib/python3.8/site-packages/virtualenv/discovery/py_info.py [DEBUG cached_py_info:87]
255 wrote python info of /usr/bin/python3.8 at /tmp/tmpr8o4y3qh/py_info/1/df0893f56f349688326838aaeea0de204df53a132722cbd565e54b24a8fec5f6.json [DEBUG via_disk_folder:162]
255 Attempting to release lock 140572762775952 on /tmp/tmpr8o4y3qh/py_info/1/df0893f56f349688326838aaeea0de204df53a132722cbd565e54b24a8fec5f6.lock [DEBUG filelock:315]
255 Lock 140572762775952 released on /tmp/tmpr8o4y3qh/py_info/1/df0893f56f349688326838aaeea0de204df53a132722cbd565e54b24a8fec5f6.lock [INFO filelock:318]
255 proposed PythonInfo(spec=CPython3.8.5.final.0-64, system=/usr/bin/python3.8, exe=/venv/bin/python, platform=linux, version='3.8.5 (default, Jul 28 2020, 12:59:40) \n[GCC 9.3.0]', encoding_fs_io=utf-8-utf-8) [INFO builtin:57]
256 accepted PythonInfo(spec=CPython3.8.5.final.0-64, system=/usr/bin/python3.8, exe=/venv/bin/python, platform=linux, version='3.8.5 (default, Jul 28 2020, 12:59:40) \n[GCC 9.3.0]', encoding_fs_io=utf-8-utf-8) [DEBUG builtin:59]
331 create virtual environment via CPython3Posix(dest=/tmp/test, clear=False, global=False) [INFO session:52]
331 create folder /tmp/test/bin [DEBUG _sync:25]
332 create folder /tmp/test/lib/python3.8/site-packages [DEBUG _sync:25]
333 write /tmp/test/pyvenv.cfg [DEBUG pyenv_cfg:34]
333 	home = /usr [DEBUG pyenv_cfg:38]
333 	implementation = CPython [DEBUG pyenv_cfg:38]
333 	version_info = 3.8.5.final.0 [DEBUG pyenv_cfg:38]
333 	virtualenv = 20.1.0 [DEBUG pyenv_cfg:38]
333 	include-system-site-packages = false [DEBUG pyenv_cfg:38]
333 	base-prefix = /usr [DEBUG pyenv_cfg:38]
333 	base-exec-prefix = /usr [DEBUG pyenv_cfg:38]
333 	base-executable = /usr/bin/python3.8 [DEBUG pyenv_cfg:38]
334 symlink /usr/bin/python3.8 to /tmp/test/bin/python [DEBUG _sync:44]
334 create virtualenv import hook file /tmp/test/lib/python3.8/site-packages/_virtualenv.pth [DEBUG api:95]
335 create /tmp/test/lib/python3.8/site-packages/_virtualenv.py [DEBUG api:98]
336 ============================== target debug ============================== [DEBUG session:54]
336 debug via /tmp/test/bin/python /venv/lib/python3.8/site-packages/virtualenv/create/debug.py [DEBUG creator:213]
336 {
  "sys": {
    "executable": "/tmp/test/bin/python",
    "_base_executable": "/tmp/test/bin/python",
    "prefix": "/tmp/test",
    "base_prefix": "/usr",
    "real_prefix": null,
    "exec_prefix": "/tmp/test",
    "base_exec_prefix": "/usr",
    "path": [
      "/usr/lib/python38.zip",
      "/usr/lib/python3.8",
      "/usr/lib/python3.8/lib-dynload",
      "/tmp/test/lib/python3.8/site-packages"
    ],
    "meta_path": [
      "<class '_virtualenv._Finder'>",
      "<class '_frozen_importlib.BuiltinImporter'>",
      "<class '_frozen_importlib.FrozenImporter'>",
      "<class '_frozen_importlib_external.PathFinder'>"
    ],
    "fs_encoding": "utf-8",
    "io_encoding": "utf-8"
  },
  "version": "3.8.5 (default, Jul 28 2020, 12:59:40) \n[GCC 9.3.0]",
  "makefile_filename": "/usr/lib/python3.8/config-3.8-x86_64-linux-gnu/Makefile",
  "os": "<module 'os' from '/usr/lib/python3.8/os.py'>",
  "site": "<module 'site' from '/usr/lib/python3.8/site.py'>",
  "datetime": "<module 'datetime' from '/usr/lib/python3.8/datetime.py'>",
  "math": "<module 'math' (built-in)>",
  "json": "<module 'json' from '/usr/lib/python3.8/json/__init__.py'>"
} [DEBUG session:55]
385 add seed packages via FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=symlink, app_data_dir=/tmp/tmpr8o4y3qh) [INFO session:59]
386 install pip from wheel /venv/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-20.2.4-py2.py3-none-any.whl via SymlinkPipInstall [DEBUG via_app_data:51]
388 install setuptools from wheel /venv/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/setuptools-50.3.2-py3-none-any.whl via SymlinkPipInstall [DEBUG via_app_data:51]
388 install wheel from wheel /venv/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/wheel-0.35.1-py2.py3-none-any.whl via SymlinkPipInstall [DEBUG via_app_data:51]
389 Attempting to acquire lock 140572759983872 on /tmp/tmpr8o4y3qh/wheel/3.8/image/1/SymlinkPipInstall/wheel-0.35.1-py2.py3-none-any.lock [DEBUG filelock:270]
389 Attempting to acquire lock 140572772360448 on /tmp/tmpr8o4y3qh/wheel/3.8/image/1/SymlinkPipInstall/pip-20.2.4-py2.py3-none-any.lock [DEBUG filelock:270]
390 Lock 140572759983872 acquired on /tmp/tmpr8o4y3qh/wheel/3.8/image/1/SymlinkPipInstall/wheel-0.35.1-py2.py3-none-any.lock [INFO filelock:274]
390 build install image for wheel-0.35.1-py2.py3-none-any.whl to /tmp/tmpr8o4y3qh/wheel/3.8/image/1/SymlinkPipInstall/wheel-0.35.1-py2.py3-none-any [DEBUG base:52]
390 Attempting to acquire lock 140572759984064 on /tmp/tmpr8o4y3qh/wheel/3.8/image/1/SymlinkPipInstall/setuptools-50.3.2-py3-none-any.lock [DEBUG filelock:270]
390 Lock 140572772360448 acquired on /tmp/tmpr8o4y3qh/wheel/3.8/image/1/SymlinkPipInstall/pip-20.2.4-py2.py3-none-any.lock [INFO filelock:274]
390 Lock 140572759984064 acquired on /tmp/tmpr8o4y3qh/wheel/3.8/image/1/SymlinkPipInstall/setuptools-50.3.2-py3-none-any.lock [INFO filelock:274]
390 build install image for pip-20.2.4-py2.py3-none-any.whl to /tmp/tmpr8o4y3qh/wheel/3.8/image/1/SymlinkPipInstall/pip-20.2.4-py2.py3-none-any [DEBUG base:52]
394 build install image for setuptools-50.3.2-py3-none-any.whl to /tmp/tmpr8o4y3qh/wheel/3.8/image/1/SymlinkPipInstall/setuptools-50.3.2-py3-none-any [DEBUG base:52]
513 Attempting to release lock 140572759983872 on /tmp/tmpr8o4y3qh/wheel/3.8/image/1/SymlinkPipInstall/wheel-0.35.1-py2.py3-none-any.lock [DEBUG filelock:315]
513 Lock 140572759983872 released on /tmp/tmpr8o4y3qh/wheel/3.8/image/1/SymlinkPipInstall/wheel-0.35.1-py2.py3-none-any.lock [INFO filelock:318]
515 generated console scripts wheel wheel-3.8 wheel3 wheel3.8 [DEBUG base:48]
947 Attempting to release lock 140572759984064 on /tmp/tmpr8o4y3qh/wheel/3.8/image/1/SymlinkPipInstall/setuptools-50.3.2-py3-none-any.lock [DEBUG filelock:315]
947 Lock 140572759984064 released on /tmp/tmpr8o4y3qh/wheel/3.8/image/1/SymlinkPipInstall/setuptools-50.3.2-py3-none-any.lock [INFO filelock:318]
950 generated console scripts easy_install3.8 easy_install easy_install-3.8 easy_install3 [DEBUG base:48]
1712 Attempting to release lock 140572772360448 on /tmp/tmpr8o4y3qh/wheel/3.8/image/1/SymlinkPipInstall/pip-20.2.4-py2.py3-none-any.lock [DEBUG filelock:315]
1712 Lock 140572772360448 released on /tmp/tmpr8o4y3qh/wheel/3.8/image/1/SymlinkPipInstall/pip-20.2.4-py2.py3-none-any.lock [INFO filelock:318]
1715 generated console scripts pip pip3.8 pip-3.8 pip3 [DEBUG base:48]
1715 add activators for Bash, CShell, Fish, PowerShell, Python, Xonsh [INFO session:64]
1719 write /tmp/test/pyvenv.cfg [DEBUG pyenv_cfg:34]
1719 	home = /usr [DEBUG pyenv_cfg:38]
1719 	implementation = CPython [DEBUG pyenv_cfg:38]
1719 	version_info = 3.8.5.final.0 [DEBUG pyenv_cfg:38]
1719 	virtualenv = 20.1.0 [DEBUG pyenv_cfg:38]
1719 	include-system-site-packages = false [DEBUG pyenv_cfg:38]
1719 	base-prefix = /usr [DEBUG pyenv_cfg:38]
1719 	base-exec-prefix = /usr [DEBUG pyenv_cfg:38]
1719 	base-executable = /usr/bin/python3.8 [DEBUG pyenv_cfg:38]
1719 remove temporary app data folder /tmp/tmpr8o4y3qh [DEBUG via_tempdir:20]
1910 created virtual environment CPython3.8.5.final.0-64 in 1775ms
  creator CPython3Posix(dest=/tmp/test, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=symlink, app_data_dir=/tmp/tmpr8o4y3qh)
    added seed packages: pip==20.2.4, setuptools==50.3.2, wheel==0.35.1
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator [WARNING __main__:17]
total 76
drwxr-xr-x 3 1000 root 4096 Oct 28 22:58 .
drwxr-xr-x 3 1000 root 4096 Oct 28 22:58 ..
drwxr-xr-x 2 1000 root 4096 Oct 28 22:58 __pycache__
lrwxrwxrwx 1 1000 root   99 Oct 28 22:58 _distutils_hack -> /tmp/tmpr8o4y3qh/wheel/3.8/image/1/SymlinkPipInstall/setuptools-50.3.2-py3-none-any/_distutils_hack
-rw-r--r-- 1 1000 root   18 Oct 28 22:58 _virtualenv.pth
-rw-r--r-- 1 1000 root 5662 Oct 28 22:58 _virtualenv.py
lrwxrwxrwx 1 1000 root  108 Oct 28 22:58 distutils-precedence.pth -> /tmp/tmpr8o4y3qh/wheel/3.8/image/1/SymlinkPipInstall/setuptools-50.3.2-py3-none-any/distutils-precedence.pth
lrwxrwxrwx 1 1000 root   99 Oct 28 22:58 easy_install.py -> /tmp/tmpr8o4y3qh/wheel/3.8/image/1/SymlinkPipInstall/setuptools-50.3.2-py3-none-any/easy_install.py
lrwxrwxrwx 1 1000 root   84 Oct 28 22:58 pip -> /tmp/tmpr8o4y3qh/wheel/3.8/image/1/SymlinkPipInstall/pip-20.2.4-py2.py3-none-any/pip
lrwxrwxrwx 1 1000 root  101 Oct 28 22:58 pip-20.2.4.dist-info -> /tmp/tmpr8o4y3qh/wheel/3.8/image/1/SymlinkPipInstall/pip-20.2.4-py2.py3-none-any/pip-20.2.4.dist-info
lrwxrwxrwx 1 1000 root  102 Oct 28 22:58 pip-20.2.4.virtualenv -> /tmp/tmpr8o4y3qh/wheel/3.8/image/1/SymlinkPipInstall/pip-20.2.4-py2.py3-none-any/pip-20.2.4.virtualenv
lrwxrwxrwx 1 1000 root   97 Oct 28 22:58 pkg_resources -> /tmp/tmpr8o4y3qh/wheel/3.8/image/1/SymlinkPipInstall/setuptools-50.3.2-py3-none-any/pkg_resources
lrwxrwxrwx 1 1000 root   94 Oct 28 22:58 setuptools -> /tmp/tmpr8o4y3qh/wheel/3.8/image/1/SymlinkPipInstall/setuptools-50.3.2-py3-none-any/setuptools
lrwxrwxrwx 1 1000 root  111 Oct 28 22:58 setuptools-50.3.2.dist-info -> /tmp/tmpr8o4y3qh/wheel/3.8/image/1/SymlinkPipInstall/setuptools-50.3.2-py3-none-any/setuptools-50.3.2.dist-info
lrwxrwxrwx 1 1000 root  112 Oct 28 22:58 setuptools-50.3.2.virtualenv -> /tmp/tmpr8o4y3qh/wheel/3.8/image/1/SymlinkPipInstall/setuptools-50.3.2-py3-none-any/setuptools-50.3.2.virtualenv
lrwxrwxrwx 1 1000 root   88 Oct 28 22:58 wheel -> /tmp/tmpr8o4y3qh/wheel/3.8/image/1/SymlinkPipInstall/wheel-0.35.1-py2.py3-none-any/wheel
lrwxrwxrwx 1 1000 root  105 Oct 28 22:58 wheel-0.35.1.dist-info -> /tmp/tmpr8o4y3qh/wheel/3.8/image/1/SymlinkPipInstall/wheel-0.35.1-py2.py3-none-any/wheel-0.35.1.dist-info
lrwxrwxrwx 1 1000 root  106 Oct 28 22:58 wheel-0.35.1.virtualenv -> /tmp/tmpr8o4y3qh/wheel/3.8/image/1/SymlinkPipInstall/wheel-0.35.1-py2.py3-none-any/wheel-0.35.1.virtualenv
Traceback (most recent call last):
  File "/tmp/test/bin/pip", line 5, in <module>
    from pip._internal.cli.main import main
ModuleNotFoundError: No module named 'pip'
Error: error building at STEP "RUN :     && virtualenv /tmp/test -vvv --with-traceback     && ls -al /tmp/test/lib/python3.8/site-packages     && /tmp/test/bin/pip install astpretty": error while running runtime: exit status 1

Note here that I'm trying to populate the cache using the root user, and then having an unprivileged user utilize that cache later on. I'd like to use this to speed up pre-commit.ci by ~80-90% for python environment upload/download

The text was updated successfully, but these errors were encountered:

asottile · 2020-10-28T23:02:21Z

so I notice that it skips the appdir if it isn't writeable -- I would like to be able to reuse a readonly cache

so I guess there's both a bug and a feature request here:

[bug] if symlink app data is not allowed, it should not symlink to some arbitrary temporary directory
[feature] allow readonly app data cache directory

gaborbernat · 2020-10-28T23:06:40Z

If the app data is read only we must fallback to copy or just raise error?

asottile · 2020-10-28T23:21:38Z

Ideally I would like it to raise an error only if it needs to write to the cache, but allow symlinking into a readonly cache directory

gaborbernat · 2020-10-28T23:25:49Z

Feel free to raise a PR 👍 achieving this.

asottile · 2020-10-28T23:33:22Z

cool cool, I'll see what I can do -- will probably do two passes (1) fix the bug (2) add the feature

asottile added the bug label Oct 28, 2020

asottile mentioned this issue Nov 16, 2020

Add --readonly-app-data option #2009

Merged

5 tasks

gaborbernat closed this as completed in #2009 Nov 21, 2020

pypa locked and limited conversation to collaborators Jan 14, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

--symlink-app-data: causes pip / wheel / setuptools to link to random tmp dir #2000

--symlink-app-data: causes pip / wheel / setuptools to link to random tmp dir #2000

asottile commented Oct 28, 2020

asottile commented Oct 28, 2020

gaborbernat commented Oct 28, 2020

asottile commented Oct 28, 2020

gaborbernat commented Oct 28, 2020

asottile commented Oct 28, 2020

--symlink-app-data: causes pip / wheel / setuptools to link to random tmp dir #2000

--symlink-app-data: causes pip / wheel / setuptools to link to random tmp dir #2000

Comments

asottile commented Oct 28, 2020

asottile commented Oct 28, 2020

gaborbernat commented Oct 28, 2020

asottile commented Oct 28, 2020

gaborbernat commented Oct 28, 2020

asottile commented Oct 28, 2020