Intermittent error while installing poetry in a gitlab-ci process.

[fabiobatalha]

• Poetry version: 1.3.2

• Python version: 3.9

• OS version and name: Python3.9 official image

• [ * ] I am on the latest stable Poetry version, installed using a recommended method.

• [ * ] I have searched the issues of this repo and believe that this is not a duplicate.

• [ * ] I have consulted the FAQ and blog for any relevant entries or release notes.

• [ * ] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

I'm having an intermittent error while installing a package using poetry in a standard/official Python3.9 image in a gitlab-ci job.

Running with gitlab-runner 15.6.1 (133d7e76)
  on runners01.prod.beluga YLknxwxw
  feature flags: FF_NETWORK_PER_BUILD:true
Resolving secrets 00:00
Preparing the "docker" executor 00:03
Using Docker executor with image python:3.9 ...
Pulling docker image python:3.9 ...
Using docker image sha256:8f1c967991c4c44b126c74b97427c57242e5f1b624ee3a0b1842c3284f9302da for python:3.9 with digest docker.io/library/python@sha256:5d0b605039499444251643500686a6e52f2549f4bb1b16cbbb67a4ea04f4c306 ...
Preparing environment 00:01
Running on runner-ylknxwxw-project-206-concurrent-0 via runners01.prod.novalocal...
Getting source from Git repository 00:02
Fetching changes...
Reinitialized existing Git repository in /builds/erudit/recherche/cyber/coshs/.git/
Checking out 8ceea06e as 0.2.1...
Removing .pytest_cache/
Removing src/research_data_manager/__pycache__/
Removing src/research_data_manager/processing/__pycache__/
Removing src/research_data_manager/processing/crkn/__pycache__/
Removing tests/__pycache__/
Skipping Git submodules setup
Executing "step_script" stage of the job script 00:28
Using docker image sha256:8f1c967991c4c44b126c74b97427c57242e5f1b624ee3a0b1842c3284f9302da for python:3.9 with digest docker.io/library/python@sha256:5d0b605039499444251643500686a6e52f2549f4bb1b16cbbb67a4ea04f4c306 ...
$ pip install poetry
Collecting poetry
  Downloading poetry-1.3.2-py3-none-any.whl (218 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 218.9/218.9 KB 13.3 MB/s eta 0:00:00
Collecting importlib-metadata<5.0,>=4.4
  Downloading importlib_metadata-4.13.0-py3-none-any.whl (23 kB)
Collecting virtualenv!=20.4.5,!=20.4.6,<21.0.0,>=20.4.3
  Downloading virtualenv-20.17.1-py3-none-any.whl (8.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.8/8.8 MB 46.3 MB/s eta 0:00:00
Collecting crashtest<0.5.0,>=0.4.1
  Downloading crashtest-0.4.1-py3-none-any.whl (7.6 kB)
Collecting jsonschema<5.0.0,>=4.10.0
  Downloading jsonschema-4.17.3-py3-none-any.whl (90 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 90.4/90.4 KB 5.6 MB/s eta 0:00:00
Collecting requests<3.0,>=2.18
  Downloading requests-2.28.2-py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 KB 3.1 MB/s eta 0:00:00
Collecting shellingham<2.0,>=1.5
  Downloading shellingham-1.5.0.post1-py2.py3-none-any.whl (9.4 kB)
Collecting dulwich<0.21.0,>=0.20.46
  Downloading dulwich-0.20.50-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (499 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 499.4/499.4 KB 18.3 MB/s eta 0:00:00
Collecting filelock<4.0.0,>=3.8.0
  Downloading filelock-3.9.0-py3-none-any.whl (9.7 kB)
Collecting pkginfo<2.0,>=1.5
  Downloading pkginfo-1.9.6-py3-none-any.whl (30 kB)
Collecting poetry-plugin-export<2.0.0,>=1.2.0
  Downloading poetry_plugin_export-1.2.0-py3-none-any.whl (10 kB)
Collecting requests-toolbelt<0.11.0,>=0.9.1
  Downloading requests_toolbelt-0.10.1-py2.py3-none-any.whl (54 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.5/54.5 KB 3.6 MB/s eta 0:00:00
Collecting cachecontrol[filecache]<0.13.0,>=0.12.9
  Downloading CacheControl-0.12.11-py2.py3-none-any.whl (21 kB)
Collecting cleo<3.0.0,>=2.0.0
  Downloading cleo-2.0.1-py3-none-any.whl (77 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77.3/77.3 KB 5.5 MB/s eta 0:00:00
Collecting tomli<3.0.0,>=2.0.1
  Downloading tomli-2.0.1-py3-none-any.whl (12 kB)
Collecting urllib3<2.0.0,>=1.26.0
  Downloading urllib3-1.26.14-py2.py3-none-any.whl (140 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.6/140.6 KB 9.2 MB/s eta 0:00:00
Collecting html5lib<2.0,>=1.0
  Downloading html5lib-1.1-py2.py3-none-any.whl (112 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 112.2/112.2 KB 7.2 MB/s eta 0:00:00
Collecting lockfile<0.13.0,>=0.12.2
  Downloading lockfile-0.12.2-py2.py3-none-any.whl (13 kB)
Collecting platformdirs<3.0.0,>=2.5.2
  Downloading platformdirs-2.6.2-py3-none-any.whl (14 kB)
Collecting pexpect<5.0.0,>=4.7.0
  Downloading pexpect-4.8.0-py2.py3-none-any.whl (59 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 59.0/59.0 KB 3.6 MB/s eta 0:00:00
Collecting tomlkit!=0.11.2,!=0.11.3,<1.0.0,>=0.11.1
  Downloading tomlkit-0.11.6-py3-none-any.whl (35 kB)
Collecting keyring<24.0.0,>=23.9.0
  Downloading keyring-23.13.1-py3-none-any.whl (37 kB)
Collecting packaging>=20.4
  Downloading packaging-23.0-py3-none-any.whl (42 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.7/42.7 KB 2.6 MB/s eta 0:00:00
Collecting trove-classifiers>=2022.5.19
  Downloading trove_classifiers-2023.1.12-py3-none-any.whl (13 kB)
Collecting poetry-core==1.4.0
  Downloading poetry_core-1.4.0-py3-none-any.whl (546 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 546.4/546.4 KB 20.6 MB/s eta 0:00:00
Collecting msgpack>=0.5.2
  Downloading msgpack-1.0.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (322 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 322.4/322.4 KB 16.0 MB/s eta 0:00:00
Collecting rapidfuzz<3.0.0,>=2.2.0
  Downloading rapidfuzz-2.13.7-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB 32.3 MB/s eta 0:00:00
Collecting webencodings
  Downloading webencodings-0.5.1-py2.py3-none-any.whl (11 kB)
Collecting six>=1.9
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting zipp>=0.5
  Downloading zipp-3.11.0-py3-none-any.whl (6.6 kB)
Collecting attrs>=17.4.0
  Downloading attrs-22.2.0-py3-none-any.whl (60 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.0/60.0 KB 3.8 MB/s eta 0:00:00
Collecting pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0
  Downloading pyrsistent-0.19.3-py3-none-any.whl (57 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.5/57.5 KB 3.9 MB/s eta 0:00:00
Collecting SecretStorage>=3.2
  Downloading SecretStorage-3.3.3-py3-none-any.whl (15 kB)
Collecting jaraco.classes
  Downloading jaraco.classes-3.2.3-py3-none-any.whl (6.0 kB)
Collecting jeepney>=0.4.2
  Downloading jeepney-0.8.0-py3-none-any.whl (48 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.4/48.4 KB 3.5 MB/s eta 0:00:00
Collecting ptyprocess>=0.5
  Downloading ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB)
Collecting idna<4,>=2.5
  Downloading idna-3.4-py3-none-any.whl (61 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 KB 4.1 MB/s eta 0:00:00
Collecting certifi>=2017.4.17
  Downloading certifi-2022.12.7-py3-none-any.whl (155 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 155.3/155.3 KB 9.1 MB/s eta 0:00:00
Collecting charset-normalizer<4,>=2
  Downloading charset_normalizer-3.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (198 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 198.8/198.8 KB 10.1 MB/s eta 0:00:00
Collecting distlib<1,>=0.3.6
  Downloading distlib-0.3.6-py2.py3-none-any.whl (468 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 468.5/468.5 KB 19.4 MB/s eta 0:00:00
Collecting cryptography>=2.0
  Downloading cryptography-39.0.0-cp36-abi3-manylinux_2_28_x86_64.whl (4.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.2/4.2 MB 41.6 MB/s eta 0:00:00
Collecting more-itertools
  Downloading more_itertools-9.0.0-py3-none-any.whl (52 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 52.8/52.8 KB 3.8 MB/s eta 0:00:00
Collecting cffi>=1.12
  Downloading cffi-1.15.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (441 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 441.2/441.2 KB 17.5 MB/s eta 0:00:00
Collecting pycparser
  Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.7/118.7 KB 7.9 MB/s eta 0:00:00
Installing collected packages: webencodings, trove-classifiers, ptyprocess, msgpack, lockfile, distlib, charset-normalizer, zipp, urllib3, tomlkit, tomli, six, shellingham, rapidfuzz, pyrsistent, pycparser, poetry-core, platformdirs, pkginfo, pexpect, packaging, more-itertools, jeepney, idna, filelock, crashtest, certifi, attrs, virtualenv, requests, jsonschema, jaraco.classes, importlib-metadata, html5lib, dulwich, cleo, cffi, requests-toolbelt, cryptography, cachecontrol, SecretStorage, keyring, poetry-plugin-export, poetry
Successfully installed SecretStorage-3.3.3 attrs-22.2.0 cachecontrol-0.12.11 certifi-2022.12.7 cffi-1.15.1 charset-normalizer-3.0.1 cleo-2.0.1 crashtest-0.4.1 cryptography-39.0.0 distlib-0.3.6 dulwich-0.20.50 filelock-3.9.0 html5lib-1.1 idna-3.4 importlib-metadata-4.13.0 jaraco.classes-3.2.3 jeepney-0.8.0 jsonschema-4.17.3 keyring-23.13.1 lockfile-0.12.2 more-itertools-9.0.0 msgpack-1.0.4 packaging-23.0 pexpect-4.8.0 pkginfo-1.9.6 platformdirs-2.6.2 poetry-1.3.2 poetry-core-1.4.0 poetry-plugin-export-1.2.0 ptyprocess-0.7.0 pycparser-2.21 pyrsistent-0.19.3 rapidfuzz-2.13.7 requests-2.28.2 requests-toolbelt-0.10.1 shellingham-1.5.0.post1 six-1.16.0 tomli-2.0.1 tomlkit-0.11.6 trove-classifiers-2023.1.12 urllib3-1.26.14 virtualenv-20.17.1 webencodings-0.5.1 zipp-3.11.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 22.0.4; however, version 22.3.1 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
$ poetry config virtualenvs.create false
$ poetry config repositories.gitlab ****
$ poetry config http-basic.gitlab gitlab-ci-token ${CI_JOB_TOKEN}
Using a plaintext file to store credentials
$ poetry install
Skipping virtualenv creation, as specified in config file.
Installing dependencies from lock file
Package operations: 70 installs, 4 updates, 0 removals
  • Updating charset-normalizer (3.0.1 -> 2.1.1)
  • Installing markupsafe (2.1.1)
  • Installing pytz (2022.7)
  • Installing smmap (5.0.0)
  • Installing alabaster (0.7.12)
  • Installing asttokens (2.2.1)
  • Installing docutils (0.17.1)
  • Installing babel (2.11.0)
  • Installing executing (1.2.0)
  • Installing gitdb (4.0.10)
  • Installing imagesize (1.4.1)
  • Updating importlib-metadata (4.13.0 -> 6.0.0)
  • Installing jinja2 (3.1.2)
  • Installing parso (0.8.3)
  • Installing pure-eval (0.2.2)
  • Installing pygments (2.14.0)
  • Updating requests (2.28.2 -> 2.28.1)
  • Updating setuptools (58.1.0 -> 65.6.3)
  • Installing snowballstemmer (2.2.0)
  • Installing sphinxcontrib-applehelp (1.0.3)
  • Installing sphinxcontrib-devhelp (1.0.2)
  • Installing sphinxcontrib-htmlhelp (2.0.0)
  • Installing sphinxcontrib-jsmath (1.0.1)
  • Installing sphinxcontrib-qthelp (1.0.3)
  CalledProcessError
  Command '['/usr/local/bin/python', '-m', 'pip', 'install', '--disable-pip-version-check', '--isolated', '--no-input', '--prefix', '/usr/local', '--upgrade', '--no-deps', '/root/.cache/pypoetry/artifacts/a9/0e/45/b0a8ce9c0e901c3989302bf1706d7c401eb9cf1ca20e32dd5db7705819/setuptools-65.6.3-py3-none-any.whl']' returned non-zero exit status 1.
  at /usr/local/lib/python3.9/subprocess.py:528 in run
       524│             # We don't call process.wait() as .__exit__ does that for us.
       525│             raise
       526│         retcode = process.poll()
       527│         if check and retcode:
    →  528│             raise CalledProcessError(retcode, process.args,
       529│                                      output=stdout, stderr=stderr)
       530│     return CompletedProcess(process.args, retcode, stdout, stderr)
       531│ 
       532│ 
The following error occurred when trying to handle this error:
  EnvCommandError
  Command ['/usr/local/bin/python', '-m', 'pip', 'install', '--disable-pip-version-check', '--isolated', '--no-input', '--prefix', '/usr/local', '--upgrade', '--no-deps', '/root/.cache/pypoetry/artifacts/a9/0e/45/b0a8ce9c0e901c3989302bf1706d7c401eb9cf1ca20e32dd5db7705819/setuptools-65.6.3-py3-none-any.whl'] errored with the following return code 1, and output: 
  Processing /root/.cache/pypoetry/artifacts/a9/0e/45/b0a8ce9c0e901c3989302bf1706d7c401eb9cf1ca20e32dd5db7705819/setuptools-65.6.3-py3-none-any.whl
  Installing collected packages: setuptools
    Attempting uninstall: setuptools
      Found existing installation: setuptools 58.1.0
      Uninstalling setuptools-58.1.0:
        Successfully uninstalled setuptools-58.1.0
    WARNING: No metadata found in /usr/local/lib/python3.9/site-packages
    Rolling back uninstall of setuptools
    Moving to /usr/local/lib/python3.9/site-packages/_distutils_hack/
     from /usr/local/lib/python3.9/site-packages/~distutils_hack
    Moving to /usr/local/lib/python3.9/site-packages/distutils-precedence.pth
     from /tmp/pip-uninstall-myv7gwjg/distutils-precedence.pth
    Moving to /usr/local/lib/python3.9/site-packages/pkg_resources/
     from /usr/local/lib/python3.9/site-packages/~kg_resources
    Moving to /usr/local/lib/python3.9/site-packages/setuptools-58.1.0.dist-info/
     from /usr/local/lib/python3.9/site-packages/~etuptools-58.1.0.dist-info
    Moving to /usr/local/lib/python3.9/site-packages/setuptools/
     from /usr/local/lib/python3.9/site-packages/~etuptools
  ERROR: Could not install packages due to an OSError: [Errno 17] File exists: '/usr/local/lib/python3.9/site-packages/setuptools/_deprecation_warning.py'
  
  
  at /usr/local/lib/python3.9/site-packages/poetry/utils/env.py:1540 in _run
      1536│                 output = subprocess.check_output(
      1537│                     command, stderr=subprocess.STDOUT, env=env, **kwargs
      1538│                 )
      1539│         except CalledProcessError as e:
    → 1540│             raise EnvCommandError(e, input=input_)
      1541│ 
      1542│         return decode(output)
      1543│ 
      1544│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:
The following error occurred when trying to handle this error:
  PoetryException
  Failed to install /root/.cache/pypoetry/artifacts/a9/0e/45/b0a8ce9c0e901c3989302bf1706d7c401eb9cf1ca20e32dd5db7705819/setuptools-65.6.3-py3-none-any.whl
  at /usr/local/lib/python3.9/site-packages/poetry/utils/pip.py:58 in pip_install
       54│ 
       55│     try:
       56│         return environment.run_pip(*args)
       57│     except EnvCommandError as e:
    →  58│         raise PoetryException(f"Failed to install {path.as_posix()}") from e
       59│ 
  • Installing sphinxcontrib-serializinghtml (1.1.5)
  • Installing text-unidecode (1.3)
  • Installing traitlets (5.8.1)
  • Installing typing-extensions (4.4.0)
  • Installing wcwidth (0.2.5)
Cleaning up project directory and file based variables 00:00
ERROR: Job failed: exit code 1

[dimbleby]

you seem to be installing both poetry and your project system-wide rather than in any virtual environment

• installing your project in the same environment as poetry is definitely a bad idea: if the project requirements overlap poetry's own requirements then installation will cause poetry to change packages while it is using them
• not using a virtual environment is almost definitely a bad idea Document why virtual environments should (nearly) always be used, and system site-packages is considered harmful #6398

[neersighted]

This is indeed Poetry pulling the rug out from under itself as you have asked it to do so; please isolate Poetry from the project being installed so this does not happen.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.