Poetry dependencies can fail to build in niche case

[TheKevJames]

Summary

Poetry dependencies which include distutils-based sub-dependencies which in turn have no dependencies of their own can cause Artifact Update Problems for the parent projects. This is due to the PIP_USER flag, which is erroneously set when running poetry update in some of the renovate base images. Though debugging was carried out via self-hosted Renovate, this should affect both hosted and self-hosted and in any case where the renovate app runs in a renovate docker image, in addition to cases where users provide their own images but have PIP_USER set for other reasons.

There are several possible solutions which might solve this, but as they could have far-reaching impacts I would defer to @rarkins at the team to point me in the right direction. Please find the potential solutions at the bottom of this report, as they really don't make much sense without the full context.

Full Explanation

We've been getting a bunch of "⚠️ Artifact update problem" in most of our Renovate bot runs. Re-running with DEBUG logs shows a whole bunch of errors along the lines of (note: some irrelevant details relating to our repo have been replaces with fake env vars, otherwise error message unchanged):

DEBUG logs

DEBUG: Contents updated (repository=$USER/$REPO, packageFile=$PATH/pyproject.toml, branch=renovate/h5py-3.x)
       "depName": "h5py"
DEBUG: poetry.updateArtifacts($PATH/pyproject.toml) (repository=$USER/$REPO, branch=renovate/h5py-3.x)
DEBUG: Updating $PATH/poetry.lock (repository=$USER/$REPO, branch=renovate/h5py-3.x)
DEBUG: Using python constraint from config (repository=$USER/$REPO, branch=renovate/h5py-3.x)                                                                                                                                                                                DEBUG: Executing command (repository=$USER/$REPO, branch=renovate/h5py-3.x)
       "command": "poetry update --lock --no-interaction h5py"
DEBUG: Failed to update $PATH/poetry.lock file (repository=$USER/$REPO, branch=renovate/h5py-3.x)
       "err": {                                                                                                                                                                                                                                                                         "killed": false,
         "code": 1,                                                                                                                                                                                                                                                                     "signal": null,
         "cmd": "poetry update --lock --no-interaction h5py",                                                                                                                                                                                                                           "stdout": "Creating virtualenv $PKG-okBmXY7s-py3.9 in /root/.cache/pypoetry/virtualenvs\nUpdating dependencies\nResolving dependencies...\n\n  PackageInfoError\n\n  Unable to determine package info for path: /tmp/tmp6u_288mb/distance\n  \n  Fa
llback egg_info generation failed.\n  \n  Command ['/tmp/tmplfjkgfye/.venv/bin/python', 'setup.py', 'egg_info'] errored with the following return code 1, and output: \n  notice: no C support available\n  Warning: 'classifiers' should be a list, got type 'tuple'\n  usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]\n     or: setup.py --help [cmd1 cmd2 ...]\n     or: setup.py --help-commands\n     or: setup.py cmd --help\n  \n  error: invalid command 'egg_info'\n\n  at /usr/local/poetry/lib/poetry/inspection/info.py:502
 in _pep517_metadata\n      498│                 try:\n      499│                     venv.run(\"python\", \"setup.py\", \"egg_info\")\n      500│                     return cls.from_metadata(path)\n      501│                 except EnvCommandError as fbe:\n    → 502│
                  raise PackageInfoError(\n      503│                         path, \"Fallback egg_info generation failed.\", fbe\n      504│                     )\n      505│                 finally:\n      506│                     os.chdir(cwd.as_posix())\n",
         "stderr": "",
         "message": "Command failed: poetry update --lock --no-interaction h5py\n",                                                                                                                                                                                                     "stack": "Error: Command failed: poetry update --lock --no-interaction h5py\n\n    at ChildProcess.exithandler (child_process.js:308:12)\n    at ChildProcess.emit (events.js:315:20)\n    at ChildProcess.EventEmitter.emit (domain.js:467:12)\n    at maybeClose (in
ternal/child_process.js:1048:16)\n    at Process.ChildProcess._handle.onexit (internal/child_process.js:288:5)"
       }

Note as well that h5py is just one of many examples, but all follow the same message as above.

This can be replicated by using the renovate/python:latest image (digest: renovate/python@sha256:f4841e0bc49c2aa6e2ad900456f99fceebc1a36afb5c0f94c7bf57f3b50f9c26) as follows:

Docker debugging commands and error log

$ docker run --rm -it -v/path/to/repo:/src renovate/python:latest bash
$ pip install poetry  # as per https://github.com/renovatebot/renovate/blob/9f9a6dc3922c9cf6461a329f9c72fab96fb63595/lib/manager/poetry/artifacts.ts#L124
$ cd /src
$ poetry update --lock --no-interaction h5py
Creating virtualenv $PKG-JLNeu2Pd-py3.9 in /home/ubuntu/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies... (14.0s)

  PackageInfoError

  Unable to determine package info for path: /tmp/tmp5oh9kkpl/distance

  Fallback egg_info generation failed.

  Command ['/tmp/tmp_7n_o4z_/.venv/bin/python', 'setup.py', 'egg_info'] errored with the following return code 1, and output:
  notice: no C support available
  Warning: 'classifiers' should be a list, got type 'tuple'
  usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: setup.py --help [cmd1 cmd2 ...]
     or: setup.py --help-commands
     or: setup.py cmd --help

  error: invalid command 'egg_info'

  at ~/.local/lib/python3.9/site-packages/poetry/inspection/info.py:502 in _pep517_metadata
      498│                 try:
      499│                     venv.run("python", "setup.py", "egg_info")
      500│                     return cls.from_metadata(path)
      501│                 except EnvCommandError as fbe:
    → 502│                     raise PackageInfoError(
      503│                         path, "Fallback egg_info generation failed.", fbe
      504│                     )
      505│                 finally:
      506│                     os.chdir(cwd.as_posix())

(Note that I also ran poetry config http-basic.fury $GEMFURY_TOKEN NOPASS as this package includes dependencies on a private PyPI server. This does not seem to be relevant to the bug report as we've seen the same issue on repos without private dependencies).

(At this point I looked into version issues, as there are several GH Issues and SO posts showing that the above error may have to do with out of date setuptools versions, etc, but all turned out to be red herrings. It may be worth noting, though, that the renovate/python image uses setuptools version 49.2.1 despite 53.0.0 being latest; it may be worth upgrading!)

Please note from the above output that the specific package which is failing is distance -- though it is not the only package to throw here, it did turn out that all instances of our projects failing eventually included either distance or another package like it in their dependency tree.

The relevant points of note for distance (and distance-like packages causing the same error) is that distance users distutils rather than setuptools (it's quite old) and has no dependencies. This, in turn, leads Poetry to execute this code: it needs to fall back to actually installing the package to verify there are no dependencies.

The interesting point here -- running the poetry update command locally (eg. without renovate) works just fine; the problem isn't that Poetry needs to fall back, but rather that it falls back incorrectly within Renovate. Debug logging through Poetry eventually leads to the point of divergence where Renovate is breaking: this method is the source of failure, as it errors when run via Renovate but not locally. Printing that stack trace gives us the crux of the issue:

Command ['/tmp/tmpz29r_m52/.venv/bin/python', '-m', 'pip', 'install', '--disable-pip-version-check', '--ignore-installed', 'pep517===0.8.2', 'toml==0.10.1'] errored with the following return code 1, and output:
ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.

But... why are we trying to do a --user install within a virtualenv anyway? In my understanding, the entire point of using virtualenvs is inorder to avoid needing to write to global-vs-user site packages, so this flag is pretty much nonsensical in our case.

Turns out the reason pip is attempting a user install here is because we have PIP_USER=true set in the environment. It looks like the reason for that is the renovate-buildpack config for install-tool python which sets this at build time.

This makes sense for the use-case where we expect to be installing packages via pip directly in the image, but will break use-cases like this where we are installing within a virtual environment.

Possible Fixes

I haven't done a full audit of everything renovate does within python images, so at this point I'll defer to @rarkins and the team for thoughts on how best to solve this. I think the following options would be potentially reasonable:

• avoid setting PIP_USER in the build pack, set it only for the subset of commands we know should be user installs -- if most of our usecases are within virtual environments, this might make the most sense! Note that it seems you've already run into cases which require you to unset this.

• unset PIP_USER within the renovate/python:latest docker image, after the install-tool commands have completed -- if renovate itself runs commands within venvs, but the install-tool commands are all global, this might make sense.

• update only the manager/poetry/artifacts.ts commands to include PIP_USER=false as an override in the env var options here (eg. by adding env: {"PIP_USER": "false"} to that map). This would solve my specific use-case without impacting other renovate features, but may not solve all instances of this issue if venvs are used anywhere else by renovate.

  • Note that this option would also solve the case where users' are running these commands outside of docker (eg. when this check returns false) but their environment also has PIP_USER=true set, for whatever reason. It might be worth having PIP_USER=false be a default somewhere in this util/exec level.

[viceice]

Without PIP_USER=true all normal pip install commands fail as pip tries to install them to s non writable global directory.

Renovate doesn't use venv itself, it even doesn't need python.

As renovate needs to install pip packages for some artifacts on the fly we need the user install.

So the solution for this is to set PIP_USER=false only for the poetry call.

[viceice]

Another solution is to add a install-pip helper, which install pip packages with PIP_USER=true when we aren't root.

@rarkins What do you think?

[viceice]

Ha, we already have install-pip tool 🙃

[viceice]

Looks like newer pip versions are falling back to user install if global isn't writable

pip install poetry
Defaulting to user installation because normal site-packages is not writeable

[viceice]

@TheKevJames It will need some time to let the changes from buildpack bubble through, so please retest in afternoon.

[TheKevJames]

Verified this solves the issue! Thanks for the fast fix @viceice !