Cannot re-lock Pipfile.lock when mysql_config is required

[wastrachan]

How are you running Renovate?

• WhiteSource Renovate hosted app on github.com
• Self hosted

If using the hosted app, please skip to the next section.
Otherwise, if self-hosted, please complete the following:

Please select which platform you are using:

• Azure DevOps (dev.azure.com)
• Azure DevOps Server
• Bitbucket Cloud (bitbucket.org)
• Bitbucket Server
• Gitea
• github.com
• GitHub Enterprise Server
• gitlab.com
• GitLab self-hosted

Renovate version: 25.56.0

Describe the bug

Renovate fails re-locking Pipfile.lock after performing a dependency update in the Pipfile.

This Pipfile contains mysqlclient which requires MySQL development headers (mysql_config or mariadb_config) to build.

Presumably these headers (mysql_config, mariadb_config) are missing from the docker.io/renovate/python Docker image. This could be overcome by making the necessary headers available in the base image or by providing a mechanism to a) override the base image or b) install additional dependencies prior to the re-locking task

Relevant debug logs

ERROR:pip.subprocessor:Command errored out with exit status 1:
 command: /home/ubuntu/.local/share/virtualenvs/4Nce-5ne/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-resolver-rhjm5afv/mysqlclient/setup.py'"'"'; __file__='"'"'/tmp/pip-resolver-rhjm5afv/mysqlclient/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-resolver-rhjm5afv/mysqlclient/pip-egg-info
     cwd: /tmp/pip-resolver-rhjm5afv/mysqlclient/
Complete output (15 lines):
mysql_config --version
/bin/sh: 1: mysql_config: not found
mariadb_config --version
/bin/sh: 1: mariadb_config: not found
mysql_config --libs
/bin/sh: 1: mysql_config: not found
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-resolver-rhjm5afv/mysqlclient/setup.py", line 15, in <module>
    metadata, options = get_config()
  File "/tmp/pip-resolver-rhjm5afv/mysqlclient/setup_posix.py", line 70, in get_config
    libs = mysql_config("libs")
  File "/tmp/pip-resolver-rhjm5afv/mysqlclient/setup_posix.py", line 31, in mysql_config
    raise OSError("{} not found".format(_mysql_config_path))
OSError: mysql_config not found

Have you created a minimal reproduction repository?

Please read the minimal reproductions documentation to learn how to make a good minimal reproduction repository.

• I have provided a minimal reproduction repository
• I don't have time for that, but it happens in a public repository I have linked to
• I don't have time for that, and cannot share my private repository
• The nature of this bug means it's impossible to reproduce publicly

Additional context

Reproduction: https://github.com/wastrachan/renovate-reproduction
Reproduction PR: https://github.com/wastrachan/renovate-reproduction/pull/2

[rarkins]

I forget where it was discussed, but I think previously someone proved that the official python Docker images include some database drivers like this. If so then we should too, but this is not the right repo for it

[wastrachan]

I have not found any evidence that the MySQL headers are included in any official python image. The official images include libsqlite3-dev, but database support stops there. I can't make the argument that they should be included in renovate's base image if the goal is parity with the official python image, but I can't imagine that my use-case is uncommon.

I do see the referenced issue (containerbase/base#16): it seems that this would probably solve my problem (and the problem of any missing headers for package builds, more broadly).

I have no illusion that this feature will be added soon-- are there any options I'm not considering? I can't see any documentation on using a custom docker image (and I'd guess this isn't supported with the hosted runner, anyways).

If there is nothing else I can do- and we can conclude that this isn't an issue or feature request that can be filled, I suppose this can be closed.

[wastrachan]

I'm going to slightly correct my previous statement. The mysql development headers aren't available in the slim python images... but they are available in the standard official python images.

buildpack-deps includes the mysql headers: https://github.com/docker-library/buildpack-deps/blob/master/debian/buster/Dockerfile#L56

python then is based on this buildpack-deps: https://github.com/docker-library/python/blob/fe2130938363f307081496dc5930c29c3ef9ddba/Dockerfile-debian.template#L1

[viceice]

I'm pretty sure installing default-libmysqlclient-dev isn't enough. You will also need build-essential to have a compiler?

@wastrachan Can you please elaborate which packages are required for ubuntu focal to successfully run pip lock?

[viceice]

Duplicate of containerbase/base#98