Error when updating lock file for PDM (pep621) dev dependencies

[aleksul]

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us what version of Renovate you run.

35.102.2

If you're self-hosting Renovate, select which platform you are using.

GitLab self-hosted

If you're self-hosting Renovate, tell us what version of the platform you run.

GitLab 15.10.7

Was this something which used to work for you, and then stopped?

I never saw this working

Wanted end result.

Lock file is updated 😃

As per PDM's documentation, to update a non-default dependency, one has to specify the group of the dependency.
https://pdm.fming.dev/latest/usage/dependency/#update-existing-dependencies

So (in this exact case) instead of running:

pdm update pytest pytest-asyncio ruff

It should be:

pdm update -G tests pytest pytest-asyncio
pdm update -G lint ruff

What you tried so far.

I've disabled lock file updates by setting "updateLockFiles": false.

Relevant debug logs

Logs

File name: pdm.lock

Command failed: pdm update pytest pytest-asyncio ruff
[ProjectError]: pytest does not exist in default dependencies.

pyproject.toml

[project]
dependencies = [
  "fastapi~=0.95.2",
  "uvicorn~=0.22.0"
]

[project.optional-dependencies]
lint = [
  "ruff>=0.0.269",
]
tests = [
  "pytest>=7.3.0",
  "pytest-asyncio>=0.20.3"
]

[rarkins]

@secustor do you need a reproduction?

[github-actions[bot]]

Hi there,

Get your issue fixed faster by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible.

Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on creating a minimal reproduction.

We may close the issue if you, or someone else, haven't created a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

[aleksul]

While trying to create a minimal reproduction on GitHub, I have encountered another error:

aleksul/renovate-reproduction-dev-deps-pdm#5

Command failed: docker run --rm --name=renovate_sidecar --label=renovate_child -v "/mnt/renovate/gh/aleksul/renovate-reproduction-pdm-dev-deps":"/mnt/renovate/gh/aleksul/renovate-reproduction-pdm-dev-deps" -v "/tmp/renovate-cache":"/tmp/renovate-cache" -v "/tmp/containerbase":"/tmp/containerbase" -e BUILDPACK_CACHE_DIR -e CONTAINERBASE_CACHE_DIR -w "/mnt/renovate/gh/aleksul/renovate-reproduction-pdm-dev-deps" ghcr.io/containerbase/sidecar bash -l -c "install-tool pdm 2.6.1 && pdm update ruff pytest pytest-asyncio"

No python defined - aborting

[aleksul]

Not sure what are the next step here tho...

[rarkins]

That part should be a simple fix. The other managers already install python first. Only question is whether we should detect which version of python from the pdm lock file?

[aleksul]

detect which version of python from the pdm lock file?

*probably you meant from pyproject.toml

I think that would be preferable, as Python packages can have different version constraints for different Python versions. Also, it should be overridable with a constraint: https://docs.renovatebot.com/modules/manager/pip-compile/#configuration-of-python-version

[rarkins]

No, I meant lock file, but either is OK. For example with poetry it's lock file:

renovate/lib/modules/manager/poetry/artifacts.ts

Lines 23 to 35 in a9179e0

	export function getPythonConstraint(
	existingLockFileContent: string
	): string | undefined | null {
	try {
	const data = parse(existingLockFileContent) as PoetryLock;
	if (is.string(data?.metadata?.['python-versions'])) {
	return data?.metadata?.['python-versions'];
	}
	} catch (err) {
	// Do nothing
	}
	return undefined;
	}

[rarkins]

Fix for python installation: #22441

[aleksul]

I have re-runned the reproduction example on the 35.102.4 - the "no python error" is gone now, thank you for the fast and awesome fix!

I have also managed to reproduce the initial "does not exist in default dependencies" error, please see aleksul/renovate-reproduction-dev-deps-pdm repo and PR #3 as one of the examples.

[aleksul]

Nope, that's a separate issue about subdirectories, quoting a comment from below ⬇️:

Just to add to this issue, I've reproduced an additional issue...

The one which is the topic starter (for which I'm attaching the reproduction example), about dependency groups, isn't fixed yet.

[rarkins]

It would be best to create a separate discussion, and just link to it from here instead of embedding here (so that such confusion is less likely to happen in future)

[mkniewallner]

It would be best to create a separate discussion, and just link to it from here instead of embedding here (so that such confusion is less likely to happen in future)

You're right, I definitely should have created a separate discussion, sorry for that.

[secustor]

#22489

[mkniewallner]

Just to add to this issue, I've reproduced an additional issue over https://github.com/mkniewallner/renovate-pdm-groups-issue, where the project contains multiple pyproject.toml.

mkniewallner/renovate-pdm-groups-issue#2 fails to update a dependency that is located in sub_directory/pyproject.toml, as it runs the command in the root directory instead of sub_directory:

Command failed: docker run --rm --name=renovate_sidecar --label=renovate_child -v "/mnt/renovate/gh/mkniewallner/renovate-pdm-groups-issue":"/mnt/renovate/gh/mkniewallner/renovate-pdm-groups-issue" -v "/tmp/renovate-cache":"/tmp/renovate-cache" -v "/tmp/containerbase":"/tmp/containerbase" -e BUILDPACK_CACHE_DIR -e CONTAINERBASE_CACHE_DIR -w "/mnt/renovate/gh/mkniewallner/renovate-pdm-groups-issue" ghcr.io/containerbase/sidecar bash -l -c "install-tool python 3.11.3 && install-tool pdm 2.6.1 && pdm update httpx"
[ProjectError]: httpx does not exist in default dependencies.

The real project that the reproduction is based on used to rely on Poetry, where dependencies correctly got updated, so I believe that this issue is specific to PDM.

PDM provides a way to specify the path to a project through --project option, so a solution could be to run:

pdm update --project sub_directory httpx

[viceice]

i think we forget to pass the package dir to the exec command, like we do in other managers, that will cause the exec to be set the cwd to the package directory

[secustor]

K, then it is missing for other managers like e.g. helm too.

I'm preparing a PR to fix this and the constraint extraction.

[viceice]

it seems not all tools need this 🤷‍♂️

[secustor]

#22450