Use of local versions in pep440 ranges (was: Renovate does not update pip dependencies when there is no semver change)

[samgiz]

How are you running Renovate?

Self-hosted Renovate

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

GitHub, Renovate version v9.37.3

Please tell us more about your question or problem

I am trying to update Python packages that follow the pattern <major>.<minor>.<patch>+<timestamp>. pep440 versioning Renovate is using already sorts the versions correctly for this use case, giving preference to versions with a larger timestamp, but there seems to be a bug stopping updates from happening correctly.

Say my requirements.txt looks like this:

some_package==1.0.0+1

and there's a new version 1.0.0+2 available.

What I observe is:

• On this line indep contains currentValue="==1.0.0+1" and currentVersion="1.0.0+1"
• On this line currentVersion gets set to 1.0.0+2 in the first call to getCurrentVersion. There is this check at the start:

  if (allVersions.includes(currentValue)) {
    return currentValue;
  }
  

  Where the intention seems to return the current value if it is a pinned value. The issue is that currentValue is prefixed with ==, whereas the versions in allVersions are not, so this doesn't get entered. Later in the function the getSatisfyingVersion function gets called where the latest version gets selected, because I assume there's no concept of a range for the local part of the pep440 versioning.
  It's also not clear to me why currentVersion needs to be retrieved like this in the first place. Say a version has been revoked and is no longer in the list of versions returned by a datasource, why would we set the current version to the latest one matching instead of using the old version?
• We end up with a situation where currentVersion is the same as the newest version, resulting in no updates. This does not seem correct.

I'm not sure what's the appropriate fix for this. It seems that currentValue received by getCurrentVersion shouldn't have the == prefix but I don't know whether that means passing something else to the function or dropping the prefix somewhere earlier in the execution. Alternatively it would also work if getSatisfyingVersion didn't return 1.0.0+2.

EDIT: here is a small self-contained reproducer: https://github.com/samgiz/renovate-34101-reproducer/

Logs (if relevant)

No response

[github-actions[bot]]

Hi there,

Get your discussion fixed faster by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible.

Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this. Discussions without reproductions are less likely to be converted to Issues.

Please follow these steps:

1. Read our guide on creating a minimal reproduction.
2. Go to our minimal reproduction template repository.
3. Select the Use this template button to create a new repository based on the template.
4. Work on the minimal reproduction in your own repository.
5. Fill out the information in your repository's README.md.
6. Add the link to your reproduction to the first post of your Discussion. If you are not the original author, you can post a new comment with the link.

If you need help with running renovate on your minimal reproduction repository, please refer to our Running Renovate guide.
Good luck,

The Renovate team

[samgiz]

I added a reproducer to my main post.

[rarkins]

Does this require a custom datasource to reproduce?

[samgiz]

It doesn't require a custom datasource to reproduce, it was just the quickest way I could think of getting versions of the required format. I originally discovered the bug with a private registry, which I couldn't include in the reproducer.
If it's not clear, the datasource url in my reproducer can be anything, I discard the fetched data.

[rarkins]

OK. The currentVersion you're seeing definitely seems wrong to me but might need some low level debugging to work out why

[rarkins]

I've added some unit tests to our pep440 matches() function. Two cases I expect to not match do match:

    ✓ matches(1.0, >=1.0.0) === true (9 ms)
    ✓ matches(3.0.0, 3.0.0) === true
    ✓ matches(1.6.2, <2.2.1.0) === true (1 ms)
    ✓ matches(>=3.8, >=3.9) === false
    ✓ matches(1.0.0+1, ==1.0.0+1) === true (1 ms)
    ✕ matches(1.0.0+1, ==1.0.0) === false (3 ms)
    ✕ matches(1.0.0+2, ==1.0.0+1) === false (1 ms)

There seems to be some intentional matching like this in Renovate's pep440 implementation:

https://github.com/renovatebot/pep440/blob/f7f90d7a8c984ff410dd6e8ad86c7d2cd3014241/test/specifier.test.js#L225-L238

      ['2.0', '==2'],
      ['2.0', '==2.0'],
      ['2.0', '==2.0.0'],
      ['2.0+deadbeef', '==2'],
      ['2.0+deadbeef', '==2.0'],
      ['2.0+deadbeef', '==2.0.0'],
      ['2.0+deadbeef', '==2+deadbeef'],
      ['2.0+deadbeef', '==2.0+deadbeef'],
      ['2.0+deadbeef', '==2.0.0+deadbeef'],
      ['2.0+deadbeef.0', '==2.0.0+deadbeef.00'],

Pep440's docs on local identifiers (+) are here: https://peps.python.org/pep-0440/#local-version-identifiers

Local version identifiers MUST comply with the following scheme:

<public version identifier>[+<local version label>]

They consist of a normal public version identifier (as defined in the previous section), along with an arbitrary “local version label”, separated from the public version identifier by a plus. Local version labels have no specific semantics assigned, but some syntactic restrictions are imposed.

Local version identifiers are used to denote fully API (and, if applicable, ABI) compatible patched versions of upstream projects."

And I think that here is the conclusion:

As as I read it, it says that ==1.0.0+1 is invalid to begin with. And that to say that 1.0.0+2 matches ==1.0.0+1 is correct.

[samgiz]

The quote does start with this:

Except where specifically noted below

If you keep reading:

Version matching
A version matching clause includes the version matching operator == and a version identifier.
...
If the specified version identifier is a local version identifier, then the local version labels of candidate versions MUST be considered when matching versions, with the public version identifier being matched as described above, and the local version label being checked for equivalence using a strict string equality comparison.

It says you can't use a local version identifier for == if it ends with .*, but I believe my use case is valid. In fact, I think that's the only situation where a local version identifier can be used (other than with ===, which doesn't have a concept of different version parts).

The sorting of local version identifiers is also well defined and I think Renovate implements that correctly already.

[rarkins]

Good catch. I've created an issue here where I think it needs to be fixed: renovatebot/pep440#764

But please be aware, you'd probably need to fix it yourself if it's important to you.

[samgiz]

I was able to work around the issue with a custom manager and regex versioning for now. I might take a stab at fixing this at a later point.