Renovate raising PRs with version conflicts

[lululaplap]

I am trying to configure renovate for our python packages which use setup-cfg. We use a nexus repository to host our internal packages. This serves as a proxy for pypi as well, so we only need one index url. However, when running renovate it updates the version without checking if the new version conflicts with other dependencies. For example, we have pinned pytorch to an earlier version for now as we need to carefully manage to update it. However, renovate raises a PR to update torchvision to the latest version. torchvision depends on pytorch, so the latest version conflicts with the pytorch version we have pinned. Is this expected behavior or is there something wrong with my setup? Cheers

module.exports = {
    "platform": "bitbucket",
    "username": process.env.BITBUCKET_USERNAME,
    "password": process.env.BITBUCKET_PASSWORD,
    "repositories": [],
    "python": {
        "registryUrls": [process.env.PIP_EXTRA_INDEX_URL],
        "enabled": true
    },

    "allowedPostUpgradeCommands": ["bumpver update --patch --no-commit"],
    "postUpgradeTasks":{
      "commands": ["bumpver update --patch --no-commit"],
      "executionMode": "branch",
      "fileFilters": ["**/**"]
      },
    "includeForks":true,
    "recreateClosed": true,
    "groupName": "all dependencies",
    "separateMajorMinor": true,
    "azure-pipelines": {
      "enabled": false
    },
    "packageRules": [
    {
      "matchPackagePatterns": ["^torch$"],
      "groupName": "pytorch",
      "allowedVersions": "1.7.0"

    },
    {
      "matchPackagePatterns": ["^python$"],
      "groupName": "python",
      "allowedVersions": "3.8"
    },
    {
        "matchUpdateTypes": ["minor", "patch", "pin", "digest"],
        "automerge": false
    },],
    "onboardingConfig": { "extends": ["config:base"]},
    "commitMessagePrefix": "PKG:",
    "logLevel": 'debug',
    "baseBranches": ["master"],
    "prConcurrentLimit": 1,

}

[rarkins]

It's expected behavior that Renovate can raise PRs which result in conflicting dependencies. Your choices are:

• Manually fix the problem in the PR before merging, or
• Add a grouping rule permanently to your repo(s) so that related dependencies are upgraded together in future, or
• Close the PR to ignore it and move on

[rarkins]

No, this is a problem which all package managers share

[ndevenish]

Thanks, that's disappointing to hear. Do you know if there is a bug tracking this (IMO rather major) issue?

[rarkins]

It's not a bug, especially in Renovate, and instead it's a natural consequence of updating dependencies. Sometimes there will be incompatible dependency combinations which need you to either resolve manually or decline the update hoping that later they will be resolved. If you like the original poster have intentionally pinned dependencies/stopped updating them then you will increase the chance of getting such problems.

Hopefully the fact that you can't find anyone else sharing your "major" concerns after 6 years and thousands of issues and discussions leads you to reconsider if you are overestimating its impact.

If you have any specific suggestions about specific package managers and handling of updates then we can always consider them. For example in Bundler we try to catch "we could not update X because of Y" errors and then try to update both X and Y and keep doing that recursively.

[ndevenish]

If renovate makes a PR updating a line in requirements.txt, and after that PR pip install -r requirements.txt fails, that's a bug in the tool that is supposed to help you manage updating package dependencies. pip install --dry-run -r requirements.txt would tell your bot that this isn't going to work. or pip check. I asked about poetry because it's plausible that pip requirements would have this issue, given problems that pip had in the past about building broken/incorrect dependency environments (which it seems to be much better for now).

I took "this is a problem which all package managers share" as just a miscommunication, but you actually believe this is a property of all package managers and not your implementation of updating them? Lots of package managers offer updating that isn't literally "look at the latest release on pypi and replace the version with that" - which is the description of the algorithm renovate uses. If I need to manually inspect every other dependency every time your tool makes an update PR, then your tool isn't reducing work. If I need to close the PR manually, then remember that the PR needs to be reopened next time the other dependency is updated, what mental effort is it saving me.

As for nobody else finding a problem (except, this literal discussion) - I suggest that people see your aggressive rejection of concerns and decide not to bother. Dependabot explicitly says that it won't surface a PR if the update would cause a dependency conflict.

If you have any specific suggestions about specific package managers and handling of updates then we can always consider them.

My specific suggestion: When updating a python requirements.txt, pip install --dry-run -r <file_being_updated>. Then don't surface a PR if this fails. Most other package mangers probably have some equivalent way of validating the dependency web.

[ndevenish]

Here's another example of a package manager explicitly refusing to let you update one dependency that would cause a conflict:

#/usr/bin/env bash
set -x
poetry init -n
poetry add 'sphinx<7' 'sphinx-rtd-theme=1.2.0'
poetry add sphinx@latest

In fact, you can even poetry add --dry-run sphinx@<specific_version_number> and poetry will tell you if this is valid without updating anything.