lockfile maintenance for go.mod files

[v0lkc]

Would be very nice if renovate could regularly

1. update transitive go dependencies
2. run go mod tidy to keep the sum files clean.

Did you already have any implementation ideas?

Please correct me if I'm wrong. The idea behind the lockfileMaintenance is to update transitive dependencies only and to keep direct deps untouched right? If this is the case, then it should be pretty straightforward to implement.

If we pass the list of direct deps to go get like below, then it will keep them untouched and only update indirect dependencies.

go get -u -d <direct-deps>

sample

Given we have the following module

module xyz

require (
	github.com/PaesslerAG/gval v1.0.1 // indirect
	github.com/PaesslerAG/jsonpath v0.1.1
}

then go get -u -d github.com/PaesslerAG/jsonpath@v0.1.1 would result in

module xyz

require (
	github.com/PaesslerAG/gval v1.1.0 // indirect
	github.com/PaesslerAG/jsonpath v0.1.1
}

How to implement?

1. collect all direct deps of a mod file
2. pass them at once to go get like go get -u -d <dep1>@<dep1_version> <dep2>@<dep2_version> ...
3. run go mod tidy if activated

[viceice]

Looks like you referring to https://docs.renovatebot.com/configuration-options/#lockfilemaintenance

Which is currently not supported for gomod

[HonkingGoose]

@rarkins and @viceice Is this feature something you want to have in Renovate bot? I'll keep this at priority-5-triage for now. 😄

[rarkins]

In other package managers we delete the lock file and then run an install command to repopulate it from scratch.

Go instead uses a checksum file but importantly its MVS algorithm should mean you don't get a change to transitive dependencies just because newer versions are released.

[eriksw]

@rarkins What should we do if we want Renovate to bump indirect dependencies?

Scenario:

• The package Renovate is maintaining is a tool meant to be used by end users.
• We depend on A.
• A is basically "done" and never gets updated.
• A depends on B.
• B is updated frequently and has a security vulnerability.

Right now, renovate will never open a PR to get B up to date, so we keep building in the insecure version of B.

[rarkins]

This is not possible due to Go's Minimum Version Selection concept. Transitive dependencies can't be "bumped" until at least one package depending on it increases its minimum version. In your scenario A must update its go.mod to bump minimum version of B

[eriksw]

This is not possible due to Go's Minimum Version Selection concept. Transitive dependencies can't be "bumped" until at least one package depending on it increases its minimum version. In your scenario A must update its go.mod to bump minimum version of B

That is not correct.

You can bump the version of a transitive dependency the same as if it was a direct dependency.

[rarkins]

Please provide more details and a reproduction repository demonstrating it so we can take a look

[rarkins]

This is 4y old but implies that there's a way to manually bump transitive dependencies using go: golang/go#24500 (comment)

PRs would be welcome if someone can achieve this in Renovate

[github-actions]

Hi there,

Help us by making a minimal reproduction repository.

Before we can start work on your issue we first need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on creating a minimal reproduction to understand what is needed.

We may close the issue if you (or someone else) have not provided a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

[rarkins]

Needs a reproduction repo (with out of date transitive deps) plus confirmation of exactly which commands to run

[eriksw]

You bump it exactly the same way you bump a direct dependency.

[eriksw]

Simple repro: https://github.com/eriksw/renovate-transitive-bump/commits/main

[eriksw]

With main at eriksw/renovate-transitive-bump@4608150 the onboarding message says:

What to Expect
It looks like your repository dependencies are already up-to-date and no Pull Requests will be necessary right away.

This is incorrect.

If you run go get -u you get a TON of updates:

➜  renovate-transitive-bump git:(main) go get -u
go: downloading github.com/aws/aws-sdk-go v1.42.44
go: downloading github.com/aws/aws-sdk-go-v2/config v1.13.1
go: downloading github.com/aws/aws-sdk-go-v2 v1.13.0
go: downloading github.com/aws/aws-sdk-go-v2/internal/ini v1.3.5
go: downloading github.com/aws/smithy-go v1.10.0
go: downloading github.com/aws/aws-sdk-go-v2/credentials v1.8.0
go: downloading github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.10.0
go: downloading github.com/aws/aws-sdk-go-v2/service/sso v1.9.0
go: downloading github.com/aws/aws-sdk-go-v2/service/sts v1.14.0
go: downloading github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.7.0
go: downloading github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.4
go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.2.0
go get: upgraded github.com/aws/aws-sdk-go v1.40.34 => v1.42.44
go get: upgraded github.com/aws/aws-sdk-go-v2 v1.9.0 => v1.13.0
go get: upgraded github.com/aws/aws-sdk-go-v2/config v1.7.0 => v1.13.1
go get: upgraded github.com/aws/aws-sdk-go-v2/credentials v1.4.0 => v1.8.0
go get: upgraded github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.5.0 => v1.10.0
go get: added github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.4
go get: added github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.2.0
go get: upgraded github.com/aws/aws-sdk-go-v2/internal/ini v1.2.2 => v1.3.5
go get: upgraded github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.3.0 => v1.7.0
go get: upgraded github.com/aws/aws-sdk-go-v2/service/sso v1.4.0 => v1.9.0
go get: upgraded github.com/aws/aws-sdk-go-v2/service/sts v1.7.0 => v1.14.0
go get: upgraded github.com/aws/smithy-go v1.8.0 => v1.10.0
go get: upgraded github.com/googleapis/gax-go/v2 v2.1.0 => v2.1.1
go get: upgraded golang.org/x/net v0.0.0-20210825183410-e898025ed96a => v0.0.0-20220127200216-cd36cc0744dd
go get: upgraded google.golang.org/api v0.56.0 => v0.66.0
go get: upgraded google.golang.org/genproto v0.0.0-20210831024726-fe130286e0e2 => v0.0.0-20220201184016-50beb8ab5c44
go get: upgraded google.golang.org/grpc v1.40.0 => v1.44.0

[rarkins]

What to Expect
It looks like your repository dependencies are already up-to-date and no Pull Requests will be necessary right away.

This is incorrect.

Not really. This statement refers to direct dependencies, which evidently are up to date. It's really just meant to say "you have no PRs", and is particularly relevant to this discussion. If/when lock file maintenance is supported by Renovate then it would list single "lock file maintenance" PR in that list instead of saying it's empty.

No Renovate manager today raises individual PRs per transitive dependency, and we don't plan to add that because the volume of PRs would be unsustainable in most cases.

What this feature (#9578) would do is add Go Modules to this capability: https://docs.renovatebot.com/configuration-options/#lockfilemaintenance

[rarkins]

Reproduction forked to https://github.com/renovate-reproductions/9578

[rarkins]

One important point: does go get -u also update direct dependencies? In this case it's not really "lock file maintenancer" (which is meant to refresh transitive dependencies only) but instead "update absolutely everything - both direct and indirect".

[eriksw]

No Renovate manager today raises individual PRs per transitive dependency, and we don't plan to add that because the volume of PRs would be unsustainable in most cases.

That's an unfortunate position because in the scenario in #9578 (comment) that is exactly what is needed.

One important point: does go get -u also update direct dependencies? In this case it's not really "lock file maintenancer" (which is meant to refresh transitive dependencies only) but instead "update absolutely everything - both direct and indirect".

There isn't a "lock file maintenancer" command because go does not have a lock file, at least, not in the sense of package managers that use range constraints instead of MVS, owing to there being no way to express a maximum version or an exact version.

From the perspective of go get -u the difference between a direct and an indirect dependency is ~nothing other than whether it'll get labeled with a // indirect comment when go.mod is re-written.

It's worth being aware that the // indirect is just a comment—it's just a hint for humans that the package isn't directly imported by any .go file in the current module. You cannot trust it because it's not funcational—you could label every dependency in go.mod with it, just as you could omit it on things that are not directly imported, and things would work just fine in both situations. (Note: You can trust it after running go mod tidy.)

[rarkins]

No Renovate manager today raises individual PRs per transitive dependency, and we don't plan to add that because the volume of PRs would be unsustainable in most cases.

That's an unfortunate position because in the scenario in #9578 (comment) that is exactly what is needed.

But in that case you have misunderstood and hijacked this feature request for a different requirement. This feature request is for "lock file maintenance", which is a Renovate term with a specific meaning.

There isn't a "lock file maintenancer" command because go does not have a lock file, at least, not in the sense of package managers that use range constraints instead of MVS, owing to there being no way to express a maximum version or an exact version.

As I said above, we invented the term lock file maintenance and so it can mean whatever we want it to. I understand Go doesn't have a lock file per se, but go.mod/go.sum work together to achieve effectively the same thing (deterministic installs). Lock file maintenance is intended to mean "keep all direct dependency constraints unchanged, while updating the resolved versions to their highest compatible versions". In other managers this normally means keeping the "package file" (e.g. package.json) unchanged, but that's not feasible with go.mod.

From the perspective of go get -u the difference between a direct and an indirect dependency is ~nothing other than whether it'll get labeled with a // indirect comment when go.mod is re-written.

Yes, and that lack of difference is precisely the problem.

For a user, there is a significance difference between a direct and indirect dependency, regardless of which language it is. In general, they don't want to care about their indirect dependencies if they can avoid it.

It's worth being aware that the // indirect is just a comment—it's just a hint for humans that the package isn't directly imported by any .go file in the current module. You cannot trust it because it's not funcational—you could label every dependency in go.mod with it, just as you could omit it on things that are not directly imported, and things would work just fine in both situations. (Note: You can trust it after running go mod tidy.)

Same comment as above. Developers care about direct dependencies, and ideally indirect dependencies would "just work".

Renovate already creates plenty of "noise" by updating direct dependencies. We don't want to overwhelm people by making them think about every single indirect dependency. Russ's comment in the linked discussion applies equally well here:

No one has asked for v1.2.4. It may be a terrible release that's going to be yanked in an hour. Until someone says explicitly "I want to do an update operation", with the implied "I'm ready for things to break and to deal with that", I don't believe the build system has any business pulling in brand new releases.

I think the same thing applies for Renovate. If some deeply nested transitive dependency has a new version, why should you care, other than the edge case that it's fixing a vulnerability.

In summary, your desire to "surgically" update a specific transitive dependency only in the case of vulnerabilities is reasonable, but it's not "lock file maintenance". Please create a separate feature request so we can narrow those requirements there, ideally with a vulnerable reproduction.

[rarkins]

Regarding the lock file maintenance feature for gomod (i.e. this original issue), I think it can be implemented using go get -u but we will need to document the caveat that it will update all dependencies to the latest, not just transitive or locked.