Use `go list -m -versions` to determine available versions of a go module #4434

mctofu · 2021-11-19T18:28:30Z

This removes usage of the native helper to resolve available go module versions. Instead we'll use the existing go tooling which has added support for this. This has a few benefits:

No longer need to maintain https://github.com/dependabot/gomodules-extracted or make calls to internal go libs
No longer need to manually handle exclude directives
Retract directives are now supported so Dependabot won't propose updates to retracted versions

We previously attempted this change in #3630 but the approach was abandoned after we found the change was negatively impacting performance. It was further noted that using go list -m -versions was causing the go tooling to download all of the modules referenced in go.mod and this accounted for the bulk of the performance hit. This appears to only happen when GOPRIVATE=* is set which we currently need to do to support private modules.

As a workaround I found that providing a go.mod with no requirements was a way to sidestep the performance issue. We still end up getting the version information we are looking for and no extra modules are downloaded! The performance is close to the native version as well.

Example output / performance

Using native helper

[dependabot-core-dev] /tmp/go $ rm -rf /home/dependabot/go/pkg/mod/cache
[dependabot-core-dev] /tmp/go $ cat go.mod
module foo

require github.com/dependabot-fixtures/go-modules-lib v1.0.0

go 1.15
[dependabot-core-dev] /tmp/go $ time echo '{"function":"getVersions","args":{"dependency":{"name":"github.com/dependabot-fixtures/go-modules-lib","version":"v1.0.0"}}}'  | GOPRIVATE='*' /opt/go_modules/bin/helper
{"result":["v1.0.0","v1.0.1","v1.0.5","v1.0.6","v1.1.0","v1.2.0-pre1","v1.2.0-pre2"]}
real    0m1.017s
user    0m0.105s
sys     0m0.087s
[dependabot-core-dev] /tmp/go $ du -sk /home/dependabot/go/pkg/mod/cache
176     /home/dependabot/go/pkg/mod/cache

Using `go list -m -versions` with project `go.mod`

[dependabot-core-dev] /tmp/go $ rm -rf /home/dependabot/go/pkg/mod/cache
[dependabot-core-dev] /tmp/go $ cat go.mod
module foo

require github.com/dependabot-fixtures/go-modules-lib v1.0.0

go 1.15
[dependabot-core-dev] /tmp/go $ time GOPRIVATE='*' go list -m -versions -json github.com/dependabot-fixtures/go-modules-lib
{
        "Path": "github.com/dependabot-fixtures/go-modules-lib",
        "Version": "v1.0.0",
        "Versions": [
                "v1.0.0",
                "v1.0.1",
                "v1.0.5",
                "v1.0.6",
                "v1.1.0",
                "v1.2.0-pre1",
                "v1.2.0-pre2"
        ],
        "Time": "2018-10-18T21:48:48Z",
        "GoMod": "/home/dependabot/go/pkg/mod/cache/download/github.com/dependabot-fixtures/go-modules-lib/@v/v1.0.0.mod"
}

real    0m8.828s
user    0m1.882s
sys     0m1.465s
[dependabot-core-dev] /tmp/go $ du -sk /home/dependabot/go/pkg/mod/cache
24844   /home/dependabot/go/pkg/mod/cache

Using `go list -m -versions` with stub `go.mod` (this PR)

[dependabot-core-dev] /tmp/go $ rm -rf /home/dependabot/go/pkg/mod/cache
[dependabot-core-dev] /tmp/go $ cat go.mod
module foo
[dependabot-core-dev] /tmp/go $ time GOPRIVATE='*' go list -m -versions -json github.com/dependabot-fixtures/go-modules-lib
{
        "Path": "github.com/dependabot-fixtures/go-modules-lib",
        "Versions": [
                "v1.0.0",
                "v1.0.1",
                "v1.0.5",
                "v1.0.6",
                "v1.1.0",
                "v1.2.0-pre1",
                "v1.2.0-pre2"
        ]
}

real    0m1.705s
user    0m0.140s
sys     0m0.135s
[dependabot-core-dev] /tmp/go $ du -sk /home/dependabot/go/pkg/mod/cache
228     /home/dependabot/go/pkg/mod/cache

Other notes

This change revealed that the v2 release of our test fixture wasn't actually valid. I created a valid v3 release and added handling for invalid version errors.
I believe the usage of SharedHelpers.run_shell_command will escape the args being passed to go mod list -m -versions and go mod edit to prevent command injection. I'd appreciate a 2nd look at this though.

go list returns a different error message: go list -m: malformed module path "pkg-errors": missing dot in first path element

go list -m properly handles retractions so this test is now passing

Prior error: go list -m: loading module retractions for github.com/dependabot-fixtures/go-modules-lib/v2@v2.0.0: version "v2.0.0" invalid: go.mod has non-.../v2 module path "github.com/dependabot-fixtures/go-modules-lib" (and .../v2/go.mod does not exist) at revision v2.0.0

jeffwidman · 2021-11-20T05:51:19Z

As a workaround I found that providing a go.mod with no requirements was a way to sidestep the performance issue. We still end up getting the version information we are looking for and no extra modules are downloaded!

Very clever!

jurre

This looks great, so glad we're able to get rid of gomodules-extracted!

I believe the usage of SharedHelpers.run_shell_command will escape the args being passed to go mod list -m -versions and go mod edit to prevent command injection. I'd appreciate a 2nd look at this though.

That's right, unless we explicitly tell run_shell_command to allow_unsafe_shell_command it runs any command we pass through Shellwords which should protect against any command injection 👍

weitzj · 2021-11-22T11:30:35Z

Looking forward to this release.

common/lib/dependabot/shared_helpers.rb

Co-Authored-By: Mattt Zmuda <mattt@github.com> Co-Authored-By: Landon Grindheim <landongrindheim@github.com>

mctofu · 2021-11-23T18:36:53Z

go_modules/lib/dependabot/go_modules/update_checker/latest_version_finder.rb

-              File.write("go.mod", go_mod.content)
+              manifest = parse_manifest
+
+              # Set up an empty go.mod so 'go list -m' won't attempt to download dependencies. This


One other observation from testing is that we can also run go list -m -versions without a go.mod and it works the same way. However, it won't know about the exclude directives and won't be able to apply them for us so creating a go.mod with the excludes is still preferred.

jeffwidman · 2021-11-23T19:22:02Z

go_modules/helpers/main.go

-	case "getVersions":
-		var args updatechecker.Args
-		parseArgs(helperParams.Args, &args)
-		funcOut, funcErr = updatechecker.GetVersions(&args)
 	case "getVcsRemoteForImport":
 		var args importresolver.Args
 		parseArgs(helperParams.Args, &args)


May be able to delete this as well, I filed #4448 for that

jeffwidman · 2021-11-23T19:26:46Z

Nicely done!

I opened dependabot/gomodules-extracted#17 and dependabot/gomodules-extracted#18 as followups to this.

mctofu · 2021-11-24T21:42:28Z

I noticed one unanticipated side effect where Dependabot now updates major versions for +incompatible versions: #4453

mctofu added 7 commits November 19, 2021 10:18

Allow providing env to SharedHelpers.run_shell_command

9902940

Switch to go command to find available module versions

b507e47

Handle new error message for bad module paths

79a059e

go list returns a different error message: go list -m: malformed module path "pkg-errors": missing dot in first path element

Enable retracted module test

57b192f

go list -m properly handles retractions so this test is now passing

Handle invalid major version errors

1d5a203

Remove unused go_modules updatechecker native helper

def8e5e

mctofu force-pushed the mctofu/go-mod-list-versions branch from 644cf65 to def8e5e Compare November 19, 2021 19:55

mctofu changed the title ~~Use go mod list -m -versions to determine available versions of a go module~~ Use go list -m -versions to determine available versions of a go module Nov 20, 2021

mctofu marked this pull request as ready for review November 20, 2021 00:37

mctofu requested a review from a team as a code owner November 20, 2021 00:37

mctofu mentioned this pull request Nov 20, 2021

Draft: Go modules ignore conditions using go list -m -versions #3630

Closed

jurre approved these changes Nov 22, 2021

View reviewed changes

mattt reviewed Nov 22, 2021

View reviewed changes

common/lib/dependabot/shared_helpers.rb Outdated Show resolved Hide resolved

Clarify how env vars are passed to shell commands

16395d3

Co-Authored-By: Mattt Zmuda <mattt@github.com> Co-Authored-By: Landon Grindheim <landongrindheim@github.com>

mattt approved these changes Nov 22, 2021

View reviewed changes

mctofu commented Nov 23, 2021

View reviewed changes

mctofu merged commit 9060bc4 into main Nov 23, 2021

mctofu deleted the mctofu/go-mod-list-versions branch November 23, 2021 18:41

mctofu mentioned this pull request Nov 23, 2021

v0.169.0 #4446

Merged

jeffwidman mentioned this pull request Nov 23, 2021

Cleanup duplicate code by standardizing on one way - maybe eliminate native helper for go? #4448

Closed

jeffwidman reviewed Nov 23, 2021

View reviewed changes

This was referenced Nov 23, 2021

Add historical explanation of the code dependabot/gomodules-extracted#17

Merged

Archive this repo dependabot/gomodules-extracted#18

Closed

mctofu mentioned this pull request Nov 24, 2021

Dependabot updates major +incompatible versions for go modules #4453

Closed

jeffwidman mentioned this pull request Nov 30, 2021

Need to bump to a version of https://github.com/dependabot/gomodules-extracted/ that uses go 1.17 code #4153

Closed

jeffwidman mentioned this pull request Nov 30, 2021

Switch go_modules from getUpdatedVersion to a public go command #3599

Closed

mctofu added the L: go:modules Golang modules label Dec 30, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use `go list -m -versions` to determine available versions of a go module #4434

Use `go list -m -versions` to determine available versions of a go module #4434

mctofu commented Nov 19, 2021 •

edited

jeffwidman commented Nov 20, 2021

jurre left a comment

weitzj commented Nov 22, 2021

mctofu Nov 23, 2021

jeffwidman Nov 23, 2021

jeffwidman commented Nov 23, 2021

mctofu commented Nov 24, 2021

Use go list -m -versions to determine available versions of a go module #4434

Use go list -m -versions to determine available versions of a go module #4434

Conversation

mctofu commented Nov 19, 2021 • edited

Example output / performance

Other notes

jeffwidman commented Nov 20, 2021

jurre left a comment

Choose a reason for hiding this comment

weitzj commented Nov 22, 2021

mctofu Nov 23, 2021

Choose a reason for hiding this comment

jeffwidman Nov 23, 2021

Choose a reason for hiding this comment

jeffwidman commented Nov 23, 2021

mctofu commented Nov 24, 2021

Use `go list -m -versions` to determine available versions of a go module #4434

Use `go list -m -versions` to determine available versions of a go module #4434

mctofu commented Nov 19, 2021 •

edited