Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

go.sum file only partially updated #3017

Closed
prymitive opened this issue Jan 1, 2019 · 7 comments
Closed

go.sum file only partially updated #3017

prymitive opened this issue Jan 1, 2019 · 7 comments
Labels
manager:gomod Go Modules priority-3-medium Default priority, "should be done" but isn't prioritised ahead of others type:feature Feature (new functionality)

Comments

@prymitive
Copy link

What Renovate type are you using?
Renovate GitHub App

Describe the bug
go.sum file includes 2 checksums per dependency, example:

 github.com/gin-contrib/gzip v0.0.0-20181105040055-d9117ce29a07 h1:JvT9X9GVkqtUrIM41BsC7wlnTyZ7QAYERN7x3a+Idek=
 github.com/gin-contrib/gzip v0.0.0-20181105040055-d9117ce29a07/go.mod h1:Iw1jLht8y5t9ZN91Qt/XzlT18He4kT9+LpFJkj4H/vY=

So an updated Go module dependency should trigger a PR with 3 changes:

  1. version change in go.mod
  2. two new lines per dependency in go.sum with checksums
    Yet for some reason I got 2 PRs that include only one new line per dependency in go.sum - checksum for go.mod, the other expected line is missing (chore(deps): update github.com/gin-contrib/cors commit hash to 5e7acb1 prymitive/karma#345 chore(deps): update github.com/gin-contrib/gzip commit hash to 0eb78e9 prymitive/karma#346).
    Running go get or go build on top of that PR results in missing lines being added:
$ git reset --hard
HEAD is now at fe070ea chore(deps): update github.com/gin-contrib/gzip commit hash to 0eb78e9
$ GO111MODULE=on go get
$ git diff
diff --git a/go.sum b/go.sum
index b24c828..aba9549 100644
--- a/go.sum
+++ b/go.sum
@@ -26,6 +26,7 @@ github.com/gin-contrib/cors v0.0.0-20181206035131-7c641a7a7dc5 h1:d4cpoJSg7KnKZ5
 github.com/gin-contrib/cors v0.0.0-20181206035131-7c641a7a7dc5/go.mod h1:cw+u9IsAkC16e42NtYYVCLsHYXE98nB3M7Dr9mLSeH4=
 github.com/gin-contrib/gzip v0.0.0-20181105040055-d9117ce29a07 h1:JvT9X9GVkqtUrIM41BsC7wlnTyZ7QAYERN7x3a+Idek=
 github.com/gin-contrib/gzip v0.0.0-20181105040055-d9117ce29a07/go.mod h1:Iw1jLht8y5t9ZN91Qt/XzlT18He4kT9+LpFJkj4H/vY=
+github.com/gin-contrib/gzip v0.0.0-20190101135810-0eb78e93402e h1:1FdtaNCz0eZF40fxeCefoQRPQkaGWE4Szw0MhEisAmw=
 github.com/gin-contrib/gzip v0.0.0-20190101135810-0eb78e93402e/go.mod h1:fGBJBCdt6qCZuCAOwWuFhBB4OOq9EFqlo5dEaFhhu5w=
 github.com/gin-contrib/sse v0.0.0-20170109093832-22d885f9ecc7 h1:AzN37oI0cOS+cougNAV9szl6CVoj2RYwzS3DpUQNtlY=
 github.com/gin-contrib/sse v0.0.0-20170109093832-22d885f9ecc7/go.mod h1:VJ0WA2NBN22VlZ2dKZQPAPnyWw5XTlK1KymzLKsr59s=
@@ -108,6 +109,7 @@ golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5h
 golang.org/x/sys v0.0.0-20181205085412-a5c9d58dba9a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20181221143128-b4a75ba826a6 h1:IcgEB62HYgAhX0Nd/QrVgZlxlcyxbGQHElLUhW2X4Fo=
 golang.org/x/sys v0.0.0-20181221143128-b4a75ba826a6/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20181228144115-9a3f9b0469bb h1:pf3XwC90UUdNPYWZdFjhGBE7DUFuK3Ct1zWmZ65QN30=
 golang.org/x/sys v0.0.0-20181228144115-9a3f9b0469bb/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=

Did you see anything helpful in debug logs?
If you are using the Renovate App, log into https://renovatebot.com/dashboard and locate the correct job log for when the problem occurred (e.g. when the PR was created).

Nothing unexpected in the log.

To Reproduce
Steps to reproduce the behavior:

  1. Force rebase on chore(deps): update github.com/gin-contrib/gzip commit hash to 0eb78e9 prymitive/karma#346
  2. Wait for new commit to be pushed
  3. Build from that branch
  4. Check git diff

Expected behavior

Renovate PR should be complete and build without generating any git diff.

Screenshots

Additional context

I wasn't able to reproduce this issue locally with Go 1.11.4, forcing a rebase still creates a commit with only half of go.sum updates. Both updated dependencies were adding support for Go modules, so in both cases go.(mod|sum) file was added in the version those deps were being updated to, which might be a factor.

@rarkins
Copy link
Collaborator

rarkins commented Jan 1, 2019

Thanks for the detailed report. Do you know if there is any documentation from the Golang team on exactly what should be expected when a go.sum is updated? e.g. the "2 checksums"?

FYI the default Renovate behaviour is that go get is run with just the go.mod and not any of the source files. Partly because running with source files sometimes takes 100x more time, and partly because it also increases the chance of changes to go.sum that are unrelated to the update being done by Renovate. So if you were to run go get against just the go.mod in a temporary folder, you might find the same result.

I do have an option - currently controllable via my admin settings only - to run go get against the full repository. If you are able to test this out locally and let me know if you think that solves it, then I can make the switch for this repo and we see if it solves it as expected.

@prymitive
Copy link
Author

I was trying to find some documentation on go.sum file format, but wasn't able to find any technical docs so far, only high level description of it. There are some test files (https://github.com/golang/go/blob/master/src/cmd/go/testdata/script/mod_sum_replaced.txt) that give some context on how it should behave, but it's not really a spec one can follow.

You're right about the root cause - I can replicate it by copying only go.mod and running go get on that - go.sum is created only with /go.mod checksums. Looks like other lines are populated only if I import given dependency. If I add a main.go file with:

package main

import _ "gopkg.in/yaml.v2"

func main() {}

I do get extra line for yaml.v2 in go.sum after running go get -d.

@rarkins
Copy link
Collaborator

rarkins commented Jan 1, 2019

I have switched your repo to use gitFs instead of API. That way the entire file system is present locally when go get is run. Can you try checking the "If you want to rebase/retry this PR, check this box" checkbox in the PR and see if it fixes it already? It may take several minutes to run though - this is the downside of a full source go get.

@prymitive
Copy link
Author

Thanks, unfortunately that failed because there's assets.go file that's auto generated on build (via Makefile) - that project is a web app with Go backend and React fronted, js/css assets are compiled into the Go binary but that means those needs to be (re)build when building Go binary. I could add those to the VCS but it's not side effect free, so I would rather keep it generated as needed.

I think that running on top of full sources would have other benefits - like being able to run go mod tidy as a feature (#2594), which seems like a great addition sincemod tidy is suppose to ensure that all deps for all OS/arch are present, but that would indeed be more expensive job for renovate.

I've raised golang/go#29492 to get some clarity on what to expect from go.sum. Thanks!

@rarkins rarkins added type:feature Feature (new functionality) manager:gomod Go Modules priority-3-medium Default priority, "should be done" but isn't prioritised ahead of others needs-requirements labels Jan 10, 2019
@prymitive
Copy link
Author

I have switched your repo to use gitFs instead of API.

@rarkins as mentioned above I can't easily take advantage of gitFS, although I appreciate enabling it for me, so maybe you want to switch it off as it makes those tasks more expensive to run? Plus when it fails with gitFS it generates a long error message. Thanks!

@rarkins
Copy link
Collaborator

rarkins commented Mar 3, 2019

@prymitive I've done that now, although I plan to switch the app over 100% to gitFs in coming months

@rarkins
Copy link
Collaborator

rarkins commented Mar 5, 2019

I'm going to close this one, as there's nothing can be done right now, e.g.

  • Do not support a build step in Renovate.. yet
  • Does not work if using plain go.mod/go.sum without full source

@rarkins rarkins closed this as completed Mar 5, 2019
bflad added a commit to hashicorp/terraform-provider-aws that referenced this issue Oct 28, 2019
bflad added a commit to hashicorp/terraform-provider-aws that referenced this issue Dec 5, 2019
bflad pushed a commit to hashicorp/terraform-provider-aws that referenced this issue Dec 5, 2019
* Update module aws/aws-sdk-go to v1.25.48

* deps: Add missing github.com/aws-sdk-go@v1.25.48 checksum

Reference: renovatebot/renovate#3017

Updated via: `go mod tidy`
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 16, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
manager:gomod Go Modules priority-3-medium Default priority, "should be done" but isn't prioritised ahead of others type:feature Feature (new functionality)
Projects
None yet
Development

No branches or pull requests

2 participants