Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

std,cmd: add test to ensure that go.mod and vendor are in sync #36852

Closed
dmitshur opened this issue Jan 28, 2020 · 12 comments
Closed

std,cmd: add test to ensure that go.mod and vendor are in sync #36852

dmitshur opened this issue Jan 28, 2020 · 12 comments

Comments

@dmitshur
Copy link
Contributor

@dmitshur dmitshur commented Jan 28, 2020

We should make it so that trybots catch and report when the src/go.mod and src/vendor directories are not in sync, to prevent them from going out of sync without anyone noticing.

See #36851 for more context.

/cc @golang/osp-team

@dmitshur
Copy link
Contributor Author

@dmitshur dmitshur commented Feb 21, 2020

@bcmills I think this issue is the same as issue #36907 (which was created slightly later), and can be considered resolved via CL 217218. Do you agree?

@dmitshur
Copy link
Contributor Author

@dmitshur dmitshur commented Feb 21, 2020

Oh, I misread, this is about divergence between go.mod and vendor. That issue was about divergence between two different go.mods.

@dmitshur
Copy link
Contributor Author

@dmitshur dmitshur commented Feb 21, 2020

That said, the test in CL 217218 seems to check for this issue too, does it not?

Edit: It only checks for missing vendored packages, not that the vendored packages are identical.

@bcmills
Copy link
Member

@bcmills bcmills commented Feb 24, 2020

Correct. The new test only checks that vendor/modules.txt is consistent with go.mod: it does not verify that the actual vendored source code is unchanged.

@dmitshur
Copy link
Contributor Author

@dmitshur dmitshur commented May 5, 2020

@bcmills I have some questions about this, since you have more context and might be able to answer more easily.

Do you think this can be implemented by adding a test that does go mod vendor in the two modules and does the equivalent of git diff? Does go mod vendor need internet beyond accessing the module mirror? Do you know of any precedent for doing the "equivalent of git diff" in tests in the standard library? Final question, would it be sufficient to do this check on one or two configurations (e.g., linux/amd64), or can you think of a reason to do it on more/all builder types?

@dmitshur
Copy link
Contributor Author

@dmitshur dmitshur commented May 5, 2020

I've confirmed there isn't an existing test (on master) that catches this issue by making a trivial edit in src/cmd/vendor and running src/all.bash; it reported "ALL TESTS PASSED".

@bcmills
Copy link
Member

@bcmills bcmills commented May 5, 2020

Do you think this can be implemented by adding a test that does go mod vendor in the two modules and does the equivalent of git diff?

Yes.

Does go mod vendor need internet beyond accessing the module mirror?

It should need only the module mirror.

Do you know of any precedent for doing the "equivalent of git diff" in tests in the standard library?

There are a few tests that check that generated source files are clean (such as cmd/go.TestDocsUpToDate), but to my knowledge none that rely on git (or on being able to write to GOROOT).

Another approach might be to set up an overlay of the std module in a temporary directory and run go mod vendor within that overlay, then diff the resulting trees (using diff -r or an equivalent Go function, perhaps implemented in terms of filepath.Walk).

The misc/reboot test illustrates how to make an overlay of GOROOT/src in order to test commands that mutate GOROOT. (Unfortunately, due to the fact that the misc repo does not have a fully-qualified import path in GOPATH mode, that part of the repo contains many duplicate copies of the file overlaydir_test.go that constructs the actual overlay.)

Final question, would it be sufficient to do this check on one or two configurations (e.g., linux/amd64), or can you think of a reason to do it on more/all builder types?

go mod vendor is intended to be platform-independent, so running the test on one or two configurations should suffice.

@gopherbot
Copy link

@gopherbot gopherbot commented Sep 18, 2020

Change https://golang.org/cl/255859 mentions this issue: misc/update: add program to update golang.org/x dependencies

@gopherbot
Copy link

@gopherbot gopherbot commented Sep 21, 2020

Change https://golang.org/cl/256357 mentions this issue: cmd/updatestd: add program to maintain standard library modules

@dmitshur
Copy link
Contributor Author

@dmitshur dmitshur commented Sep 24, 2020

Thanks for answering @bcmills. Based on those answers (and our other discussion of this topic), I think we want a normal test that is skipped in -short test mode (so it only runs on longtest builders or when executed locally manually).

As long as it's possible to implement a reasonably fast/efficient test without needing to rely on git, I don't see a reason for the test to be implemented out of tree; that would make it difficult to reproduce the result of builders locally, and it would contribute negatively to #39054. The cmd/internal/moddeps, misc/reboot, and test/winbatch.go tests may be good sources of inspiration. (To avoid needing git, a temporary directory can be used to compute the expected result, and check the tree against that.)

So, I'll retitle this to be less about x/build and builders and more about investigating adding a test in the main repo. Once a test is added, it will be automatically executed by longtest builders (and via SlowBots). (If it turns out to be infeasible to add as a normal test in-tree, we can reconsider.)

@dmitshur dmitshur changed the title x/build: add check to trybot to catch src/go.mod and src/vendor divergence std,cmd: add test to ensure that go.mod and vendor are in sync Sep 24, 2020
@dmitshur dmitshur removed the Builders label Sep 24, 2020
@dmitshur
Copy link
Contributor Author

@dmitshur dmitshur commented Oct 22, 2020

Managing backports can become much more expensive if skew is introduced in a new major Go release. We've been lucky that go.mod and vendor skew hasn't made it into any major Go release so far (although it did happen on tip, e.g., #36851), but I think it's important we have an automated measure so we can be confident it can't happen due to human error.

I'm going to make this a release-blocker for Go 1.16: we should either resolve this issue by adding a test, or if there isn't enough time, we must manually verify that there is no skew introduced in Go 1.16 release candidates and the final release.

CC @golang/release.

@gopherbot
Copy link

@gopherbot gopherbot commented Jan 14, 2021

Change https://golang.org/cl/283643 mentions this issue: cmd/internal/moddeps: check content of all modules in GOROOT

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
@toothrot @dmitshur @bcmills @gopherbot and others