Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: allow verifying vendored code #27348

Open
krancour opened this issue Aug 29, 2018 · 14 comments
Open

cmd/go: allow verifying vendored code #27348

krancour opened this issue Aug 29, 2018 · 14 comments
Labels
FeatureRequest modules NeedsInvestigation
Milestone

Comments

@krancour
Copy link

@krancour krancour commented Aug 29, 2018

go mod verify is extremely useful for validating the integrity of modules in the local cache.

It would be great if projects that choose to vendor their modules (then presumably building with go build -mod vendor ...) had a similar command to verify the integrity of modules in that directory.

This would satisfy a major requirement that many projects need to account for in their CI process-- ensuring that vendored code hasn't been tampered with.

@FiloSottile FiloSottile added NeedsInvestigation FeatureRequest modules labels Aug 29, 2018
@FiloSottile FiloSottile added this to the Go1.12 milestone Aug 29, 2018
@FiloSottile FiloSottile changed the title cmd/go: feature request: verify vendored code cmd/go: allow verifying vendored code Aug 29, 2018
@bcmills bcmills removed this from the Go1.12 milestone Oct 25, 2018
@bcmills bcmills added this to the Go1.13 milestone Oct 25, 2018
@bcmills
Copy link
Member

@bcmills bcmills commented Feb 15, 2019

This is still on our radar, but probably not happening for 1.13. (We have a lot to do this cycle!)

I'm hoping to get to it in 1.14, but we don't have a 1.14 milestone defined yet.

@bcmills bcmills removed this from the Go1.13 milestone Feb 15, 2019
@bcmills bcmills added this to the Unplanned milestone Feb 15, 2019
stp-ip added a commit to stp-ip/caddy that referenced this issue Mar 5, 2019
amshinde added a commit to kata-containers/tests that referenced this issue Jul 31, 2019
While we move to go modules, perform the dep check for repos
that still use dep.
Run `go mod verify` instead for go modules.
Note, this just verifies the integrity of modules in the local
cache. We would have instead wanted to verify the vendored code
here, but that is still not supported.
golang/go#27348

Fixes #1879

Signed-off-by: Archana Shinde <archana.m.shinde@intel.com>
@bcmills bcmills removed this from the Unplanned milestone Aug 15, 2019
@bcmills bcmills added this to the Go1.14 milestone Aug 15, 2019
@rsc rsc removed this from the Go1.14 milestone Oct 9, 2019
@rsc rsc added this to the Backlog milestone Oct 9, 2019
stevendanna added a commit to chef/automate that referenced this issue Nov 4, 2019
`go mod verify` does not verify the vendored copies of
dependencies:

golang/go#27348

As such, it seems that this change snuck in. This commit is the result
of commit the all changes produced by `make revendor` on master.

Signed-off-by: Steven Danna <steve@chef.io>
stevendanna added a commit to chef/automate that referenced this issue Nov 4, 2019
`go mod verify` does not verify the vendored copies of
dependencies:

golang/go#27348

As such, it seems that this change snuck in. This commit is the result
of commit the all changes produced by `make revendor` on master.

Signed-off-by: Steven Danna <steve@chef.io>
@sgreene570
Copy link

@sgreene570 sgreene570 commented Jan 2, 2020

I'm hoping to get to it in 1.14, but we don't have a 1.14 milestone defined yet.

@bcmills is there an update on this? Switching from dep to go mod means losing the ability to verify vendored dependencies before performing builds, etc, which is a big concern.

@bcmills
Copy link
Member

@bcmills bcmills commented Jan 10, 2020

@sgreene570, note that in the interim you can simply re-run go mod vendor and check for diffs.

Verifying the checksums of the vendored modules requires the full module content (because that is what is checksummed), so either way you're going to have to download the full module into the local module cache.

@bcmills
Copy link
Member

@bcmills bcmills commented Jan 31, 2020

This functionality would also be useful for #36852.

@bcmills
Copy link
Member

@bcmills bcmills commented Jan 31, 2020

@jayconrod, @matloob: I think we should aim to get this implemented for 1.15.

@bcmills bcmills removed this from the Backlog milestone Jan 31, 2020
@bcmills bcmills added this to the Go1.15 milestone Jan 31, 2020
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented May 19, 2020

Do you still hope to get this into 1.15?

@bcmills bcmills removed this from the Go1.15 milestone May 19, 2020
@bcmills bcmills added this to the Go1.16 milestone May 19, 2020
@bcmills
Copy link
Member

@bcmills bcmills commented May 19, 2020

No, it's definitely not happening for 1.15.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Apr 19, 2021

Moving milestone to Backlog.

@ianlancetaylor ianlancetaylor removed this from the Go1.17 milestone Apr 19, 2021
@ianlancetaylor ianlancetaylor added this to the Backlog milestone Apr 19, 2021
@iholder-redhat
Copy link

@iholder-redhat iholder-redhat commented May 26, 2021

This is indeed valuable! Projects that choose to vendor their modules usually do not want to count on every package to be available 100% of the time, therefore validation that enforces re-downloading the package is problematic.

Would be happy to see this implemented.

@ArsenArsen
Copy link

@ArsenArsen ArsenArsen commented Feb 28, 2022

There seems to be a fundamental issue here, vendor directories exclude many files not used in the actual build. Verifying against go.sum partially becomes impossible.

I'd like to automatically generate vendored tarballs from sources, while being able to verify them, so this issue becomes relevant to me.

Is there any discussed solution that went unrecorded in this thread?

Thanks in advance

@bcmills
Copy link
Member

@bcmills bcmills commented Feb 28, 2022

@ArsenArsen, the plan at the moment is to have go mod verify download the full module source code for each dependency.

Then we can demonstrate the integrity of the vendor tree by showing that, for each package in the vendor tree, the directory for that package contains exactly the expected subset of source files, and the contents of each of those files match the contents of the corresponding file in the module cache.

@bcmills
Copy link
Member

@bcmills bcmills commented Feb 28, 2022

(However, note that go mod verify cannot be implemented for vendor trees in a way that is independent of the module cache.)

@ArsenArsen
Copy link

@ArsenArsen ArsenArsen commented Feb 28, 2022

@bcmills Hm, while that works and will produce valid results, the thing we're implementing requires us to be able to do so without the network.

Would it be possible to create a vendor archive that contains all the code required for building and verification to run in this constrained environment? I was considering shipping a tarred up version of GOMODCACHE, however that is likely unsupported.

Thanks in advance, and for the quick response.

@bcmills
Copy link
Member

@bcmills bcmills commented Mar 2, 2022

Would it be possible to create a vendor archive that contains all the code required for building and verification to run in this constrained environment?

In theory you could checksum each package, but the go.sum file (and sum.golang.org) only contains checksums for modules and I don't see that changing.

I was considering shipping a tarred up version of GOMODCACHE, however that is likely unsupported.

Actually, that would work (once we implement go mod verify for vendored code at all).
You can set GOPROXY to point to $GOMODCACHE/cache/download to treat a module cache as a module proxy. (See #43646.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
FeatureRequest modules NeedsInvestigation
Projects
None yet
Development

No branches or pull requests

9 participants