cmd/go: allow verifying vendored code

[krancour]

go mod verify is extremely useful for validating the integrity of modules in the local cache.

It would be great if projects that choose to vendor their modules (then presumably building with go build -mod vendor ...) had a similar command to verify the integrity of modules in that directory.

This would satisfy a major requirement that many projects need to account for in their CI process-- ensuring that vendored code hasn't been tampered with.

[bcmills]

This is still on our radar, but probably not happening for 1.13. (We have a lot to do this cycle!)

I'm hoping to get to it in 1.14, but we don't have a 1.14 milestone defined yet.

[sgreene570]

I'm hoping to get to it in 1.14, but we don't have a 1.14 milestone defined yet.

@bcmills is there an update on this? Switching from dep to go mod means losing the ability to verify vendored dependencies before performing builds, etc, which is a big concern.

[bcmills]

@sgreene570, note that in the interim you can simply re-run go mod vendor and check for diffs.

Verifying the checksums of the vendored modules requires the full module content (because that is what is checksummed), so either way you're going to have to download the full module into the local module cache.

[bcmills]

This functionality would also be useful for #36852.

[bcmills]

@jayconrod, @matloob: I think we should aim to get this implemented for 1.15.

[ianlancetaylor]

Do you still hope to get this into 1.15?

[bcmills]

No, it's definitely not happening for 1.15.

[ianlancetaylor]

Moving milestone to Backlog.

[iholder-redhat]

This is indeed valuable! Projects that choose to vendor their modules usually do not want to count on every package to be available 100% of the time, therefore validation that enforces re-downloading the package is problematic.

Would be happy to see this implemented.

[ArsenArsen]

There seems to be a fundamental issue here, vendor directories exclude many files not used in the actual build. Verifying against go.sum partially becomes impossible.

I'd like to automatically generate vendored tarballs from sources, while being able to verify them, so this issue becomes relevant to me.

Is there any discussed solution that went unrecorded in this thread?

Thanks in advance

[bcmills]

@ArsenArsen, the plan at the moment is to have go mod verify download the full module source code for each dependency.

Then we can demonstrate the integrity of the vendor tree by showing that, for each package in the vendor tree, the directory for that package contains exactly the expected subset of source files, and the contents of each of those files match the contents of the corresponding file in the module cache.

[bcmills]

(However, note that go mod verify cannot be implemented for vendor trees in a way that is independent of the module cache.)

[ArsenArsen]

@bcmills Hm, while that works and will produce valid results, the thing we're implementing requires us to be able to do so without the network.

Would it be possible to create a vendor archive that contains all the code required for building and verification to run in this constrained environment? I was considering shipping a tarred up version of GOMODCACHE, however that is likely unsupported.

Thanks in advance, and for the quick response.

[bcmills]

Would it be possible to create a vendor archive that contains all the code required for building and verification to run in this constrained environment?

In theory you could checksum each package, but the go.sum file (and sum.golang.org) only contains checksums for modules and I don't see that changing.

I was considering shipping a tarred up version of GOMODCACHE, however that is likely unsupported.

Actually, that would work (once we implement go mod verify for vendored code at all).
You can set GOPROXY to point to $GOMODCACHE/cache/download to treat a module cache as a module proxy. (See #43646.)