Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: relax module consistency checks if vendor/modules.txt is missing #37948

Open
dmgk opened this issue Mar 19, 2020 · 7 comments
Open

cmd/go: relax module consistency checks if vendor/modules.txt is missing #37948

dmgk opened this issue Mar 19, 2020 · 7 comments

Comments

@dmgk
Copy link
Contributor

@dmgk dmgk commented Mar 19, 2020

Strong consistency checks in go1.14 and requirement that vendor/modules.txt must be present make it hard to package Go software (e.g. as FreeBSD ports).

Because network is not available during the port build, ports are built in -mod=vendor mode. All required dependencies are fetched beforehand and then unpacked into vendor directory simulating the result of go mod vendor, except that vendor/modules.txt is not present. If software's go.mod specifies go.14, missing vendor/modules.txt is a hard error (... is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt) and working around it requires patching go.mod back to go1.13

Proposed change relaxes module consistency checks and switches back to go1.13 behaviour if vendor/modules.tx is not present during the build.

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Mar 19, 2020

Change https://golang.org/cl/224057 mentions this issue: cmd/go: relax module consistency checks if vendor/modules.txt is missing

@bcmills bcmills added this to the Unplanned milestone Mar 19, 2020
@jayconrod

This comment has been minimized.

Copy link
Contributor

@jayconrod jayconrod commented Mar 19, 2020

Could you say more about why go mod vendor can't be used build the vendor directory? That's the only supported mechanism for building a vendor directory in module mode.

Also, why not build a vendor/modules.txt file that satisfies the consistency check? Information from that file is used in several commands, so it's important that it's present and accurate.

@dmgk

This comment has been minimized.

Copy link
Contributor Author

@dmgk dmgk commented Mar 19, 2020

@jayconrod Because network is not available during the build. The list of required dependencies is generated from vendor/modules.txt (after go mod vendor) and committed along with port and vendor dir contents is then reconstructed using that list. We don't keep vendor/modules.txt because (a) it wasn't required with go1.13 and (b) dependencies list is already consistent with go.mod by construction.

@jayconrod

This comment has been minimized.

Copy link
Contributor

@jayconrod jayconrod commented Mar 19, 2020

If you have a full copy of vendor/modules.txt at an earlier point in the build, why not keep that? It sounds like you're able to run go mod vendor at some point, so I don't understand why it needs to be rebuilt after that.

You may be able to construct the vendor directory using go mod vendor offline by adding replace directives to go.mod, or an alternative go.mod file specified with -modfile. A local GOPROXY implementation may also be an option.

I don't think we should disable the vendor/modules.txt consistency check though. It's important.

@yurivict

This comment has been minimized.

Copy link

@yurivict yurivict commented Mar 19, 2020

Alternatively, you can add an option to go to optionally wave the consistency check.

(A full copy of vendor/modules.txt is available at some point during the port maintainer update, but it's problematic to keep it into the build phase within the FreeBSD ports framework.)

@yurivict

This comment has been minimized.

Copy link

@yurivict yurivict commented Mar 19, 2020

The design of Go assumes that network is available at all times during build, while during package build the network is not available for security reasons. The new consistency checking exacerbate this conflict, putting more and more burden on port developers/maintainers.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Mar 19, 2020

The consistency check does not use the network, by design. It compares the lexical contents of the go.mod and vendor/modules.txt files.

And note that replace directives in go.mod files also avoid the network. I would expect that in most packaging systems, a replace directive would express the intended substitution more clearly than a vendor tree.

uqs pushed a commit to freebsd/freebsd-ports that referenced this issue Apr 2, 2020
…sing

Starting from go1.14, go verifies that vendor/modules.txt matches the
requirements and replacements listed in the main module go.mod file, and it is
a hard failure if vendor/modules.txt is missing.

Relax module consistency checks and switch back to pre go1.14 behaviour if
vendor/modules.txt is missing and GO_NO_VENDOR_CHECKS=1 is set in the
environment regardless of go version requirement in go.mod.

Upstream PR: golang/go#37948

PR:		244783
Reported by:	Christopher Hall <hsw@bitmark.com>
Reviewed by:	mikael swills yuri
Approved by:	jlaffaye (maintainer timeout, 2 weeks)
MFH:		2020Q2
Differential Revision:	https://reviews.freebsd.org/D24122


git-svn-id: svn+ssh://svn.freebsd.org/ports/head@530387 35697150-7ecd-e111-bb59-0022644237b5
uqs pushed a commit to freebsd/freebsd-ports that referenced this issue Apr 2, 2020
…sing

Starting from go1.14, go verifies that vendor/modules.txt matches the
requirements and replacements listed in the main module go.mod file, and it is
a hard failure if vendor/modules.txt is missing.

Relax module consistency checks and switch back to pre go1.14 behaviour if
vendor/modules.txt is missing and GO_NO_VENDOR_CHECKS=1 is set in the
environment regardless of go version requirement in go.mod.

Upstream PR: golang/go#37948

PR:		244783
Reported by:	Christopher Hall <hsw@bitmark.com>
Reviewed by:	mikael swills yuri
Approved by:	jlaffaye (maintainer timeout, 2 weeks)
MFH:		2020Q2
Differential Revision:	https://reviews.freebsd.org/D24122
Jehops pushed a commit to Jehops/freebsd-ports that referenced this issue Apr 2, 2020
…sing

Starting from go1.14, go verifies that vendor/modules.txt matches the
requirements and replacements listed in the main module go.mod file, and it is
a hard failure if vendor/modules.txt is missing.

Relax module consistency checks and switch back to pre go1.14 behaviour if
vendor/modules.txt is missing and GO_NO_VENDOR_CHECKS=1 is set in the
environment regardless of go version requirement in go.mod.

Upstream PR: golang/go#37948

PR:		244783
Reported by:	Christopher Hall <hsw@bitmark.com>
Reviewed by:	mikael swills yuri
Approved by:	jlaffaye (maintainer timeout, 2 weeks)
MFH:		2020Q2
Differential Revision:	https://reviews.freebsd.org/D24122


git-svn-id: svn+ssh://svn.freebsd.org/ports/head@530387 35697150-7ecd-e111-bb59-0022644237b5
uqs pushed a commit to freebsd/freebsd-ports that referenced this issue Apr 3, 2020
lang/go: relax module consistency checks if vendor/modules.txt is missing

Starting from go1.14, go verifies that vendor/modules.txt matches the
requirements and replacements listed in the main module go.mod file, and it is
a hard failure if vendor/modules.txt is missing.

Relax module consistency checks and switch back to pre go1.14 behaviour if
vendor/modules.txt is missing and GO_NO_VENDOR_CHECKS=1 is set in the
environment regardless of go version requirement in go.mod.

Upstream PR: golang/go#37948

PR:		244783
Reported by:	Christopher Hall <hsw@bitmark.com>
Reviewed by:	mikael swills yuri
Approved by:	jlaffaye (maintainer timeout, 2 weeks)
Differential Revision:	https://reviews.freebsd.org/D24122

Approved by:	ports-secteam (joneum)
mat813 pushed a commit to mat813/freebsd-ports that referenced this issue Apr 6, 2020
…sing

Starting from go1.14, go verifies that vendor/modules.txt matches the
requirements and replacements listed in the main module go.mod file, and it is
a hard failure if vendor/modules.txt is missing.

Relax module consistency checks and switch back to pre go1.14 behaviour if
vendor/modules.txt is missing and GO_NO_VENDOR_CHECKS=1 is set in the
environment regardless of go version requirement in go.mod.

Upstream PR: golang/go#37948

PR:		244783
Reported by:	Christopher Hall <hsw@bitmark.com>
Reviewed by:	mikael swills yuri
Approved by:	jlaffaye (maintainer timeout, 2 weeks)
MFH:		2020Q2
Differential Revision:	https://reviews.freebsd.org/D24122


git-svn-id: https://svn.freebsd.org/ports/head@530387 35697150-7ecd-e111-bb59-0022644237b5
mat813 pushed a commit to mat813/freebsd-ports that referenced this issue Apr 6, 2020
lang/go: relax module consistency checks if vendor/modules.txt is missing

Starting from go1.14, go verifies that vendor/modules.txt matches the
requirements and replacements listed in the main module go.mod file, and it is
a hard failure if vendor/modules.txt is missing.

Relax module consistency checks and switch back to pre go1.14 behaviour if
vendor/modules.txt is missing and GO_NO_VENDOR_CHECKS=1 is set in the
environment regardless of go version requirement in go.mod.

Upstream PR: golang/go#37948

PR:		244783
Reported by:	Christopher Hall <hsw@bitmark.com>
Reviewed by:	mikael swills yuri
Approved by:	jlaffaye (maintainer timeout, 2 weeks)
Differential Revision:	https://reviews.freebsd.org/D24122

Approved by:	ports-secteam (joneum)


git-svn-id: https://svn.freebsd.org/ports/branches/2020Q2@530488 35697150-7ecd-e111-bb59-0022644237b5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.