Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: non-top-level packages ending in "/vendor" are omitted from module zip files #37397

Open
bep opened this issue Feb 24, 2020 · 9 comments
Open

Comments

@bep
Copy link
Contributor

@bep bep commented Feb 24, 2020

The following program fails in both go1.14rc1 and go1.13.7 darwin/amd64.

package main

import (
	"fmt"

	"github.com/bep/testmodlib"
	"github.com/bep/testmodlib/somepackage"
	"github.com/bep/testmodlib/somepackage/vendor"
	"github.com/bep/testmodlib/somepackage/vendors"
)

func main() {
	fmt.Println(testmodlib.Hello())
	fmt.Println(somepackage.Hello())
	fmt.Println(vendor.Hello())
	fmt.Println(vendors.Hello())
}

Fails with:

build github.com/bep/temp: cannot load github.com/bep/testmodlib/somepackage/vendor: module github.com/bep/testmodlib@latest found (v1.0.1), but does not contain package github.com/bep/testmodlib/somepackage/vendor

If I add a replace directive for that module:

replace github.com/bep/testmodlib => /Users/bep/dev/go/bep/testmodlib

It compiles/runs fine:

testmodlib
somepackage
vendor package
vendors package

I understand that the vendor top level folder in a module has a special meaning, but according to https://golang.org/ref/spec#Packages that package name should be fine -- and just deleting it sounds rather harsh.

I assume this is related to #31562 -- I have not read that one in detail, but I don't see how the conclusion can be correct.

  • github.com/bep/testmodlib/somepackage/vendor seem to be a valid package with a replace, and I assume that the compiler works as expected here.
  • Also, a module may contain non-Go folders (test resources etc. to make the test pass).
@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Feb 24, 2020

Change https://golang.org/cl/220640 mentions this issue: cmd/go/internal/modfetch: delete unused isVendoredPackage function

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Feb 24, 2020

I assume this is related to #31562 -- I have not read that one in detail, but I don't see how the conclusion can be correct.

You are correct: this is closely related. https://play.golang.org/p/zCP7U8cKAGc

@bcmills bcmills added this to the Unplanned milestone Feb 24, 2020
@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Feb 24, 2020

CC @jayconrod @matloob, @FiloSottile FYI.

(I don't think we can plausibly fix this without invalidating checksums, though.)

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Feb 24, 2020

Change https://golang.org/cl/220657 mentions this issue: zip: document isVendoredPackage false-positives

@bcmills bcmills changed the title modules: github.com/bep/testmodlib/somepackage/vendor fails cmd/go: packages named "vendor" are erroneously omitted from module zip files Feb 24, 2020
@bcmills bcmills changed the title cmd/go: packages named "vendor" are erroneously omitted from module zip files cmd/go: non-top-level packages named "vendor" are erroneously omitted from module zip files Feb 24, 2020
@bcmills bcmills changed the title cmd/go: non-top-level packages named "vendor" are erroneously omitted from module zip files cmd/go: non-top-level packages ending in "/vendor" are omitted from module zip files Feb 24, 2020
gopherbot pushed a commit that referenced this issue Feb 24, 2020
This function is apparently unused since CL 204917.

Updates #35290
Updates #37397

Change-Id: Id7f5f5d5176fdbd1c5c6227e81d0854ceafc3f12
Reviewed-on: https://go-review.googlesource.com/c/go/+/220640
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
gopherbot pushed a commit to golang/mod that referenced this issue Feb 24, 2020
I had thought that the known bug in isVendoredPackage was strictly
conservative, but it turns out not to be.

Updates golang/go#37397
Updates golang/go#31562

Change-Id: I60f6062c41ec2d116766930f751d7df083535355
Reviewed-on: https://go-review.googlesource.com/c/mod/+/220657
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
@FiloSottile

This comment has been minimized.

Copy link
Member

@FiloSottile FiloSottile commented Feb 24, 2020

I suppose we could gate this sort of changes on the go.mod version?

@jayconrod

This comment has been minimized.

Copy link
Contributor

@jayconrod jayconrod commented Feb 25, 2020

@FiloSottile Unfortunately not. Old versions of cmd/go should still work with go.mod files that declare newer versions. So different versions of Go would produce different sums.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Feb 25, 2020

That proposal is #30369, but it seems unlikely to be accepted.

@dmitshur

This comment has been minimized.

Copy link
Member

@dmitshur dmitshur commented Mar 7, 2020

(I don't think we can plausibly fix this without invalidating checksums, though.)

To be more specific and confirm, do you mean without invalidating checksums of the affected modules; i.e., modules that have a non-top-level package with "/vendor" import path suffix that was unfortunately excluded from the .zip?

@jayconrod

This comment has been minimized.

Copy link
Contributor

@jayconrod jayconrod commented Mar 9, 2020

To be more specific and confirm, do you mean without invalidating checksums of the affected modules; i.e., modules that have a non-top-level package with "/vendor" import path suffix that was unfortunately excluded from the .zip?

Right. If we fixed this, then for a given repository at a given commit, we would include a different set of files in the module zip file. The zip file would have a different sum.

We don't have any safe mechanism for making that kind of change (though #30369, if implemented, is a possibility). The only change we can make is adding files that previously caused hard errors (for example, we could expand the set of allowed characters or increase the file size limit). We can't add files that were previously excluded.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants
You can’t perform that action at this time.