Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: document restrictions on module zip extraction #33778

Open
leighmcculloch opened this issue Aug 22, 2019 · 9 comments

Comments

@leighmcculloch
Copy link
Contributor

commented Aug 22, 2019

What version of Go are you using (go version)?

$ go version
go version go1.13rc1 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN="/home/leighmcculloch/local/bin"
GOCACHE="/home/leighmcculloch/.cache/go-build"
GOENV="/home/leighmcculloch/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/leighmcculloch/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/leighmcculloch/local/bin/go/1.13rc1"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/leighmcculloch/local/bin/go/1.13rc1/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/leighmcculloch/devel/test/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build749101513=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Attempted to go get a package at a version.

go get github.com/gobuffalo/packr@v1.21.4

What did you expect to see?

The packages to be go getd without any issue.

What did you see instead?

go get error'd saying that the package in the sum db is 410 Gone.

$ go get github.com/gobuffalo/packr@v1.21.4
go: finding github.com/gobuffalo/packr v1.21.4
go: finding github.com v1.21.4
go: finding github.com/gobuffalo v1.21.4
go: finding github.com/gobuffalo/packr v1.21.4
go: downloading github.com/gobuffalo/packr v1.21.4
verifying github.com/gobuffalo/packr@v1.21.4: github.com/gobuffalo/packr@v1.21.4: reading https://sum.golang.org/lookup/github.com/gobuffalo/packr@v1.21.4: 410 Gone

If I visit the sum.golang.org URL manually in a browser, I see this error:

not found: unzip /tmp/gopath/pkg/mod/cache/download/github.com/gobuffalo/packr/@v/v1.21.4.zip: case-insensitive file name collision: "SHOULDERS.md" and "shoulders.md"

The packr project did indeed have two files named SHOULDERS.md and shoulders.md in v1.21.4.

Setting GONOSUMDB and turning off GOPROXY moves past the limitation in sum.golang.org, but then a similar error appears locally:

$ GONOSUMDB=github.com/gobuffalo/packr GOPROXY=direct go1.13rc1 get github.com/gobuffalo/packr@v1.21.4
go: finding github.com/gobuffalo/packr v1.21.4
go: downloading github.com/gobuffalo/packr v1.21.4
go: extracting github.com/gobuffalo/packr v1.21.4
-> unzip /home/leighmcculloch/go/pkg/mod/cache/download/github.com/gobuffalo/packr/@v/v1.21.4.zip: case-insensitive file name collision: "SHOULDERS.md" and "shoulders.md"
go get: unzip /home/leighmcculloch/go/pkg/mod/cache/download/github.com/gobuffalo/packr/@v/v1.21.4.zip: case-insensitive file name collision: "SHOULDERS.md" and "shoulders.md"

It appears that versions of projects that contain case insensitive collisions are currently not supported in Go Modules. Checking out the package into a GOPATH manually as go get used to still works, and dep successfully works with it too.

I think for portability there's some value in not allowing files that collide on case insensitive file systems, but existing packages that were published are now inaccessible.

This issue was encountered while attempting to upgrade a repo from dep to modules stellar/go#1634.

@jayconrod

This comment has been minimized.

Copy link
Contributor

commented Aug 22, 2019

This is unfortunate but it seems necessary to ensure reproducible builds, which is one of the major design goals of modules. It's a stricter system than GOPATH, and some adaptation is needed. There are a number of other constraints, too, for example import paths are restricted to ASCII alphanumeric characters with a handful of allowed punctuation characters.

The packages to be go getd without any issue.

I'm not sure what you mean here. Could you elaborate?

There's no way to avoid an issue on systems with case-insensitive file systems. We'd have to pick one of the files but not the other. If this were a .go file, code would be missing.

Even on a case-sensitive file system, if you were cross-compiling to some target platform, you'd get a different result than if you had built from a case-insensitive file system.

@leighmcculloch

This comment has been minimized.

Copy link
Contributor Author

commented Aug 22, 2019

I'm not sure what you mean here. Could you elaborate?

I expected to be able to run go get github.com/gobuffalo/packr@v1.21.4 and the command to succeed. That command and other commands that attempt to get the module fail which means it is not possible to use that version of packr in a project that uses go modules.

Is not supporting files that share the same name that differ only by case a known and accepted limitation of Go modules? I couldn't find it reported or noted anywhere in the Go Modules wiki.

@jayconrod

This comment has been minimized.

Copy link
Contributor

commented Aug 22, 2019

I expected to be able to run go get github.com/gobuffalo/packr@v1.21.4 and the command to succeed.

It can't succeed on a case-insensitive file system, and it can't produce the same result on a case-sensitive file system as on another system though.

Is not supporting files that share the same name that differ only by case a known and accepted limitation of Go modules?

It's intended behavior, but like many aspects of modules, it's not adequately documented. I think probably the best thing to do here is to document exactly how the Go command produces a module zip file from a repository. There's a lot of undocumented, nuanced behavior around here, for example around symbolic links, directories without source files, and file size limits.

@jayconrod

This comment has been minimized.

Copy link
Contributor

commented Aug 22, 2019

In the framework proposed in #33637, this would be a page in the specification section.

@leighmcculloch

This comment has been minimized.

Copy link
Contributor Author

commented Aug 22, 2019

For context, I'm attempting to use the dependency on a case sensitive filesystem, where it is been used with the dep tool successfully, and this issue only arises as I try to switch a repo to modules.

it can't produce the same result on a case-sensitive file system as on another system though.

Could you elaborate on this statement?

@jayconrod

This comment has been minimized.

Copy link
Contributor

commented Aug 22, 2019

Could you elaborate on this statement?

Suppose you check out a repo on a linux/amd64 machine with a case-sensitive file system. You build with GOOS=windows GOARCH=amd64 go build cmd/foo. Ideally, you should get the same result, bit-for-bit, as if you had run go build cmd/foo in the same repository on a windows/amd64 machine with a case-insensitive file system. That's not quite true today, but we're moving in that direction.

@jayconrod jayconrod changed the title cmd/go: go get fails for modules containing files with case insensitive collisions cmd/go: document restrictions on module zip extraction Aug 23, 2019

@leighmcculloch

This comment has been minimized.

Copy link
Contributor Author

commented Aug 23, 2019

If this isn't an issue that will be fixed, rather than document restrictions regarding zip extraction, I think it would be clearer to document that Go Modules do not support containing two files with names that differ only by case. Anyone can get into this situation. In the packr module it was accidental and the file it occurred with is even irrelevant to Go, it was a markdown file.

@leighmcculloch

This comment has been minimized.

Copy link
Contributor Author

commented Aug 27, 2019

I think this needs more attention than just documentation.

I'm trying to avoid the version of packr that has this problem, but Go keeps wanting to get v1.21.4, even when I'm telling it to exclude that version and even when it's not a compatible version with my other dependencies.

I think the tool might be traversing available versions, and getting stuck along the way to a good version, because the version I need is older than the version with the filename collision.

This could become a problem for any repo that accidentally commits two files that only differ in name by their case. Any version prior to that version becomes difficult to get to. Repos shouldn't become inaccessible just because a version was accidentally released with SHOULDERS.md and shoulders.md like @gobuffalo's packr was. In the case of packr the author even fixed the problem in v1.21.5 but the error in v1.21.4 seems to be preventing the Go tool from finding the version I need, which is v1.12.1.

The only way I've managed to get around this is to manually hack my go.mod file to be specific versions to avoid the Go tool from traversing known versions.

You can simulate this by importing these two packages side-by-side in a new module. Below is a set of commands to see what I see. I even try excluding the version.

$ go mod init test
$ cat > main.go <<-EOF
package main

import (
       "fmt"

       migrate "github.com/rubenv/sql-migrate"
       "github.com/spf13/cobra"
)

func main() {
       fmt.Println(cobra.EnableCommandSorting)
       fmt.Println(migrate.MigrationDialects)
}
EOF
$ go mod tidy
$ go get github.com/rubenv/sql-migrate@87ce952f7079
$ go get github.com/spf13/cobra@9c28e4bbd74e
verifying github.com/gobuffalo/packr@v1.21.4/go.mod: github.com/gobuffalo/packr@v1.21.4/go.mod: reading https://sum.golang.org/lookup/github.com/gobuffalo/packr@v1.21.4: 410 Gone
$ cat >> go.mod <<-EOF
exclude github.com/gobuffalo/packr v1.21.4
EOF
$ go get github.com/spf13/cobra@9c28e4bbd74e
verifying github.com/gobuffalo/packr@v1.21.4/go.mod: github.com/gobuffalo/packr@v1.21.4/go.mod: reading https://sum.golang.org/lookup/github.com/gobuffalo/packr@v1.21.4: 410 Gone
$ sed -i 's$github\.com/gobuffalo/packr v1\.30\.1$github.com/gobuffalo/packr v1.12.1$' go.mod
$ go get github.com/spf13/cobra@9c28e4bbd74e

I see that when @rsc first added the limitation on case folding (5c622a5) he commented that the goal was to restrict what was supported up front then maybe loosen requirements.

I agree portability is important, but probably not at the sacrifice of the Go tool being resilient to simple mistakes in discarded versions of our modules.


Side note, this problem is great exasperated because the dependencies of the tests in my dependencies are included in the dependency tree, even though I have no interest in running my dependencies tests. packr is only required by sql-migrates tests, and packr has a dependency on cobra. I believe there are other issues where this has been mentioned: #26955 #32380.

@Momo733

This comment has been minimized.

Copy link

commented Sep 16, 2019

I built a LAN private code repository proxy to speed up the construction of docker images, because I usually need to pull a lot of packages when I build. I use Goproxy for storage. However, when using Go1.13, it was found that there would be conflicts between the zip packages retrieved, not only case-insensitive, but also between the zip packages containing . files or folders.

zip for git.apache.org/thrift.git@v0.0.0-20180902110319-2566ecd5d999 has unexpected file git.apache.org/thrift.git@v0.12.0/.clang-format
zip for github.com/Azure/go-autorest@v11.2.7+incompatible has unexpected file github.com/Azure/go-autorest@v13.0.1+incompatible/.github/PULL_REQUEST_TEMPLATE.md
zip for github.com/DataDog/datadog-go@v0.0.0-20180822151419-281ae9f2d895 has unexpected file github.com/DataDog/datadog-go@v2.2.0+incompatible/.travis.yml
zip for github.com/census-instrumentation/opencensus-proto@v0.0.2-0.20180913191712-f303ae3f8d6a has unexpected file github.com/census-instrumentation/opencensus-proto@
v0.2.1/.github/CODEOWNERS
zip for github.com/cloudflare/cfssl@v0.0.0-20190510060611-9c027c93ba9e has unexpected file github.com/cloudflare/cfssl@v0.0.0-20190911221928-1a911ca1b1d6/.dockerignore
zip for github.com/coreos/go-systemd@v0.0.0-20190321100706-95778dfbb74e has unexpected file github.com/coreos/go-systemd@v0.0.0-20190719114852-fd7a80b32e1f/.travis.yml
zip for github.com/dnaeon/go-vcr@v0.0.0-20180920040454-5637cf3d8a31 has unexpected file github.com/dnaeon/go-vcr@v1.0.1/.gitignore
zip for github.com/docker/go@v0.0.0-20160303222718-d30aec9fd63c has unexpected file github.com/docker/go@v1.5.1-1/.gitignore
zip for github.com/golang/mock@v1.2.0 has unexpected file github.com/golang/mock@v1.3.1/.gitignore
zip for github.com/google/go-cmp@v0.3.0 has unexpected file github.com/google/go-cmp@v0.3.1/.travis.yml
zip for github.com/google/go-querystring@v0.0.0-20170111101155-53e6ce116135 has unexpected file github.com/google/go-querystring@v1.0.0/.gitignore
zip for github.com/grpc-ecosystem/grpc-gateway@v1.9.0 has unexpected file github.com/grpc-ecosystem/grpc-gateway@v1.11.1/.bazelci/presubmit.yml
zip for github.com/hashicorp/go-msgpack@v0.0.0-20150518234257-fa3f63826f7c has unexpected file github.com/hashicorp/go-msgpack@v0.5.5/.circleci/config.yml
github.com/hashicorp/go-sockaddr@v0.0.0-20180320115054-6d291a969b86: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /go/pkg/
mod/cache/vcs/f91778a63b5f3d926440e4af6c5efa337003cdc2b67ab9b8bc0e5c6b6efeb6a9: exit status 128:
        fatal: unable to access 'https://github.com/hashicorp/go-sockaddr/': Could not resolve host: github.com
reading github.com/hashicorp/hcl/go.mod at revision v1.0.0: unknown revision v1.0.0
zip for github.com/pascaldekloe/goe@v0.0.0-20150719195608-f99183613f48 has unexpected file github.com/pascaldekloe/goe@v0.1.0/.travis.yml
zip for github.com/prometheus/client_model@v0.0.0-20190129233127-fd36f4220a90 has unexpected file github.com/prometheus/client_model@v0.0.0-20190812154241-14fe0d1b01d4
/.gitignore
zip for github.com/prometheus/common@v0.6.0 has unexpected file github.com/prometheus/common@v0.7.0/.gitignore
zip for github.com/samuel/go-zookeeper@v0.0.0-20180130194729-c4fab1ac1bec has unexpected file github.com/samuel/go-zookeeper@v0.0.0-20190810000440-0ceca61e4d75/.gitign
ore
zip for github.com/xordataexchange/crypt@v0.0.3-0.20170626215501-b2862e3d0a77 has unexpected file github.com/xordataexchange/crypt@v0.0.2/.gitignore
zip for golang.org/x/crypto@v0.0.0-20190605123033-f99c8df09eb5 has unexpected file golang.org/x/crypto@v0.0.0-20190911031432-227b76d455e7/.gitattributes
zip for golang.org/x/exp@v0.0.0-20190121172915-509febef88a4 has unexpected file golang.org/x/exp@v0.0.0-20190912063710-ac5d2bfcbfe0/.gitattributes
zip for golang.org/x/net@v0.0.0-20190613194153-d28f0bde5980 has unexpected file golang.org/x/net@v0.0.0-20190912160710-24e19bdeb0f2/.gitattributes
zip for golang.org/x/sync@v0.0.0-20190423024810-112230192c58 has unexpected file golang.org/x/sync@v0.0.0-20190911185100-cd5d95a43a6e/AUTHORS
zip for golang.org/x/sys@v0.0.0-20190801041406-cbf593c0f2f3 has unexpected file golang.org/x/sys@v0.0.0-20190913121621-c3b328c6e5a7/.gitattributes
zip for golang.org/x/tools@v0.0.0-20190606050223-4d9ae51c2468 has unexpected file golang.org/x/tools@v0.0.0-20190916034716-92af9d69eff2/.gitattributes
zip for honnef.co/go/tools@v0.0.0-20190106161140-3f1c8253044a has unexpected file honnef.co/go/tools@v0.0.1-2019.2.3/.circleci/config.yml
zip for k8s.io/apimachinery@v0.0.0-20180704011316-f534d624797b has unexpected file k8s.io/apimachinery@v0.0.0-20190913075813-344bcc0201c9/.github/PULL_REQUEST_TEMPLATE
.md
zip for k8s.io/client-go@v8.0.0+incompatible has unexpected file k8s.io/client-go@v11.0.0+incompatible/.github/PULL_REQUEST_TEMPLATE.md
zip for k8s.io/helm@v2.9.1+incompatible has unexpected file k8s.io/helm@v2.14.3+incompatible/.circleci/bootstrap.sh
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.