Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: "go get: unzip: malformed file path: double dot" #27299

Closed
sirkon opened this issue Aug 28, 2018 · 12 comments
Closed

cmd/go: "go get: unzip: malformed file path: double dot" #27299

sirkon opened this issue Aug 28, 2018 · 12 comments

Comments

@sirkon
Copy link

@sirkon sirkon commented Aug 28, 2018

  • go version

    go version go1.11 linux/amd64
    

    downloaded yesterday (27 august)

  • $ go env
    GOARCH="amd64"
    GOBIN=""
    GOCACHE="/home/emacs/.cache/go-build"
    GOEXE=""
    GOFLAGS=""
    GOHOSTARCH="amd64"
    GOHOSTOS="linux"
    GOOS="linux"
    GOPATH="/home/emacs/go"
    GOPROXY=""
    GORACE=""
    GOROOT="/usr/local/go"
    GOTMPDIR=""
    GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
    GCCGO="gccgo"
    CC="gcc"
    CXX="g++"
    CGO_ENABLED="1"
    GOMOD="/home/emacs/Desktop/main/go.mod"
    CGO_CFLAGS="-g -O2"
    CGO_CPPFLAGS=""
    CGO_CXXFLAGS="-g -O2"
    CGO_FFLAGS="-g -O2"
    CGO_LDFLAGS="-g -O2"
    PKG_CONFIG="pkg-config"
    GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build252846370=/tmp/go-build -gno-record-gcc-switches"
    
  • I am trying to use ANTLR4 runtime support via go modules, steps to reproduce:

    1. Choose some directory
    2. Create new Go package/module:
      mkdir newpack
      cd newpack
      go mod init main
      
    3. Try to get ANTLR4 runtime support module
      go get github.com/antlr/antlr4/runtime/Go/antlr
      
    4. Obviously, successful download was expected, got an error message instead:
      go: finding github.com/antlr/antlr4/runtime/Go/antlr latest
      go: finding github.com/antlr/antlr4/runtime/Go latest
      go: finding github.com/antlr/antlr4/runtime latest
      go: finding github.com/antlr/antlr4 latest
      go: extracting github.com/antlr/antlr4 v0.0.0-20180728001836-7d0787e29ca8
      -> unzip /home/emacs/go/pkg/mod/cache/download/github.com/antlr/antlr4/@v/v0.0.0-20180728001836-7d0787e29ca8.zip: malformed file path "runtime/Cpp/demo/Windows/antlr4-cpp-demo/antlr4-cpp-demo-vs2015.vcxproj..filters": double dot
      go get github.com/antlr/antlr4/runtime/Go/antlr: unzip /home/emacs/go/pkg/mod/cache/download/github.com/antlr/antlr4/@v/v0.0.0-20180728001836-7d0787e29ca8.zip: malformed file path "runtime/Cpp/demo/Windows/antlr4-cpp-demo/antlr4-cpp-demo-vs2015.vcxproj..filters": double dot
      

    I guess double dots in some unrelated file is hardly a reason to diagnose a error

@sirkon sirkon changed the title go modules related `go get` issue [go modules] go get error Aug 29, 2018
@FiloSottile FiloSottile changed the title [go modules] go get error cmd/go: "go get: unzip: malformed file path: double dot" Aug 30, 2018
@FiloSottile FiloSottile added this to the Go1.12 milestone Aug 30, 2018
@sjbodzo

This comment has been minimized.

Copy link

@sjbodzo sjbodzo commented Sep 5, 2018

I am also facing this issue. Since DynamoDB uses ANTLR to parse expression grammar, this effectively means we cannot use the aws-sdk in Golang with go modules until there is a fix.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Oct 24, 2018

Consolidating into #28001.

@bcmills bcmills closed this Oct 24, 2018
@bcmills bcmills reopened this Dec 6, 2018
klingtnet added a commit to klingtnet/klingt.net that referenced this issue Dec 15, 2018
- installation instruction of go-bindata has changed, see
golang/go#27299 for details
- using a item-loop for installing multiple apt packages is deprecated,
instead an array of package names should be used
- remove filebrowser plugin from caddy since it's broken: filebrowser/filebrowser#554
- hash user password
@odeke-em

This comment has been minimized.

Copy link
Member

@odeke-em odeke-em commented Jan 20, 2019

Hey @bcmills @rsc, I realize that perhaps we plainly reject all double-dot-containing paths because we are trying to avoid trafficking/arbitrary/malicious filesystem access e.g. "foo/../secrets" where ".." contains the keys to the kingdom.
However, this seems broadly sweeping and I think we can make an exception to this rule like other filesystems to, by checking that the base of the path doesn't just consist of
dots(after checking if the entire path has "..") so basically something like this

        if strings.Contains(path, "..") {
                // Rejecting paths with ".." in them is a security consideration meant to
                // prevent arbitrary filesystem accesses. However, the exception to
                // this rule is that if the path's base consists of more
                // than just "."+ which would disambiguate between ".", ".." and valid
                // paths, then accept it.
                base := filepath.Base(path)
                if strings.Count(base, ".") == len(base) {
                        // Consists entirely of "."*
                        return fmt.Errorf("double dot")
                }
        }

which would match and allow say "foo..ps..mp4" as well as the bug report from the ANTLR parser generator.

I realize that we are almost releasing Go1.12 and perhaps could punt this to Go1.13 for safety but if this issue is deemed high priority perhaps let's discuss this -- also note that the ANTLR project replace the double dots that triggered this bug report, in August 2018(right after this issue was filed) as per antlr/antlr4#2341 but there might be other projects like this in the wild.

I'll also kindly page @mikesamuel for an evaluation of the security considerations of how I've proposed accepting ".." in the base.

@DanielJonesEB

This comment has been minimized.

Copy link

@DanielJonesEB DanielJonesEB commented Mar 14, 2019

Is there a workaround for this issue? Hitting it trying to depend on https://github.com/SpectoLabs/hoverfly

go: extracting github.com/SpectoLabs/hoverfly v0.17.7
-> unzip /Users/deejay/go/pkg/mod/cache/download/github.com/!specto!labs/hoverfly/@v/v0.17.7.zip: malformed file path "core/handlers/v2/hoverfly_upstream_proxy_handler..go": double dot
@sirkon

This comment has been minimized.

Copy link
Author

@sirkon sirkon commented Mar 14, 2019

@DanielJonesEB good workaround is to create an issue in the project

@DanielJonesEB

This comment has been minimized.

Copy link

@DanielJonesEB DanielJonesEB commented Mar 15, 2019

good workaround is to create an issue in the project

Sadly I'm depending on an old version of the library, so we can't go back in time to fix it.

@keitwb

This comment has been minimized.

Copy link

@keitwb keitwb commented Mar 17, 2019

Found this same problem in the latest version of the Telegraf repo. The use of double dots appears to be perfectly legitimate since they are testing that the config loader doesn't descend into directories that start with double dots. I guess the test could be changed to dynamically create the double dot directory to avoid it being committed to the repo, but would be nice not to have to and still work with modules.

@zouyee

This comment has been minimized.

Copy link

@zouyee zouyee commented Mar 28, 2019

go get -u golang.org/x/tools/... found the same issue.

stefan-improbable added a commit to stefan-improbable/go-flagz that referenced this issue Apr 25, 2019
- create go.sum
- do not use .. in filenames in this repo due to golang/go#27299

Testing strategy: I almost successfully ran `test_all.sh` and found a
data race, but I don't think it's related to this PR:

```
WARNING: DATA RACE
Read at 0x00c00016cac0 by goroutine 29:
  github.com/spf13/pflag.(*FlagSet).VisitAll()
      /home/stefan/usr/go/main/pkg/mod/github.com/spf13/pflag@v1.0.3/flag.go:277 +0x14a
  github.com/mwitkow/go-flagz.ChecksumFlagSet()
      /home/stefan/Projects/go-flagz/checksum.go:15 +0xc8
  github.com/mwitkow/go-flagz/monitoring.(*flagSetCollector).Collect()
      /home/stefan/Projects/go-flagz/monitoring/collector.go:57 +0x2a2
  github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1()
      /home/stefan/usr/go/main/pkg/mod/github.com/prometheus/client_golang@v0.9.2/prometheus/registry.go:434 +0x1eb

Previous write at 0x00c00016cac0 by goroutine 28:
  github.com/spf13/pflag.sortFlags()
      /home/stefan/usr/go/main/pkg/mod/github.com/spf13/pflag@v1.0.3/flag.go:204 +0x2f2
  github.com/spf13/pflag.(*FlagSet).VisitAll()
      /home/stefan/usr/go/main/pkg/mod/github.com/spf13/pflag@v1.0.3/flag.go:270 +0x1b0
  github.com/mwitkow/go-flagz.ChecksumFlagSet()
      /home/stefan/Projects/go-flagz/checksum.go:15 +0xc8
  github.com/mwitkow/go-flagz/monitoring.(*flagSetCollector).Collect()
      /home/stefan/Projects/go-flagz/monitoring/collector.go:55 +0xe9
  github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1()
      /home/stefan/usr/go/main/pkg/mod/github.com/prometheus/client_golang@v0.9.2/prometheus/registry.go:434 +0x1eb
```
umk added a commit to umk/go-dymessage that referenced this issue May 21, 2019
@andybons andybons modified the milestones: Go1.13, Go1.14 Jul 8, 2019
DavidGamba added a commit to DavidGamba/go-getoptions that referenced this issue Jul 28, 2019
@DavidGamba

This comment has been minimized.

Copy link

@DavidGamba DavidGamba commented Jul 28, 2019

Used the workaround provided here: #26672 (comment) to bypass the error, basically add a go.mod file in my test file tree so it is considered as a separate module and it doesn't get unzipped with the main one.

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Sep 27, 2019

Change https://golang.org/cl/197720 mentions this issue: cmd/go: do not reject internal double-dots in path elements

@gopherbot gopherbot closed this in db87c9f Sep 27, 2019
@liesauer

This comment has been minimized.

Copy link

@liesauer liesauer commented Oct 10, 2019

i compiled the latest master source and still get this error, am i missing something?

go version devel +3322f3e0ce Wed Oct 9 23:03:55 2019 +0000 darwin/amd64
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/imac/Library/Caches/go-build"
GOENV="/Users/imac/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/imac/Projects/go/bin"
GOPRIVATE=""
GOPROXY="https://goproxy.cn"
GOROOT="/Users/imac/Projects/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/imac/Projects/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/imac/Projects/go/src/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/xb/5ys3kh657kdd45gb9kmlqjq80000gn/T/go-build166768544=/tmp/go-build -gno-record-gcc-switches -fno-common"
@liesauer

This comment has been minimized.

Copy link

@liesauer liesauer commented Oct 10, 2019

alright! vscode use the absolute path /usr/local/bin/go which is still a soft link of /usr/local/Cellar/go/1.12.7/bin/go, relink to /Users/imac/Projects/go/bin/go will get fix

crufter added a commit to improbable-eng/go-flagz that referenced this issue Oct 16, 2019
- create go.sum
- do not use .. in filenames in this repo due to golang/go#27299

Testing strategy: I almost successfully ran `test_all.sh` and found a
data race, but I don't think it's related to this PR:

```
WARNING: DATA RACE
Read at 0x00c00016cac0 by goroutine 29:
  github.com/spf13/pflag.(*FlagSet).VisitAll()
      /home/stefan/usr/go/main/pkg/mod/github.com/spf13/pflag@v1.0.3/flag.go:277 +0x14a
  github.com/mwitkow/go-flagz.ChecksumFlagSet()
      /home/stefan/Projects/go-flagz/checksum.go:15 +0xc8
  github.com/mwitkow/go-flagz/monitoring.(*flagSetCollector).Collect()
      /home/stefan/Projects/go-flagz/monitoring/collector.go:57 +0x2a2
  github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1()
      /home/stefan/usr/go/main/pkg/mod/github.com/prometheus/client_golang@v0.9.2/prometheus/registry.go:434 +0x1eb

Previous write at 0x00c00016cac0 by goroutine 28:
  github.com/spf13/pflag.sortFlags()
      /home/stefan/usr/go/main/pkg/mod/github.com/spf13/pflag@v1.0.3/flag.go:204 +0x2f2
  github.com/spf13/pflag.(*FlagSet).VisitAll()
      /home/stefan/usr/go/main/pkg/mod/github.com/spf13/pflag@v1.0.3/flag.go:270 +0x1b0
  github.com/mwitkow/go-flagz.ChecksumFlagSet()
      /home/stefan/Projects/go-flagz/checksum.go:15 +0xc8
  github.com/mwitkow/go-flagz/monitoring.(*flagSetCollector).Collect()
      /home/stefan/Projects/go-flagz/monitoring/collector.go:55 +0xe9
  github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1()
      /home/stefan/usr/go/main/pkg/mod/github.com/prometheus/client_golang@v0.9.2/prometheus/registry.go:434 +0x1eb
```
matzf added a commit to matzf/scion that referenced this issue Nov 7, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.