New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: relative imports do not work with vendoring #15478

Open
titpetric opened this Issue Apr 28, 2016 · 11 comments

Comments

Projects
None yet
4 participants
@titpetric

titpetric commented Apr 28, 2016

  1. What version of Go are you using (go version)?

go version go1.6.1 linux/amd64

  1. What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT="1"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

Linux serenity 4.3.0-1-amd64 #1 SMP Debian 4.3.5-1 (2016-02-06) x86_64 GNU/Linux
Running golang with Docker image, official golang build, latest image from docker hub.
Full reproduction script available with ./run

Source files: https://cdn.si/assets/go-gh-issue-vendoring.tgz

  1. What did you do?

I created a small program, using a local subpackage.
main.go (using a subpackage + vendor) = experienced problem.
main2.go (using vendored import from main) = works as expected.

Subpackage is imported with: import "./apiservice". Import of vendored package from this subpackage results in a run/build error.

Also included is a test-runner using the official golang docker image. It's the same environment, that I'm running the example in. The folder vendor was created with gvt fetch github.com/namsral/flag, full source provided.

  1. What did you expect to see?

Expected output with main.go:

# go run main2.go
Hello world! Network port: 8080
  1. What did you see instead?
# go run main.go
apiservice/apiservice.go:4:8: cannot find package "_/go/src/app/vendor/github.com/namsral/flag" in any of:
        /usr/local/go/src/_/go/src/app/vendor/github.com/namsral/flag (from $GOROOT)
        /go/src/_/go/src/app/vendor/github.com/namsral/flag (from $GOPATH)

Problem:

Import path for local subpackage is not honored. The dot gets converted to underscore, and as a consequence, it's looking for the vendor directory in the wrong path:

Expected path:

        /usr/local/go/src/app/vendor/github.com/namsral/flag (from $GOROOT)
@davecheney

This comment has been minimized.

Contributor

davecheney commented Apr 28, 2016

Your need to put your source in GOPATH. From the example you posted (sorry
I haven't run your script) you have not done this.

On Thu, 28 Apr 2016, 19:33 Tit Petric, notifications@github.com wrote:

  1. What version of Go are you using (go version)?

go version go1.6.1 linux/amd64

  1. What operating system and processor architecture are you using (go
    env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT="1"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

Linux serenity 4.3.0-1-amd64 #1 #1
SMP Debian 4.3.5-1 (2016-02-06) x86_64 GNU/Linux
Running golang with Docker image, official golang build, latest image from
docker hub.
Full reproduction script available with ./run

Source files: https://cdn.si/assets/go-gh-issue-vendoring.tgz

  1. What did you do?

I created a small program, using a local subpackage.
main.go (using a subpackage + vendor) = experienced problem.
main2.go (using vendored import from main) = works as expected.

Subpackage is imported with: import "./apiservice". Import of vendored
package from this subpackage results in a run/build error.

Also included is a test-runner using the official golang docker image.
It's the same environment, that I'm running the example in. The folder
vendor was created with gvt fetch github.com/namsral/flag, full source
provided.

  1. What did you expect to see?

Expected output with main.go:

go run main2.go

Hello world! Network port: 8080

  1. What did you see instead?

go run main.go

apiservice/apiservice.go:4:8: cannot find package "/go/src/app/vendor/github.com/namsral/flag" in any of:
/usr/local/go/src/
/go/src/app/vendor/github.com/namsral/flag (from $GOROOT)
/go/src/_/go/src/app/vendor/github.com/namsral/flag (from $GOPATH)

Problem:

Import path for local subpackage is not honored. The dot gets converted to
underscore, and as a consequence, it's looking for the vendor directory in
the wrong path:

Expected path:

    /usr/local/go/src/app/vendor/github.com/namsral/flag (from $GOROOT)


You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#15478

@titpetric

This comment has been minimized.

titpetric commented Apr 28, 2016

@davecheney I've been putting it in GOROOT, I updated now to use GOPATH, same issue persists.

#!/bin/bash
printf "Expected:\n\n"
docker run --rm -it -v `pwd`:/go/src/app -w /go/src/app golang go run main2.go
printf "\nActual:\n\n"
docker run --rm -it -v `pwd`:/go/src/app -w /go/src/app golang go run main.go
# ./run_gopath
Expected:

Hello world! Network port: 8080

Actual:

apiservice/apiservice.go:4:8: cannot find package "_/go/src/app/vendor/github.com/namsral/flag" in any of:
        /usr/local/go/src/_/go/src/app/vendor/github.com/namsral/flag (from $GOROOT)
        /go/src/_/go/src/app/vendor/github.com/namsral/flag (from $GOPATH)

So, sorry, no. I also tried putting everything in /go instead of /go/src/app but that breaks both examples.

@davecheney

This comment has been minimized.

Contributor

davecheney commented Apr 28, 2016

I belive GOPATH is not set in your enviroment correctly. I believe this
because of the leading _/ in the error report. This is the path that the go
tool uses when source is outside GOPATH or GOPATH is not set correctly.
If GOPATH is not set correctly, vendoring will not be used.

Is it possible to isolate your issue by removing docker?

On Thu, Apr 28, 2016 at 8:10 PM, Tit Petric notifications@github.com
wrote:

@davecheney https://github.com/davecheney I've been putting it in
GOROOT, I updated now to use GOPATH, same issue persists.

#!/bin/bash
printf "Expected:\n\n"
docker run --rm -it -v pwd:/go/src/app -w /go/src/app golang go run main2.go
printf "\nActual:\n\n"
docker run --rm -it -v pwd:/go/src/app -w /go/src/app golang go run main.go

./run_gopath

Expected:

Hello world! Network port: 8080

Actual:

apiservice/apiservice.go:4:8: cannot find package "/go/src/app/vendor/github.com/namsral/flag" in any of:
/usr/local/go/src/
/go/src/app/vendor/github.com/namsral/flag (from $GOROOT)
/go/src/_/go/src/app/vendor/github.com/namsral/flag (from $GOPATH)

So, sorry, no. I also tried putting everything in /go instead of
/go/src/app but that breaks both examples.


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#15478 (comment)

@titpetric

This comment has been minimized.

titpetric commented Apr 28, 2016

I don't have an environment outside of docker where I can run the tests. If I get one, I'll post the results, and if you have one, please just stick the sources from the tgz into $GOPATH/src/app and run go run main.go to see if the error occurs with you.

But, my sanity check here tells me that you shouldn't be so suspicious, go env reports /go (same as $GOPATH in bash), sources don't reference some /other or ../other locations. I might be quite new at this, but I'm pretty sure that nowhere in about 10-20 lines of code I'm going outside of GOPATH, and I'm not touching it in any way. The obvious question is "what's the correct way to set up GOPATH?" if the official docker image doesn't do this. Source Dockerfile is here, if you can spot the issue: https://github.com/docker-library/golang/blob/master/1.6/Dockerfile

I'll attach my results if I can get my hands on on a local install of go somewhere.

@davecheney

This comment has been minimized.

Contributor

davecheney commented Apr 28, 2016

You shouldn't need docker to compile your tests -- they may not pass, but
you'll at least confirm that you can compile your code.

On Thu, Apr 28, 2016 at 8:28 PM, Tit Petric notifications@github.com
wrote:

I don't have an environment outside of docker where I can run the tests.
If I get one, I'll post the results, and if you have one, please just stick
the sources from the tgz into $GOPATH/src/app and run go run main.go to
see if the error occurs with you.

But, my sanity check here tells me that you shouldn't be so suspicious, go
env reports /go (same as $GOPATH in bash), sources don't reference some
/other or ../other locations. I might be quite new at this, but I'm
pretty sure that nowhere in about 10-20 lines of code I'm going outside
of GOPATH, and I'm not touching it in any way. The obvious question is
"what's the correct way to set up GOPATH?" if the official docker image
doesn't do this. Source Dockerfile is here, if you can spot the issue:
https://github.com/docker-library/golang/blob/master/1.6/Dockerfile

I'll attach my results if I can get my hands on on a local install of go
somewhere.


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#15478 (comment)

@davecheney

This comment has been minimized.

Contributor

davecheney commented Apr 28, 2016

I just checked out your sample. Sorry I should have done that before.

The root cause is you are using go run, which does not support vendoring.

I recommend you lay out your code in the prescribed package format as
documented here https://golang.org/doc/code.html

Then compile and run your program.

On Thu, Apr 28, 2016 at 8:29 PM, Dave Cheney dave@cheney.net wrote:

You shouldn't need docker to compile your tests -- they may not pass, but
you'll at least confirm that you can compile your code.

On Thu, Apr 28, 2016 at 8:28 PM, Tit Petric notifications@github.com
wrote:

I don't have an environment outside of docker where I can run the tests.
If I get one, I'll post the results, and if you have one, please just stick
the sources from the tgz into $GOPATH/src/app and run go run main.go to
see if the error occurs with you.

But, my sanity check here tells me that you shouldn't be so suspicious, go
env reports /go (same as $GOPATH in bash), sources don't reference some
/other or ../other locations. I might be quite new at this, but I'm
pretty sure that nowhere in about 10-20 lines of code I'm going outside
of GOPATH, and I'm not touching it in any way. The obvious question is
"what's the correct way to set up GOPATH?" if the official docker image
doesn't do this. Source Dockerfile is here, if you can spot the issue:
https://github.com/docker-library/golang/blob/master/1.6/Dockerfile

I'll attach my results if I can get my hands on on a local install of go
somewhere.


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#15478 (comment)

@titpetric

This comment has been minimized.

titpetric commented Apr 28, 2016

I'm having the same result with go build, but I'll check the layout extensively. I followed the recommended layout from glide - in the main README, which should be this:

$GOPATH(/go)/src/your-project
  |__ main.go (Where your main code lives)
  |__ subpackage/
  |     |__ subpackage.go (You can create subpackages)
  |     \__ ...
  |__ ... (Other files or subpackages)
  \__ vendor/
        |__ github.com/namsral/flag
        |__ github.com/gorilla/mux
        \__ ...

So if I understand you, instead of using "subpackage" I should create src/subpackage and use import "subpackage" as a replacement? I'm just about to go AFK into a meeting, and I'll try this asap. But it really seems that I shouldn't. Importing subpackages which reside in the same level is done with "./package", which is a documented thing somewhere. It actually works... at least some packages use a slightly different approach to structure the sources in the same way, with the double-slash import option:

package main // import "github.com/janeczku/go-dnsmasq"
import "github.com/janeczku/go-dnsmasq/hostsfile"

Is there a way to achieve something similar for local development without a recognized VCS location? I believed until now that: import "./package" was it.

@davecheney

This comment has been minimized.

Contributor

davecheney commented Apr 28, 2016

Is there a way to achieve something similar for local development without
a recognized VCS location? I believed until now that: import "./package"
was it.

Yes, packages are just directories on disk. They follow the url based
convention to avoid namespace collision and to enable go get. You should
layout your code using the url based pattern, but there is no requirement
(unless you want someone else to go get your code) to push it to a remove
vcs location

On Thu, Apr 28, 2016 at 8:45 PM, Tit Petric notifications@github.com
wrote:

I'm having the same result with go build, but I'll check the layout
extensively. I followed the recommended layout from glide
https://github.com/Masterminds/glide - in the main README, which should
be this:

$GOPATH(/go)/src/your-project
|__ main.go (Where your main code lives)
|__ subpackage/
| |__ subpackage.go (You can create subpackages)
| __ ...
|__ ... (Other files or subpackages)
__ vendor/
|__ github.com/namsral/flag
|__ github.com/gorilla/mux
__ ...

So if I understand you, instead of using "subpackage" I should create
src/subpackage and use import "subpackage" as a replacement? I'm just
about to go AFK into a meeting, and I'll try this asap. But it really seems
that I shouldn't. Importing subpackages which reside in the same level is
done with "./package", which is a documented thing somewhere. It actually
works... at least some packages use a slightly different approach to
structure the sources in the same way, with the double-slash import option:

package main // import "github.com/janeczku/go-dnsmasq"
import "github.com/janeczku/go-dnsmasq/hostsfile"

Is there a way to achieve something similar for local development without
a recognized VCS location? I believed until now that: import "./package"
was it.


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#15478 (comment)

@kostya-sh

This comment has been minimized.

Contributor

kostya-sh commented Apr 28, 2016

I think that currently local (relative) imports do not work with vendoring. I am not sure if it is even possible to make these features work together. Local imports do not work in some other cases as well (see for example http://stackoverflow.com/questions/10687627/relative-import-from-parent-directory).

@titpetric

This comment has been minimized.

titpetric commented Apr 28, 2016

@davecheney It is just slightly awkward if you're using go get to mix remote packages from github, etc, with whatever you have locally. main.go and package/package.go with import "./package" would seem like a valid usage pattern.

@kostya-sh That's unfortunate. I guess I'd mark this issue as a feature request then, I would very much like local imports to work with vendoring.

@titpetric

This comment has been minimized.

titpetric commented Apr 28, 2016

If anybody is coming here for a fix, this worked for me:

  1. My app name is 'app' (/go/src/app),
  2. instead of importing ./submodule, import app/submodule
  3. Vendoring starts to work, everything works with go run.

I will not close this issue, just because I expect that this should be fixed in a future version. I hope someone agrees and tags/assigns the issue properly. Relative imports should work with vendoring, but I understand the motivations for FQDN imports. Considering that i'm using them from the main package, which is a reasonable "root" of the project, and if I'll ever reuse it from another package i would use the FQDN import from a VCS/other.

In the current state I would have to rewrite './' to $(basename $(dirname $0)) before doing go build/go run, which would require some external tooling like gb. Don't want external tools for something that simple.

@ianlancetaylor ianlancetaylor changed the title from Subpackage vendoring issue to cmd/go: relative imports do not work with vendoring Apr 28, 2016

@ianlancetaylor ianlancetaylor added this to the Unplanned milestone Apr 28, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment