New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: go/build: vendor directories are ignored when outside of $GOPATH #14566

Closed
aspeteRakete opened this Issue Feb 29, 2016 · 37 comments

Comments

Projects
None yet
@aspeteRakete
Copy link

aspeteRakete commented Feb 29, 2016

  1. What version of Go are you using (go version)?
    1.6
  2. What operating system and processor architecture are you using (go env)?
    ubuntu 16.04 x86_64
  3. What did you do?
    I want to build a go project using gitlab-ci. The project gets clone on the build machine in /build/project, which is not inside the go path. The build fails due to import errors.
    I reproduced this manually: moving a project outside the GOPATH leads to ignoring the vendor folder.
  4. What did you expect to see?
    go searching for imports in the vendor directory
  5. What did you see instead?
    The vendor directory is ignored and go only searches in GOROOT and GOPATH
@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

ianlancetaylor commented Feb 29, 2016

The go tool is based on GOPATH. If you don't want to use GOPATH, it's hard to use the go tool.

Note that GOPATH can have multiple entries; see go help gopath.

I'm going to close this; we're not going to be adding support for things like the vendor directory outside of the existing GOPATH support.

@aspeteRakete

This comment has been minimized.

Copy link

aspeteRakete commented Mar 1, 2016

Sorry, but this doesn't make sense to me. The vendoring feature is supposed to solve the problem of dependency management. I got all my dependencies except the go standard library as submodules of my project. Why am I forced to put this to a specific location in my filesystems? My GOPATH is set and the go tool searches for modules in there. When a project is located outside the GOPATH/src i would expect the go tool to search the vendor folder first and then the GOPATH for modules.

@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Mar 1, 2016

The design of the "vendor" mechanism is what you should expect: https://golang.org/s/go15vendor

Any deviations from that document should be filed as a bug, but this is not one.

Let's move this to the mailing list. See https://golang.org/wiki/Questions for links.

@trans

This comment has been minimized.

Copy link

trans commented May 3, 2016

Honestly I find it odd too. I don't get why the "go way" is so phobic to freely located project directories. This has been one of the big three pains-in-my-ass when working with the language.

@mcluseau

This comment has been minimized.

Copy link

mcluseau commented Jun 25, 2016

Since the linked specification doesn't state anywhere that the GOPATH is required, making the go tool handle this case, for instance with an option to specify the package path of the current working directory, is not going to be bug. My understanding is that this issue is then a feature request, and thus doesn't deserve to be closed because it's not a bug.
In fact, the linked specification states «This proposal allows the Go community to move from builds that require custom GOPATH configuration beforehand to builds that just work», and one concerned by this feature request can interpret this failure as a deviation from this specification, since any build made where no GOPATH exists will require a custom GOPATH as a consequence.

@mcluseau

This comment has been minimized.

Copy link

mcluseau commented Jun 25, 2016

Here's what works for me in GitLab-CI (building the required custom GOPATH to allow the use of vendoring):

build:
    script:
        - mypkg=x/y/z
        - rm -fr workspace
        - mkdir -p workspace/src/$(dirname $mypkg)
        - ln -s $PWD workspace/src/$mypkg
        - GOPATH=$PWD/workspace go build $mypkg/cmd/my-cmd
@quentinmit

This comment has been minimized.

Copy link
Contributor

quentinmit commented Jul 13, 2016

The design of the "vendor" mechanism is what you should expect: https://golang.org/s/go15vendor

Any deviations from that document should be filed as a bug, but this is not one.

This issue, as specified, is describing a deviation from that document. It simply says that, when compiling code located in "d/", the go command will look for imports in "d/vendor/$path". It makes no mention of GOPATH except in an example.

I'm going to reopen this issue to look at for Go 1.8. I think fixing this bug could be a key part of solving #12488.

@quentinmit quentinmit reopened this Jul 13, 2016

@quentinmit quentinmit added this to the Go1.8 milestone Jul 13, 2016

@quentinmit quentinmit changed the title Projects vendor folder ignored when outside of $GOPATH go/build: vendor directories are ignored when outside of $GOPATH Jul 13, 2016

@rasky

This comment has been minimized.

Copy link
Member

rasky commented Aug 6, 2016

While this is useful, it doesn't solve the problem for projects that have local packages (subdirectories) in addition to vendor packages.

If you just fix this (which makes sense), you're solving the GOPATH issue only for projects made of a single package. If they have sub-package, they would still need to use a GOPATH workspace.

@metakeule

This comment has been minimized.

Copy link

metakeule commented Aug 19, 2016

@rasky You could put sub-packages inside vendor/lib/ or something similar. That may be considered misuse of vendor/, but it works and has not the problems of typical relative package paths. But it means that the vendor directory now must be committed to source control (at least the lib subfolder and no external tool may clean vendor).

@RichyHBM

This comment has been minimized.

Copy link

RichyHBM commented Aug 22, 2016

@rasky You can import packages via "./foo/bar", this seems to work fine with packages under the current directory. Not ideal but does the job

@rasky

This comment has been minimized.

Copy link
Member

rasky commented Aug 22, 2016

If that already works outside of GOPATH, then it's perfectly fine for me, and I agree it does job. Together with the modification described in this issue, it would allow development outside GOPATH (assuming a vendor tool that allows you to download/update dependencies directly within vendor/)

@RichyHBM

This comment has been minimized.

Copy link

RichyHBM commented Aug 22, 2016

Works fine for me, though I cant use it to access libraries inside the vendor folder odly enough (Gives an error saying the github.com/foo/bar should be in the GoPath)

Ofcourse If you just have packages directly in your roject folder, via submodules for example, then this should work I imagine

@leipzig leipzig referenced this issue Aug 31, 2016

Closed

Update http.go #4

@erjoalgo

This comment has been minimized.

Copy link

erjoalgo commented Sep 17, 2016

Has there been any update on this? Why is Go so obsessed with forcing GOPATH on everyone and everything? #3515

@mcluseau

This comment has been minimized.

Copy link

mcluseau commented Sep 26, 2016

#12488 is not going to focus on solving this, so this is still a request per se ;-)

@ascotan

This comment has been minimized.

Copy link

ascotan commented Sep 29, 2016

fetch:
    glide install
    @for dep in $(shell cd vendor; find * -type d -maxdepth 2 -mindepth 2); do \
      echo "Linking $$dep into $$GOPATH"; \
      rm -rf $$GOPATH/src/$$dep; \
        mkdir -p $$GOPATH/src/$${dep%/*}; \
        ln -fs $(shell pwd)/vendor/$$dep $$GOPATH/src/$$dep; \
    done

build: fetch
    ln -fs $(shell pwd) ${GOPATH}/src/gitlab.com/whois/$(shell basename $(shell pwd))
    go build -v

Because I personally have multiple repos in different languages and I just prefer to not install half of them in a GOPATH. Everytime I jump back into go I'm like "Yes I love programming in go! Now create a new project, add a dependency.... Oh god."

@dragonfax

This comment has been minimized.

Copy link

dragonfax commented Nov 8, 2016

vendor is used only for storing source code (the repositories), not the generated binaries (executables and package objects). But these items are created and managed solely by the "go tools " and could IMHO be considered a cache, or rather transient.

So perhaps it wouldn't be out of the question to have the "go tools" default to using a path such as /tmp for the contents of "bin" and "pkg" when it finds that GOPATH is unset. Choosing some arbitrary directory like /tmp/ or ~/.<application> is a common practice for command line applications to stuff their generated files into.

@dragonfax

This comment has been minimized.

Copy link

dragonfax commented Nov 8, 2016

With this, you have options:

  • vendor/ but no GOPATH - Your project is fully self-contained in its repo using vendoring.
  • GOPATH but no vendor/ - Your project shares its dependencies with your other projects.
  • GOPATH and vendor/ - Dependencies you've vendored with your favorite tool are tracked in your repo, but other dependencies are shared with other projects using the GOPATH.

But GOPATH is completely optional, and left as just a "good idea".

You still get an error if you have no GOPATH and no created vendor/ directory.

@SamWhited

This comment has been minimized.

Copy link
Member

SamWhited commented Nov 8, 2016

So perhaps it wouldn't be out of the question to have the "go tools" default to using a path such as /tmp for the contents of "bin" and "pkg" when it finds that GOPATH is unset.

@dragonfax Note that in 1.8 this is already the case (more or less; see #17262); GOPATH is $HOME/go by default, and it can be overriden by setting $GOPATH.

@dragonfax

This comment has been minimized.

Copy link

dragonfax commented Nov 8, 2016

@SamWhited That doesn't really solve the problem of this issue. The tools are still going to require your source to be under GOPATH. Even if its choosing a default GOPATH for you. This is about not requiring a GOPATH at all, let alone forcing the source to be under it.

@RichyHBM

This comment has been minimized.

Copy link

RichyHBM commented Nov 8, 2016

@dragonfax Agreed, @SamWhited it would be ideal to be able to have a go project in any directory of your computer regardless of where the GOPATH is pointing at, which at the moment is not the case as the go tools only recognise the vendor folder if it lies somewhere bellow GOPATH

@SamWhited

This comment has been minimized.

Copy link
Member

SamWhited commented Nov 8, 2016

Sure, sure; I was just pointing out that a default path (where build files can be put) will already exist in 1.8, not suggesting that it somehow solves the problem. Unless the point of using /tmp was to not polute the GOPATH with things built outside the GOPATH, in which case please disregard.

@dragonfax

This comment has been minimized.

Copy link

dragonfax commented Nov 8, 2016

I didn't know about the default path addition. Thanks. With that, I guess I'm just suggesting loosening the requirement for source to be under GOPATH when its not really needed. i.e all deps under vendor/.

@bradfitz bradfitz added NeedsDecision and removed NeedsFix labels Dec 2, 2016

@bradfitz bradfitz changed the title go/build: vendor directories are ignored when outside of $GOPATH proposal: go/build: vendor directories are ignored when outside of $GOPATH Dec 2, 2016

@bradfitz bradfitz added the Proposal label Dec 2, 2016

@bradfitz bradfitz modified the milestones: Proposal, Go1.9Early Dec 2, 2016

@go101

This comment has been minimized.

Copy link

go101 commented Dec 3, 2016

I propose to split the GOPATH env into there envs: GOPATH_SRC, GOPATH_PKG and GOPATH_BIN.
This will make building a project outside of GOPATH easier.
We can just call

GOPATH_SRC=$PWD:$GOPATH_SRC go build 

in the project folder to build the project.

Splitting the GOPATH env will make writing custom build shell much simpler.

@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Dec 3, 2016

@Golang101, we already have $GOBIN. And ideally we'd get rid of the "pkg" directory entirely (#4719).

@go101

This comment has been minimized.

Copy link

go101 commented Dec 3, 2016

The headache one is the src folder.
To add a path in GOPATH, the path must contain a src sub folder.
This is not always feasible.

@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Dec 3, 2016

@Golang101, yes, we understand. That's what this proposal and #17271 are about.

@eikenb

This comment has been minimized.

Copy link

eikenb commented Dec 3, 2016

#17271 seems to be in limbo as you yourself proposed shelving it to see the impact of $HOME/go (#17262). But I don't see how #17262 addresses the problem brought up by this ticket.

@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Dec 3, 2016

@eikenb, if by "shelving it" you mean taking things slowly and seeing the impact of one change before doing three things at once, yes.

@eikenb

This comment has been minimized.

Copy link

eikenb commented Dec 4, 2016

I have no problem with being methodical and taking time to do things. But the proposed timeline would make it about 2 years before considering a fix for this issue, which seemed a bit long to me.

@rsc

This comment has been minimized.

Copy link
Contributor

rsc commented Dec 5, 2016

There are many problems with running outside GOPATH. Among others, there's no caching of built object files, so everything is much slower than it should be (go build, go test, and so on). Vendoring is pretty low on the list of problems.

It may be that GOPATH should be reconsidered but if so it should be done as a whole change, not just one piece at a time, like @bradfitz said above. The solution for now is to use GOPATH.

Duplicate of #17271 if anything.

@rsc rsc closed this Dec 5, 2016

@ascotan

This comment has been minimized.

Copy link

ascotan commented Dec 6, 2016

#17271 is a different but similar issue. That issue is that people should be able to have an auto-configured GOPATH. The resolution was proposed to auto resolve GOPATH to ./vendor but instead it would be autoresolved to $HOME/go

This issue is that I have a GOPATH configured, but I want to checkout code outside of that path and the ./vendor directory would override packages in the configured GOPATH.

@rsc

This comment has been minimized.

Copy link
Contributor

rsc commented Dec 7, 2016

OK, so not a duplicate of #17271 - updated my comment. The rest of my reply still stands.

@joeblew99

This comment has been minimized.

Copy link

joeblew99 commented Dec 8, 2016

@steelbrain

This comment has been minimized.

Copy link

steelbrain commented Jan 4, 2017

I personally think that the GoLang contributors should give developers at least some amount of freedom. Closing an issue that is requested by so many because you think it'll be slower, for people who are already aware of it and are still requesting it because of how painful the alternatives are is not the best way to go. I have projects in monorepo that use golang for some components.

All other languages work fine but working with Go is a pain because of having to link individual modules into the GOPATH. We get that it'll be slow, but we want it anyway. IMHO a language should empower it's users instead of dictating them.

In my humble opinion, vendor directories should not be ignored outside of GOPATH.

@steelbrain

This comment has been minimized.

Copy link

steelbrain commented Jan 4, 2017

FWIW, I've created a shell script for my project that I now use to compile go stuff. It temporarily links stuff into GOPATH, compiles and then unlinks. I get to keep my stuff out of GOPATH and into the repo and don't have to wait for any new features

@RichyHBM

This comment has been minimized.

Copy link

RichyHBM commented Jan 4, 2017

@steelbrain I agree!
Would you happen to have a gist of that script?

@bradfitz

This comment has been minimized.

Copy link
Member

bradfitz commented Jan 4, 2017

@steelbrain, the discussion of removing the requirement for $GOPATH is mostly happening in #17271. It is on our agenda to figure out in the next few months.

Please watch that issue but keep in mind https://golang.org/wiki/NoMeToo

I'm going to lock this issue so we can focus conversation in one place.

@golang golang locked and limited conversation to collaborators Jan 4, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.