Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dep init fails if in not in $GOPATH[...]/src/{somedir..} #148

Closed
pbennett opened this issue Jan 25, 2017 · 30 comments
Closed

dep init fails if in not in $GOPATH[...]/src/{somedir..} #148

pbennett opened this issue Jan 25, 2017 · 30 comments
Labels

Comments

@pbennett
Copy link

@pbennett pbennett commented Jan 25, 2017

If GOPATH is foo/bar/ then running dep init from /foo/bar/src
yields:
determineProjectRoot: /foo/bar/src not in any $GOPATH
yet it most certainly is in the GOPATH.

I'll leave the commentary off about why I want this, but if this is going to be disallowed for now, a better error is needed.

@Mahito

This comment has been minimized.

Copy link

@Mahito Mahito commented Jan 26, 2017

I could got same issue.

% echo $GOPATH
/Users/mahito/.go
% cd $GOPATH/src/sandbox
% dep init  
% ls
lock.json      manifest.json
% cd ..
% dep init
determineProjectRoot: /Users/mahito/.go/src not in any $GOPATH

This issue is happened every time in $GOPATH/src.
https://github.com/golang/dep/blob/master/context.go#L115 create srcprefix($GOPATH/src/) .
Next line compare path ($GOPATH/src) and srcprefix($GOPATH/src/) and get false, because len(path) is less length than len(srcprefix).

@machiel

This comment has been minimized.

Copy link

@machiel machiel commented Jan 26, 2017

Yes, I've got the same issue, for private projects we work with $GOPATH as the project git root. Trying to do a dep init in $GOPATH yields:

determineProjectRoot: $GOPATH not in any $GOPATH

Though I was told this isn't the intended way of using the tool though, so it might be out of scope.

@sdboyer

This comment has been minimized.

Copy link
Member

@sdboyer sdboyer commented Jan 26, 2017

I think this approach to things should work fine, though because it's not a design case we've had in mind, there's likely to be at least a few hinky issues related to the base project root being empty. (Yeah...I can think of a few now.) The only important note to make is that the resulting repository will be useless for anyone trying to import it...but I imagine that's not a concern for anyone taking this approach.

To be clear - this approach means that dep will ensure there is exactly one version of all dependencies in your system, and all of them will be managed as a group. In general, if you leave all your third-party deps under vendor/, and keep all the stuff you actually work on (your monorepo-ish) directly in GOPATH not under vendor/, then this oughtta be OK.

At minimum, we need a better error here. Ideally, we'd support this as a first-class use case.

@tro3

This comment has been minimized.

Copy link
Contributor

@tro3 tro3 commented Jan 31, 2017

Yeah, this looks related to #201. I'll add an integration test for the cases described above then clean up the code.

@tro3

This comment has been minimized.

Copy link
Contributor

@tro3 tro3 commented Jan 31, 2017

Actually, this is not the same issue as #201 after all. This is more of a defintion question. The dependency import path is calculated as everything past $GOPATH/src, becuase that is where Go will look for the imports anyway. So if you start a project in $GOPATH/src, it can't figure out the import path. (I'm a little curious what $GOPATH/src/src would do.) I am too new to the subtleties of the Go import model to be able to make a call here, but it does sound to me like starting a repo directly in $GOPATH goes against the opinion of the tool, which assumes a $GOPATH/src, $GOPATH/pkg, $GOPATH/bin?

@machiel

This comment has been minimized.

Copy link

@machiel machiel commented Jan 31, 2017

Just to clarify, I don't put my sourcecode directly in GOPATH.

My tree for most of my projects look somewhat like this: (Or vendor directly in GOPATH, if I use gb)

Hope this clear some things up :)

$GOPATH
├── bin
├── pkg
├── resources
│   ├── config
│   ├── templates
│   └── uploads
└── src
    ├── web
    ├── mail
    ├── notifier
    ├── server
    └── vendor
@sdboyer

This comment has been minimized.

Copy link
Member

@sdboyer sdboyer commented Jan 31, 2017

@tro3 yeah, this issue describes an alternate use pattern to what's typical; @machiel's description is a good description of one variant of it. It can work within the general model of dep, sliding through on a technicality (actually, more like a door-left-carefully-ajar).

However, fixing the handling to make the "GOPATH/src as root" case work is more involved, more or less because of what you note - there is no import path to figure out. Even once we solve it in dep, gps is going to need a bit of special handling for empty root import paths to support this case.

...and, actually, some other things, too. Ugh. I need to open an issue on gps for it.

@tro3

This comment has been minimized.

Copy link
Contributor

@tro3 tro3 commented Jan 31, 2017

Oh, I see. Actually, I kind of like that use case, though presumably $GOPATH is then set per-project. (Still a newbie, here.)

So if it works as-is for $GOPATH (thru the door left ajar :-)) and as intended for $GOPATH/src/x, maybe just a better error message for dep init run in $GOPATH/src is needed? In any case, #201 is a different beast. If you want an explicit error called for this case, let me know and I'll add.

campoy pushed a commit to campoy/dep that referenced this issue Mar 10, 2017
campoy pushed a commit to campoy/dep that referenced this issue Mar 10, 2017
campoy pushed a commit to campoy/dep that referenced this issue Mar 10, 2017
campoy pushed a commit to campoy/dep that referenced this issue Mar 10, 2017
kris-nova pushed a commit to kris-nova/dep that referenced this issue Apr 21, 2017
@typeless

This comment has been minimized.

Copy link

@typeless typeless commented Sep 1, 2017

Looking forward to the progress on this. We have a proprietary project that has many small utilities built into a busybox-like executable and has some vendored packages from the stdlib and third-parties for those utilities. Currently the best solution is GB but it seems abandoned. We also have to integrate the project into some BSPs, including AOSP and other buildroot-based ones, which makes the process a little messy. Hopefully a canonical tool like dep will ease our job soon.

@sekrett

This comment has been minimized.

Copy link

@sekrett sekrett commented Oct 19, 2017

@sdboyer so your design is to force everyone to put their project under src folder? I don't understand why I should group my repositories by language, instead I should be able to group by projects. For example I have a project, which contains one service written in Ruby and another in Golang. In general I would like to decide myself how to name and organize my folders.

@andrewrk

This comment has been minimized.

Copy link

@andrewrk andrewrk commented Oct 26, 2017

why does dep depend on GOPATH? isn't the whole point to ignore GOPATH and use vendor as the place for dependencies, so projects can be independent?

@sdboyer

This comment has been minimized.

Copy link
Member

@sdboyer sdboyer commented Oct 27, 2017

@andrewrk sadly, dep can't liberate us from GOPATH on its own, as the compiler still requires it. and, being that dep's goal is to have as easy a transition into the toolchain as possible, we're focusing minimally on those areas that complement/extend the existing go toolchain, rather than supplanting existing systems.

given that we know we have to accept GOPATH's existence for now, we do rely on it for figuring out the root import path of the current project. this is important because we use it to determine which import paths are "internal" vs "external" when inspecting the code from the current project. A fair bit more on those mechanics here: #313 (comment)

when we're ready to move away from GOPATH (it might look something like this), we'll probably add a field to Gopkg.toml for declaring the project root so that we no longer need infer it from filesystem position + GOPATH variable.

@andrewrk

This comment has been minimized.

Copy link

@andrewrk andrewrk commented Oct 27, 2017

Ah. Thanks for this explanation. I was getting ridiculed in #go-nuts for the same question but your answer is clear and helpful.

@sekrett

This comment has been minimized.

Copy link

@sekrett sekrett commented Nov 1, 2017

Another bad situation is when I run go dep from Gitlab CI. I have to create src directory and symlink the project to it. It works but the hack is ugly. Golang should not dictate path conventions.

@DarthHater

This comment has been minimized.

Copy link

@DarthHater DarthHater commented Dec 11, 2017

Thanks for the explanation @sdboyer , I am trying to run a dep project in Jenkins and this was driving me insane. Leaving a comment here so I can know when/if this becomes possible.

@zhexuany

This comment has been minimized.

Copy link

@zhexuany zhexuany commented Dec 14, 2017

An interesting observation. When I try to run dep init at a symlink but it failed and give me error message root project import: xxxx is not within any GOPATH/src. The interesting part is that current directory is actually a symlink to a project under GOPATH. Any solution for this one?

@tombh

This comment has been minimized.

Copy link

@tombh tombh commented Jan 6, 2018

I'm really confused by all this. So just to be explicit: if you're developing a project and you want to use dep then that project needs to live on your computer under $GOPATH/src? If that's the case, where's the documentation for that.

So my current fix is to symlink my project into $GOPATH/src and have a separate CLI open at $GOPATH/src/myproject to run the dep commands in.

@sdboyer

This comment has been minimized.

Copy link
Member

@sdboyer sdboyer commented Jan 6, 2018

@tombh it's not in the docs, as the current docs are paltry, and GOPATH is one of those foundational realities that's just implied.

i realize that's a terrible answer in general, though. so, ive made it explicit in the docs overhaul - #1499.

@frostyplanet

This comment has been minimized.

Copy link

@frostyplanet frostyplanet commented Jan 20, 2018

We hate to put our project in GOPATH, as the project itself might contain submodule for other project, or contain code written by other language.

@sdboyer

This comment has been minimized.

Copy link
Member

@sdboyer sdboyer commented Jan 20, 2018

@frostyplanet we know the reasons why it's important to do, and agree with them. it doesn't change the reality that it's not a choice we can make in dep right now.

@frostyplanet

This comment has been minimized.

Copy link

@frostyplanet frostyplanet commented Jan 21, 2018

@sdboyer
There is some way to build a project which is not in gopath. We are using these practices for local develop and it's easy to pack up everything inside working dir to build RPM packages on dedicated build host.
All we want is a tool that download dependences into "vendor". "vendor" may or may not be managed by version control.

  1. For simple code which does not have complex structure, *.go is right under working dir, like any other golang projects,
    create a build directory (which was set as GOPATH) , sync the code into it and build,
    Using this Makefile:
    compile: internal_dep FORCE
    @mkdir -p build/src/
    @rsync -a --delete *.go vendor build/src/mypackage/
    env GOPATH="$$(pwd)/build" go build -o build/$(PROG) $(PROJECT)
    @echo "Build done"

  2. Using this directory tree
    repo
    | -- src (code written in go)
    | -- | -- mypackage1
    | -- | -- mypackage2 (might be git submodule following practice 1)
    | -- scripts
    | -- xxx ( code not written in go )
    | -- thirdparty
    | -- | -- src
    | -- | -- | -- thirdparty package

Set GOPATH with
export GOPATH="$(pwd):$(pwd)/thirdparty"
and then using "go get xxx", will download them into "thirdparty/src".

"go get" works fine with public code. For private code we have written some script to download them into "thirdparty" and deal with the branch/tag/commit thing, but still our script is not good at resolving dependency.

@andrewrk

This comment has been minimized.

Copy link

@andrewrk andrewrk commented Feb 21, 2018

@sdboyer

sadly, dep can't liberate us from GOPATH on its own, as the compiler still requires it. and, being that dep's goal is to have as easy a transition into the toolchain as possible, we're focusing minimally on those areas that complement/extend the existing go toolchain, rather than supplanting existing systems.

Looks like Russ Cox has decided to fix this issue: https://research.swtch.com/vgo-intro

The most significant change, though, is the end of GOPATH as a required place to work on Go code. Because the go.mod file includes the full module path and also defines the version of every dependency in use, a directory with a go.mod file marks the root of a directory tree that serves as a self-contained work space, separate from any other such directories. Now you just git clone, cd, and start writing. Anywhere. No GOPATH required.

Great! It took a long time, but looks like we'll get a happy ending after all.

@sam3d

This comment has been minimized.

Copy link

@sam3d sam3d commented Feb 21, 2018

Now you just git clone, cd, and start writing. Anywhere. No GOPATH required.

Yay! God that's such exciting news.

You can find a mirror for vgo (a prototype implementation of the spec outlined in @andrewrk's comment) at golang/vgo.

@sdboyer

This comment has been minimized.

Copy link
Member

@sdboyer sdboyer commented Feb 21, 2018

@EddieOne

This comment has been minimized.

Copy link

@EddieOne EddieOne commented May 9, 2018

So what is the fix?

@frayhan32

This comment has been minimized.

Copy link

@frayhan32 frayhan32 commented May 14, 2018

In my case how i can solve this is, First i run the export for my GOPATH. Lets assume my GoPath would be. /Users/your-username/Documents/mataharimall-development/www/go/. After that make sure you have directory src under your gopath and your project directory under src folder

@codeAndxv

This comment has been minimized.

Copy link

@codeAndxv codeAndxv commented Jul 15, 2018

you should use this command "dep init" in $GOPATH/src/your projectname

@sam3d

This comment has been minimized.

Copy link

@sam3d sam3d commented Jul 15, 2018

It's annoying that we can't do it without using $GOPATH yet, but it's on the way!

@luisdavim

This comment has been minimized.

Copy link

@luisdavim luisdavim commented Sep 28, 2018

I have a project that I can build from any path in my workstation and I can fetch all the dependencies with:

go get -t -d -v ./...

why does dep ensure fail in that scenario?

@notzippy

This comment has been minimized.

Copy link

@notzippy notzippy commented Oct 25, 2018

Not sure if this is the same issue or not - but it is the same error.

If the GOPATH is based on symlinks the dep tool fails reporting that the realpath does not match any GOPATH. eg

$ export GOPATH=~/home/symlinkedpath/gopath
$ realpath $GOPATH
$ /mnt/dev/gopath
$ cd ~/home/symlinkedpath/gopath/src/test && dep ensure
/mnt/dev/gopath is not within a known GOPATH/src

My quick fix for this is running the dep tool by proxy and creating a new GOPATH based on the filepath.EvalSymlinks for each existing part in the GOPATH

	realPath := &bytes.Buffer{}
	for _, p := range filepath.SplitList(build.Default.GOPATH) {
		rp,_ := filepath.EvalSymlinks(p)
		if realPath.Len() > 0 {
			realPath.WriteString(string(filepath.ListSeparator))
		}
		realPath.WriteString(rp)
	}
        // Call dep using realPath for GOPATH
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.