Allow custom names to be used

[mofirouz]

What version of dep are you using (dep version)?

dep 0.3.1 (installed via Brew)

What dep command did you run?

dep ensure

What did you expect to see?

dependency to get resolved + added

What did you see instead?

The following issues were found in Gopkg.toml:

  ✗ "nakama" is not a valid import path

ProjectRoot name validation failed

=====

We have an internal project that depends (and extends) an open-source project. The project in mind is Nakama. The internal project extends Nakama and implements various interfaces, and uses various packages available. We therefore have to use Nakama's import paths.

Previously we were using Glide and had setup the dependency as this:

- package: nakama
  repo: git@github.com:heroiclabs/nakama.git

With dep this cannot be repeated because of the overzealous regex found here (?).

FWIW, the exact compiler error when compiling Nakama is:

cannot use client (type *"project/vendor/github.com/heroiclabs/nakama/pkg/social".Client) as type *"nakama/pkg/social".Client

Could you kindly advice on whether this is some form of Bug/inflexibility or our misuse of dep?

[darkowlzz]

@mofirouz hi, any reason for not using github.com/heroiclabs/nakama as the package name?

[mofirouz]

@darkowlzz yep, because Nakama itself uses this package naming structure:
"nakama/pkg/social" in https://github.com/heroiclabs/nakama/blob/master/server/pipeline.go#L21

and in my project I also need to use the same code, which of course in Dep it will be
"github.com/heroiclabs/nakama/pkg/social"

at which point golang compiler complains because the two structs are not the same because of their import path.

cannot use client (type *"project/vendor/github.com/heroiclabs/nakama/pkg/social".Client) as type *"nakama/pkg/social".Client

[darkowlzz]

oh! maybe @sdboyer can help us here.

[sdboyer]

hi! so, the real issue here is that Nakama has evidently chosen to use a non standard import structure that simply isn't portable (read: not go-gettable), and they probably need to fix that. I'm in transit right now and don't have time to check their repository for any justification they may offer as to why they use this structure, but i do not know of any general reason why doing so would be necessary. I suspect it's just a choice they made, perhaps because the project is already not go-gettable for other reasons (eg has a makefile)

unfortunately, supporting this is not very easy in our current model, for a few reasons. it's conceivable that we could do it if we fold it in with #860, though this would likely require a few additional tweaks. however, I'd want to see at least one case of projects creating import structures like this needing it for defensible reasons before we consider supporting it.

[mofirouz]

@sdboyer I'm also one of the authors of Nakama. The modules in the pkg folder are written for the benefit of the community and can be go-gettable. Nakama is not go-gettable because the server package shouldn't be for the general public, but in this particular case, there is a need to wrap the Nakama server with custom code (for internal purposes).

As far as I've (briefly) looked into the dep code, you've whitelisted certain domains and whitelisted a particular regex structure.

I'd like to understand why this limitation exists? The go-compiler definitely doesn't have that limitation and prior art (like Glide) specifically allowed this use-case?

[lsegal]

Note that this has also become a problem for imports of the go-plist project using the expected canonical import path of howett.net/plist (even though the source repository is at GitHub), since the canonical path is currently not go gettable (see reported issue).

I think in this case the problem can be fixed by the project maintainer, but it seems like there should be a way to support this type of use case so that downstream users do not have to be blocked by these external variables, like defensible positions, or even non-defensible positions, that project maintainers choose to make. Even though they break convention, these package names are fundamentally supported by the Go language, which means they can, will, and do exist.

Also, pragmatically speaking, it seems like without this type of escape hatch, there's a real possibility for a network outage or server failure to completely disable dep from working with no reasonable workaround. If, for example, howett.net/plist was working but was experiencing some temporal server issue (404s, 500s) for hours or days, end users would be completely blocked.

[sdboyer]

@mofirouz ugh, i'm sorry, i was in the midst of a bunch of conferences when you replied, and didn't see your response 😢

I'd like to understand why this limitation exists? The go-compiler definitely doesn't have that limitation

so, the go compiler isn't the relevant comparison here; dep is a compiler-adjacent tool, but it is not itself a compiler, nor does it ever invoke one. rather, go get is the better comparison. what dep has isn't whitelists; rather, they're rules that teach dep how to deal with specific hosting services - specifically, what type of VCS is in use, and which path element represents the repository root. (e.g., for github, it's always the third.)

with the exception of special rules we've created in order to convert the semantics of gopkg.in to dep's model, the domains we have special rules for are the same as the ones supported by go get. in other words, we're following convention - just, extending it further and being more uniform in its application.

and prior art (like Glide) specifically allowed this use-case?

there's more discussion on the specifics of this in #860. while allowing a workaround is something i'd like dep to do, it still shouldn't be the preferred mechanism for supporting cases like this.

Nakama is not go-gettable because the server package shouldn't be for the general public, but in this particular case, there is a need to wrap the Nakama server with custom code (for internal purposes).

i think i understand your goal (though i do not know what these internal purposes are that motivate the design), but the established approach to signaling that code is not for public consumption is to place such packages beneath an internal path. both that and your approach are things that the user can ultimately circumvent, but only the internal path unambiguously signals to users what your intent is.

@lsegal I think in this case the problem can be fixed by the project maintainer, but it seems like there should be a way to support this type of use case so that downstream users do not have to be blocked by these external variables, like defensible positions, or even non-defensible positions, that project maintainers choose to make.

as a general principle, i 💯 agree. and dep does quite a lot of work to fulfill that principle. there are inevitably some cases, though, where supporting them would require us to undermine our models and rules to the point where they're no longer really rules. and that would sink dep's utility as a tool.

now, the workaround outlined in #860 is likely sufficient for this particular case (though i kind of wish it weren't, as IMO it is an antipattern to force the use of a source field on users).

Also, pragmatically speaking, it seems like without this type of escape hatch, there's a real possibility for a network outage or server failure to completely disable dep from working with no reasonable workaround.

yes, right now, dep is uncomfortably bonded to the network. this is not the only place where it's a problem. however, the preferable solution to this problem is a more pervasive "offline mode", as that's something we can transparently switch on as long as we have a populated cache. achieving such an offline mode is a next-phase goal following after #431.

[mattmoor]

Why not allow this via: <import path>[:alt source URL][@<constraint>]?

e.g. dep ensure -add nakama:github.com/heroiclabs/nakama.git@master

[chendrix]

Canonical import paths are part of the golang spec. Dep must be able to handle them.

[sdboyer]

@mattmoor that doesn't actually change anything. CLI options are not and cannot be any more powerful or expressive than what's allowed by Gopkg.toml. If they were, there'd be no way of persisting the rule expressed on the CLI for future runs. It takes us right back to the aforementioned issues.

@chendrix canonical import paths - that is, the functionality added in Go 1.4 - is separate from this issue. We've wired in the basic capability to read them, though we haven't actually hooked up a rule into the solver yet - #902. Still a separate concern from this issue, though.

[cupen]

My internal project was hosted in a internal server. There is no domain name for it, but IP.
So ……

import "192.168.1.119/my/project"

haha, it's funny.