New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"source" declaration not followed? #860

Open
calmh opened this Issue Jul 20, 2017 · 39 comments

Comments

Projects
None yet
@calmh
Copy link

calmh commented Jul 20, 2017

Dep version v0.1.0-236-g98f1d99

Attempting to work around #767. The following declaration feels like it should handle my situation (Gopkg.toml):

[[constraint]]
  name = "stvn.cc/broadcaster"
  source = "https://github.com/stvnrhodes/broadcaster.git"
  revision = "4d2a8f427f43c008b3e854b91e241196e23310b2"

However,

$ dep ensure
solve error: No versions of <redacted> met constraints:
	master: unable to deduce repository and source type for: "stvn.cc/broadcaster"
ensure Solve(): No versions of <redacted> met constraints:
	master: unable to deduce repository and source type for: "stvn.cc/broadcaster"

Should the source declaration not remove the need to talk to the (down) stvn.cc domain?

Attempting to do it explicitly results in roughly the same thing:

$ dep ensure stvn.cc/broadcaster:github.com/stvnrhodes/broadcaster@4d2a8f427f43c008b3e854b91e241196e23310b2
could not infer project root from dependency path: stvn.cc/broadcaster: unable to deduce repository and source type for: "stvn.cc/broadcaster"

$ dep ensure github.com/stvnrhodes/broadcaster:stvn.cc/broadcaster@4d2a8f427f43c008b3e854b91e241196e23310b2
solve error: No versions of <redacted> met constraints:
	master: unable to deduce repository and source type for: "stvn.cc/broadcaster"
ensure Solve(): No versions of <redacted> met constraints:
	master: unable to deduce repository and source type for: "stvn.cc/broadcaster"

There should be some way to say "grab this github repo into this import path" without the import path having to resolve?

@carolynvs

This comment has been minimized.

Copy link
Collaborator

carolynvs commented Jul 20, 2017

There should be some way to say "grab this github repo into this import path" without the import path having to resolve?

That would be nice. So far we have been telling people to make sure that their imports have the .git suffix and there is an epic for supporting private repos which I hope will address this long term.

#286 (comment) has more info on the workaround and the on-going efforts to implement a solution.

@sdboyer

This comment has been minimized.

Copy link
Member

sdboyer commented Jul 21, 2017

hi! thanks for the issue. This is a great question.

Should the source declaration not remove the need to talk to the (down) stvn.cc domain?

So, I totally get why these should seem like the same thing. But they're not, because there's two separate things that are intertwined, here.

Everything comes back to project root deduction - given an import string, which subcomponents of it identify where the root of the repository - and, because dep assumes that repo root == project root - and root of the project lie?

In a case like this, it seems obvious - broadcaster is the final element in both the import path and the source, but that's not a relationship that's guaranteed to hold. Additionally, and most problematically, we're not guaranteed that the value of name actually identifies the repo root (e.g. #843 (comment)), because who knows what people put in there? It's only valid to put root names in there, but if it's not a pattern we can recognize, either through general deduction rules or a vanity import server, then we can't know that it's actually a valid root.

And that just ruins everything. If we can't trust that name is actually a valid root, then we can't know how it should correspond to source.

The obvious option here is to assume that name IS correctly pointing at a root, and work from there. This didn't occur to me as a serious possibility in the early design of gps, in large part because I was specifically remaining agnostic on the idea of project root == repo root. But, that's an assumption that is now foundational to dep, which changes the landscape.

I've been turning it over in my mind for a few weeks, and I think it's something we can probably do. However, changing the meaning of that relationship has some pretty far-reaching implications, so I need to walk it out to its logical conclusion before being sure. That's probably going to take a little while, unfortunately - I don't have a lot of spare time at the moment.

@calmh

This comment has been minimized.

Copy link

calmh commented Jul 21, 2017

I see. As a user, I never considered that the given source wouldn't have to correspond exactly to the name it "sources". I don't think I really know what it would mean if they didn't, or how dep could possibly handle that case. In my mind, dep would need to do the usual dance to figure out which part of the source is the repo root. That is, I might say something like:

name = "example.com/pkg/broadcaster"
source = "https://github.com/calmh/forks/e/p/bcst"

because I have vendored the example.com/pkg/broadcaster in my special repo of forked packages, under my special naming convention, because I'm just special that way. (I'm not that special really, but I would expect something like this to work.)

Given that I would expect dep to inspect the source as necessary to figure out that https://github.com/calmh/forks is the repo and the rest must be a path, and then grab whatever is found at that path and make it the package at example.com/pkg/broadcaster because that's what I claimed it is.

It would not gain any information by trying to talk to example.com, which does not exist.

Clearly my example is a bit unrealistic but honestly I've seen worse, especially in in-house development where the source repo might live on a server called internal-gitlab.city.state.country.internal.companydomain.com under a path with a structure dictated by the IT department, and we really want something nicer to be the package path. Maybe company.com/pkg/foo although the main company.com website will have no knowledge of the package. Talking to the marketing department to have them add the appropriate go-get headers to the main website is probably out of the question for a myriad of reasons. :)

@sdboyer

This comment has been minimized.

Copy link
Member

sdboyer commented Jul 21, 2017

As a user, I never considered that the given source wouldn't have to correspond exactly to the name it "sources".

Nor did I, as a user. Only when I made the leap to designer did I veer off that particular happy path 😄

I don't think I really know what it would mean if they didn't, or how dep could possibly handle that case.

It would probably be meaningless in terms of the semantics we expect, but this is a problem with stringly typed systems - they tend to allow a lot of meaningless things. It's generally very difficult to draw boundaries around what is or is not valid, in any strictly enforceable sense.

and then grab whatever is found at that path and make it the package at example.com/pkg/broadcaster because that's what I claimed it is.

s/package/project/, but basically yeah, this is what I'm saying I think we can do with minimal harm.

@justinfx

This comment has been minimized.

Copy link

justinfx commented Oct 12, 2017

Directed here from #174, where I am having similar issues with "source" definition not preventing an http lookup. At my studio we have a private internal network with limited access to the external network via http(s) proxy. First off, I cannot use dep init because it will try to talk to both the internal and external git endpoints. If my proxy is set, it will fail on the internals, if it is not set it will hang on the externals. Ok, so I hand write a Gopkg.toml file...

Given the Gopkg.toml file:

[[constraint]]
  name = "github.com/external/project"
  version = "1.0.0"

[[constraint]]
  name = "gitlab.mycompany.com/internal/project"
  version = "1.0.0"
  source = "https://gitlab.mycompany.com/internal/project.git"

I will then set the proxy so that the external dependencies can be reached. But running dep ensure still results in the tool trying to talk to the internal gitlab server, which fails with a 403 because the proxy is set. No combination of setting the source parameter seems to prevent this communication.

How can I go about using the dep tool in this workflow? How can I control the difference between endpoints that need to use the proxy and ones that do not?

@JelteF

This comment has been minimized.

Copy link
Contributor

JelteF commented Oct 24, 2017

Maybe this is a stupid idea, but I think I see quite an easy solution for this. We could simply allow an extra key in a constraint, called something like repo-root, that specifies the repository root.

@calmh

This comment has been minimized.

Copy link

calmh commented Oct 24, 2017

I still think the most reasonable thing to do is to have source correspond exactly to name. If this means you need to point to the repo root and not to a package somewhere inside it, that's fine. You can still have dep check out the repo and use a package inside it.

[[constraint]]
  name = "golang.org/x/crypto"
  version = "whatevs"
  source = "https://github.com/calmh/my-crypto-fork.git"

This should result in my fork of the x/crypto package under the import path golang.org/x/crypto. Similarly,

[[constraint]]
  name = "golang.org/x/crypto/sha768"
  version = "whatevs"
  source = "https://github.com/calmh/my-sha768.git"

should give me my hypothetical SHA768 package at golang.org/x/crypto/sha768 and at no point go talking to golang.org to see what it thinks about that package path or any other.

@JelteF

This comment has been minimized.

Copy link
Contributor

JelteF commented Oct 24, 2017

@calmh, I think you make a good point and my previous solution is just confusing. The only problem I see with yours is that in that case there's no way to have a constraint like this:

[[constraint]]
  name = "golang.org/x/crypto"
  version = "whatevs"
  source = "https://github.com/calmh/my-forks.git/x/crypto"

This should be easily solvable however by adding an extra key like I proposed before, but the opposite this time:

[[constraint]]
  name = "golang.org/x/crypto"
  version = "whatevs"
  source = "https://github.com/calmh/my-forks.git"
  subdir = "x/crypto"

Here source points to the repo root, which makes sense to me. and you can specify a subdir key if the package root is not at the root of the repo. I think I've see something similar in other package managers.

@calmh

This comment has been minimized.

Copy link

calmh commented Oct 24, 2017

Potentially that could be handled like the Go tool handles import paths: look for the .git and assume that's where the repo root is. I think it's fine to make a few requests to interrogate the path mentioned in the source, as long as dep doesn't try to talk to what is mentioned as the name (it being just an import path).

@sdboyer

This comment has been minimized.

Copy link
Member

sdboyer commented Oct 24, 2017

we do use the .git infix as part of deduction - same as go get. to be clear: including that infix in your import paths (or...i think? the source paths) is sufficient to make any repository work. that basic fact is why i don't consider any of this to be especially urgent - there's already recourse here.

subdir = "x/crypto"

This is also a no-go - we don't allow chopping up repos. see e.g. #982. if nothing else, that'd permit a project to bypass internal/ dir structures with just a few lines (doing so is still possible, but "possible" and "dep facilitates it" are two different beasts).

@easyrasta

This comment has been minimized.

Copy link

easyrasta commented Nov 16, 2017

workaround: use 0.3.0 release which works with a source not equals to name

@sdboyer

This comment has been minimized.

Copy link
Member

sdboyer commented Nov 16, 2017

@easyrasta that might be a partial workaround for certain cases, but there is no workaround for the general case where name itself is not deducible via the static rules, and no go-get HTTP metadata server responds when dep makes a request for that info.

@calmh

This comment has been minimized.

Copy link

calmh commented Nov 16, 2017

we do use the .git infix as part of deduction - same as go get. to be clear: including that infix in your import paths (or...i think? the source paths) is sufficient to make any repository work. that basic fact is why i don't consider any of this to be especially urgent - there's already recourse here.

I tried this now, and it does in fact work, which is good. The problem is, it doesn't solve the original problem as it requires a change of the import path. Taking my example, modified with .git for anchoring:

[[constraint]]
  name = "stvn.cc/broadcaster.git"
  source = "https://github.com/stvnrhodes/broadcaster.git"

this results in

Solving failure: No versions of stvn.cc/broadcaster.git met constraints:
	master: Could not introduce stvn.cc/broadcaster.git
  (from https://github.com/stvnrhodes/broadcaster.git)@master, as its
  subpackage stvn.cc/broadcaster.git does not contain usable Go code
  (*pkgtree.NonCanonicalImportRoot).. (Package is required by (root).)

because the stvn.cc/broadcaster package uses the canonical import path thing (which I've learned to loathe...) thus stvn.cc/broadcaster.git is not usable.

So yeah, it's a recourse, in some cases.

(The stvn.cc domain is now up and works so there is no need for this specific gymnastic, this is just an example.)

@sdboyer

This comment has been minimized.

Copy link
Member

sdboyer commented Nov 17, 2017

because the stvn.cc/broadcaster package uses the canonical import path thing (which I've learned to loathe...) thus stvn.cc/broadcaster.git is not usable.

oh man, i'd not thought about the interaction of these two pieces before. yeah, that really sucks. we're really starting to overload names quite a bit 😢

@nmiyake

This comment has been minimized.

Copy link
Contributor

nmiyake commented Dec 17, 2017

Various issues around GHE seem to point down to this one as the one root issue, but trying to understand exactly what the current proposal is for the solution.

Let's say I have a GHE installation at github.company.internal and want to take a dependency on org/repo. I set the following constraint:

[[constraint]]
  name = "github.company.internal/org/repo.git"
  source = "git@github.company.internal:org/repo.git"

However, dep ensure fails with:

ensure Solve(): No versions of github.company.internal/org/repo.git met constraints:
        0.1.0: unable to deduce repository and source type for "github.company.internal/org/repo/repo-api": unable to read metadata: go-import metadata not found

I'm guessing that this is because the code in org/repo has code that uses the import path "github.company.internal/org/repo/repo-api" rather than "github.company.internal/org/repo.git/repo-api".

Doing gymnastics (like adding .git to import paths) in the project using dep is somewhat feasible, but if using dep requires updating all of the import paths for all of the internal projects that might ever be added using dep, that's a pretty steep cost (and would also break other workflows).

Is there a current proposal (or thoughts for a proposal) for how to deal with this case?

@hoshsadiq

This comment has been minimized.

Copy link

hoshsadiq commented Dec 23, 2017

To add to this issue, my current implementation as mentioned in #411 is as below but to no avail.

[[constraint]]
  name = "github.company.com/org/repo.git"
  branch = "master"
  source = "ssh://github.company.com/org/repo"

And the import

import (
	"os"
	log "github.com/sirupsen/logrus"
	myapp "github.company.com/org/repo.git/some_import"
	"github.com/fatih/structs"
	"path/filepath"
	"strings"
)

Output (copy pasted from Jenkins):

+ /home/jenkins/workspace/my-app/bin/dep ensure

Solving failure: unable to deduce repository and source type for "ssh://github.company.com/org/repo": unable to read metadata: go-import metadata not found
@sdboyer

This comment has been minimized.

Copy link
Member

sdboyer commented Dec 28, 2017

@floren if you're still on a rev that is in both your fork and the upstream, then it is honoring the source declaration. whether or not you're at the latest master is an orthogonal consideration - though we are venturing into territory here where the semantics of constraints get a little fuzzy, when combined with source.

have you tried running dep ensure -update github.com/google/gopacket?

@floren

This comment has been minimized.

Copy link

floren commented Dec 29, 2017

@sdboyer weirdly it "just worked" after one more -update, despite having not worked before. I got someone else to check out the repo and test it, it seemed to work for him too. So barring any recurrence later on I'm going to assume I'd just misconfigured it initially and eventually a combination of tweaking the config and running update fixed it!

@hoshsadiq

This comment has been minimized.

Copy link

hoshsadiq commented Jan 12, 2018

By the way, I was able to do this which worked:

[[constraint]]
  name = "github.company.com/org/repo.git"
  branch = "master"
  source = "ssh://git@github.company.com/org/repo.git"

Then

import (
	mylib "github.company.com/org/repo.git/library"
)
@floren

This comment has been minimized.

Copy link

floren commented Jan 12, 2018

@hoshsadiq That requires re-writing import paths which rather conflicts with our desire to transparently drop in our own copy of the repo for testing.

@sdboyer

This comment has been minimized.

Copy link
Member

sdboyer commented Jan 14, 2018

well, in addition to technical solutions that we're contemplating...

IF YOUR COMPANY IS A GITHUB ENTERPRISE CUSTOMER, please DM me on twitter. i'm trying to organize some collective pressure.

@hoshsadiq

This comment has been minimized.

Copy link

hoshsadiq commented Feb 7, 2018

@sdboyer is there a reason we need to use DM on twitter? I feel like this thread should be enough. who is the collective pressure for?

@qknight

This comment has been minimized.

Copy link

qknight commented Feb 19, 2018

how to use with git+ssh

SOLUTION: we have our own git server via ssh on nixcloud.io and ssh is running on port 20202 and we don't want to copy our source code to github or similar.

so here is how we did it:

dep version

dep:
 version     : 0.4.1
 build date  : 
 git hash    : v0.4.1
 go version  : go1.9.3
 go compiler : gc
 platform    : linux/amd64

installed via nix-env -i dep but this is not a nix or nixos specific solution and it should work on every linux using dep.

git clone

first make sure, that git clone works with your ssh setup:

git clone ssh://gitolite@nixcloud.git:20202/nixcloud.stalker.git
Klone nach 'nixcloud.stalker' ...
remote: Counting objects: 53, done.
remote: Compressing objects: 100% (46/46), done.
remote: Total 53 (delta 23), reused 0 (delta 0)
Empfange Objekte: 100% (53/53), 10.81 KiB | 10.81 MiB/s, Fertig.
Löse Unterschiede auf: 100% (23/23), Fertig.

in Gopkg.toml

[[constraint]]
  name = "nixcloud.io/nixcloud.stalker.git"
  source = "ssh://gitolite@nixcloud.git:20202/nixcloud.stalker.git"

/root/.ssh/config

Host nixcloud.git
    hostname nixcloud.io
    Port 20202
    User gitolite
    IdentityFile ~/.ssh/id_rsa-gitolite-nixcloud

main.go

package main

import (
  "nixcloud.io/nixcloud.stalker.git"
  ...

drawback: we have to rename the import with the .git suffix. this is ok for our in-house code but it is no transparent replacement if we would want to hack on third party git dependencies.

@qknight

This comment has been minimized.

Copy link

qknight commented Feb 19, 2018

@sdboyer could we add a file:/// backend also, so that we could bundle our own libraries using a git subdirectory? this would make it so much easier to hack on third party libraries until PRs are accepted and one can change back to the upstream version.

[[constraint]]
  name = "nixcloud.io/nixcloud.stalker"
  source = "file://./nixcloud.stalker"

update I

as @hoshsadiq pointed out (below), it would be important, that this directory is a symlink or hardlink to the directory referenced in source from inside the vendor folder so that it can be edited like the source code of the project itself.

update II

i've got the impression that we just need to add a new deducer in something like func pathDeducerTrie() *deducerTrie { which has the pre-knowledge and is called first.

additionally we could overload the source with information for the new deducer as:

[[constraint]]
  name = "nixcloud.io/nixcloud.stalker"
  source = "git||ssh://gitolite@nixcloud.io/nixcloud.stalker"
or
  source = "file||./nixcloud.stalker.git"
@hoshsadiq

This comment has been minimized.

Copy link

hoshsadiq commented Feb 19, 2018

I think the main issue is having to import with a .git suffix. When developing you'll not be editing your files from the vendor directory, and since you'll have your repo in $GOPATH/src/mycompany.com/team/repo your IDE (and go get) will pick that up. However, when you import with .git suffix, that means you'll have a different version in your vendor directory compared to what you're actually working on.

@outcoldman

This comment has been minimized.

Copy link

outcoldman commented Feb 25, 2018

I believe I have a solution for private repos with ssh access, see PR #1717

@domino14

This comment has been minimized.

Copy link

domino14 commented Apr 3, 2018

Adding the .git suffix does not work on AWS Codecommit. Any other workarounds?

@outcoldman

This comment has been minimized.

Copy link

outcoldman commented Apr 3, 2018

@domino14 unfortunately what I have found, you cannot just use AWS Codecommit repos as is, because of the folder structure they use. As a workaround I use in gitconfig (make sure to use correct region instead of us-east-1)

[url "ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/"]
  insteadOf = git+ssh://example.com/
  insteadOf = ssh://example.com/

After that you will be able to use .git suffix

example.com/myrepo.git

But that will work only with the patch I have sent.

@domino14

This comment has been minimized.

Copy link

domino14 commented Apr 3, 2018

They haven't merged that patch yet... this is a serious issue, we should really merge this, dep is completely unusable on codecommit, and godep is an archived project (and i don't know if it would even work with this). Anyone want to merge the patch or have any workarounds? Why does go get work?

@outcoldman

This comment has been minimized.

Copy link

outcoldman commented Apr 3, 2018

@domino14 go get works, because dep made some assumptions on how to resolve the projects, which aren't compatible with go dep, like the way they resolve remotes and the order (ssh vs https) of how they resolve it.

The only possible workaround is for every project you use as a dependency from CodeCommit to have a configuration in gitconfig

[url "ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/myrepo"]
  insteadOf = git+ssh://example.com/myrepo.git
  insteadOf = ssh://example.com/myrepo.git
@domino14

This comment has been minimized.

Copy link

domino14 commented Apr 3, 2018

@outcoldman

This comment has been minimized.

Copy link

outcoldman commented Apr 3, 2018

@domino14 it is a workaround with current dep, the change I made will not be required.

@domino14

This comment has been minimized.

Copy link

domino14 commented Apr 3, 2018

@g-gaston

This comment has been minimized.

Copy link

g-gaston commented Jun 10, 2018

@sdboyer any update or possible solution?
In my case, my repo url has a custom port so I can't use it as the import path (plus the url is kind of ugly as in many private corporate repositories, so it would be nice to be able to override it).

I agree with @calmh on removing the need to talk to name when source is declared. I would like to do something like:

[[constraint]]
  name = "company.com/repo1"
  version = "whatevs"
  source = "https://team1.city.go.us.bitbucket.company.com:7823/projects/f/sc/repo1"

This is preventing me for using my own repos with dep so I wouldn't mind putting some time on this, but some guidance before starting the work would be great.

@anitgandhi

This comment has been minimized.

Copy link

anitgandhi commented Jun 13, 2018

I'm in a similar situation in that we don't have GitHub Enterprise v2.13 deployed yet

https://enterprise.github.com/releases/2.13.0/notes

Support Go's remote import path when private mode is configured.

So, that means importing a library hosted on our GHE doesn't work. go get github.mycompany.com/namespace/repo will work if you explicitly tack on .git to the end of that path. I tried setting the source in Gopkg.toml to that path suffixed with .git, but since source path isn't followed, it still tries to hit the name, where it gets the old unable to read metadata: go-import metadata not found error.

In my opinion this isn't using source to get around network access issues, it's just using it to enable a pre GHE 2.13 workflow.

@yanyandenuonuo

This comment has been minimized.

Copy link

yanyandenuonuo commented Jul 24, 2018

if can't custom the repo url with the "source", what's the means of the "source" field

@jhagege

This comment has been minimized.

Copy link

jhagege commented Aug 21, 2018

+1, is there a workaround for go-get ?
We need to use the URL in source and not in name..

@michaelkunzmann-sap

This comment has been minimized.

Copy link

michaelkunzmann-sap commented Oct 1, 2018

I have a related issue. We upgraded to GitHub Enterpise 2.14 but we now have the following issue:

$ dep ensure -vendor-only
The following issues were found in Gopkg.toml:
unable to deduce repository and source type for "github.company.com/org/repo": unable to read metadata: unable to fetch raw metadata: failed HTTP request to URL "http://github.company.com/org/repo?go-get=1": Get http://github.company.com/org/repo?go-get=1: dial tcp 10.67.76.20:80: i/o timeout

It seems like godep is trying to use port 80 and is probably assuming that the GitHub Enterprise instance answers with a redirect, like github.com does it:

curl -v github.com
* Rebuilt URL to: github.com/
*   Trying 192.30.253.113...
* Connected to github.com (192.30.253.113) port 80 (#0)
> GET / HTTP/1.1
> Host: github.com
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Content-length: 0
< Location: https://github.com/
< Date: Mon, 01 Oct 2018 22:54:54 GMT
< Age: 0
< Connection: keep-alive
< Via: 1.1 akamai (ACE 5.10.2/5.10.2)
<
* Connection #0 to host github.com left intact

But what our instance actually does:

curl -v http://github.company.com
* Rebuilt URL to: http://github.company.com/
*   Trying 10.67.76.20...
* Connected to github.company.com (10.67.76.20) port 80 (#0)
> GET / HTTP/1.1
> Host: github.company.com
> User-Agent: curl/7.47.0
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

We tried overriding the URL by using:

[[constraint]]
  name = "github.company.com/org/repo"
  source = "https://github.company.com/org/repo.git"

But no success. Is there another way we might be able to "force" dep to use https://?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment