New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for private repositories e.g. Github Enterprise #174

Open
mikkeloscar opened this Issue Jan 26, 2017 · 43 comments

Comments

Projects
None yet
@mikkeloscar
Contributor

mikkeloscar commented Jan 26, 2017

I didn't see this addressed in any of the docs, so I'll raise the question
here.

Have you considered support for private repositories?

Many of the existing vendor tools does not handle private repositories very
well or at all. godep and govender can sort of handle it because they get
the dependencies from your $GOPATH. But existing tools that fetches directly
from the remote does not work with private repos in my experience (If someone
knows any tools that does work, please let me know :) ).

It would be great if dep could break this trend and work well with private
repositories from the start.

I'm probably not covering all use cases, but these are the use cases I would
like to see solved.

As a developer (consumer) I want to import dependencies from a private Github
Enterprise repository my.ghe.com.
Assuming I can already clone repositories either via https or ssh, I don't
want to also configure the authentication in dep. Configuring it one time
should be enough e.g. in git (or ssh).

As a developer I want to avoid committing the vendor folder, thus my CI/CD
system must be able to pull in all dependencies (also those from a private
repo) when running a build.

As a developer I don't want to encode the type of repository in my source
code e.i. I don't want to specify a vcs extension in my import: `import
"my.ghe.com/repo/package**.git**"

From my point of view this could be as simple as introducing an environment
variable which maps your private repo URL to a certain protocol. E.g.:

GO_DEP_REPO=my.ghe.com=git+ssh:my.ghe2.com=git+https

Which means whenever dep encounters an import path which starts with
my.ghe.com it will automatically choose git+ssh when fetching the repo. And
since I already have configured an ssh key on my machine it should know how to
authenticate.
Similarly when it encounters an import path with my.ghe2.com it
will choose git+https and ask me for username+password or just work if I
have configured what token to use in .gitconfig.

I'm sure someone could think of a better name for the environment variable and
a better format, but I hope you get my point.

@zevdg

This comment has been minimized.

Show comment
Hide comment
@zevdg

zevdg Jan 26, 2017

At my company, we solve this exact problem using go's vanity imports. We have something like https://github.com/dominikh/go-vanity which serves up the appropriate protocol depending on various factors and we use the url of the vanity server instead of our ghe url in our import paths.

zevdg commented Jan 26, 2017

At my company, we solve this exact problem using go's vanity imports. We have something like https://github.com/dominikh/go-vanity which serves up the appropriate protocol depending on various factors and we use the url of the vanity server instead of our ghe url in our import paths.

@freeformz

This comment has been minimized.

Show comment
Hide comment
@freeformz

freeformz Jan 27, 2017

Contributor

This is part of our spec (https://docs.google.com/document/d/1qnmjwfMmvSCDaY4jxPmLAccaaUI5FfySNE90gB0pTKQ/edit#). Search for "alt location".

Does that work, at least basically for your needs?

With that said it doesn't exist atm.

Contributor

freeformz commented Jan 27, 2017

This is part of our spec (https://docs.google.com/document/d/1qnmjwfMmvSCDaY4jxPmLAccaaUI5FfySNE90gB0pTKQ/edit#). Search for "alt location".

Does that work, at least basically for your needs?

With that said it doesn't exist atm.

@sdboyer

This comment has been minimized.

Show comment
Hide comment
@sdboyer

sdboyer Jan 27, 2017

Member

First - solving this with vanity imports via an external system, as @zevdg describes, is far easier. (I didn't know about go-vanity - that's awesome.) That said, vanity imports still have some of the same issues as what's described here.

Have you considered support for private repositories?

We didn't talk about it much directly within the committee as it wasn't an immediate necessity, but I've thought about this quite a lot. It's in the top three nasty problems I've grappled with over the last year.

To be clear - these problems are solvable, but they add a ton of new, potentially obnoxious failure modes, so I've been very cautious about adding support for them in gps. I've also been thinking about them for way too long, and have gotten somewhat lost in the forest - help with weighing tradeoffs is very welcome :)

This is kind of a slapdash writeup, but hopefully it covers the basic bases.

The essential problem is portability: we have to be able to deduce the source/repo for all import paths we encounter. If that's only possible by relying on custom rules, then anyone trying to work with your project without those custom rules won't be able to resolve those import paths. The really nasty part, though, is that we're unable to provide users in that situation with useful error messages, because it's impossible to tell the difference between "invalid import path" and "invalid import path because you're missing some custom deduction rules." How could we know such custom rules exist?

If you're working on a private, internal company system, then this sort of portability isn't really a concern. The problem with a feature like this is that it wouldn't just be used for that - how many people would write their own little custom rule for, say, gitlab (for which we don't currently have a built-in rule)?

We could mitigate this by having the custom rules live in the manifest, alongside constraints/ignores/requires/overrides. composer does this, generally to good effect. But, if you're in a company with a bunch of different projects, then each would need to define the same rules in their manifests. (Pattern definition would have to be a root-manifest-only thing, for basically the same reasons as composer gives). If you have to make any changes in how those rules work, you have to make them in all your projects, and it could potentially make it impossible to use older versions of those projects, because your new rules don't work with the old imports.

But all this works well enough in composer, right? Sure. Things are strictly more difficult for Go, though, because this problem of deducing projects/repositories from import paths is one that composer simply doesn't have. Composer is just dealing with the name of a dep and a location from which to retrieve it and its metadata. Because dep tries hard to be truly idiomatic, we still treat the import graph, not the manifest, as the canonical expression of what code is required (c.f. writeup; we follow option 2). So, rather than just naming a dep and where to get it from, we have to specify a pattern that can be used to match import paths that we encounter and map them back to their repository root.

Plus, composer constrains names to have two parts, but we have no such guarantee - launchpad.net/someproject has two, but github.com/golang/dep has three. Variable-length root names make it impossible to designate a general rule that, say, "all custom rules must have two path parts," which would make ensuring that we don't have any conflicting rules much easier.

Also, as one final kick in the pants, because we're ultimately talking about a system to decide what names (import paths) mean, it would have to be incorporated into the input hashing system we use to determine when you need to regenerate your lock. (That's the memo field in your lock.json - this is a very experimental idea that we may drop)

Again, I do think there are reasonable solutions here. But I have yet to figure out a good, balanced compromise that ratchets down control to avoid rendering the entire system unsafe; I've been enmeshed in this problem for a long time, and have lost a lot of the perspective necessary to weigh tradeoffs. Feedback is super welcome :)

Member

sdboyer commented Jan 27, 2017

First - solving this with vanity imports via an external system, as @zevdg describes, is far easier. (I didn't know about go-vanity - that's awesome.) That said, vanity imports still have some of the same issues as what's described here.

Have you considered support for private repositories?

We didn't talk about it much directly within the committee as it wasn't an immediate necessity, but I've thought about this quite a lot. It's in the top three nasty problems I've grappled with over the last year.

To be clear - these problems are solvable, but they add a ton of new, potentially obnoxious failure modes, so I've been very cautious about adding support for them in gps. I've also been thinking about them for way too long, and have gotten somewhat lost in the forest - help with weighing tradeoffs is very welcome :)

This is kind of a slapdash writeup, but hopefully it covers the basic bases.

The essential problem is portability: we have to be able to deduce the source/repo for all import paths we encounter. If that's only possible by relying on custom rules, then anyone trying to work with your project without those custom rules won't be able to resolve those import paths. The really nasty part, though, is that we're unable to provide users in that situation with useful error messages, because it's impossible to tell the difference between "invalid import path" and "invalid import path because you're missing some custom deduction rules." How could we know such custom rules exist?

If you're working on a private, internal company system, then this sort of portability isn't really a concern. The problem with a feature like this is that it wouldn't just be used for that - how many people would write their own little custom rule for, say, gitlab (for which we don't currently have a built-in rule)?

We could mitigate this by having the custom rules live in the manifest, alongside constraints/ignores/requires/overrides. composer does this, generally to good effect. But, if you're in a company with a bunch of different projects, then each would need to define the same rules in their manifests. (Pattern definition would have to be a root-manifest-only thing, for basically the same reasons as composer gives). If you have to make any changes in how those rules work, you have to make them in all your projects, and it could potentially make it impossible to use older versions of those projects, because your new rules don't work with the old imports.

But all this works well enough in composer, right? Sure. Things are strictly more difficult for Go, though, because this problem of deducing projects/repositories from import paths is one that composer simply doesn't have. Composer is just dealing with the name of a dep and a location from which to retrieve it and its metadata. Because dep tries hard to be truly idiomatic, we still treat the import graph, not the manifest, as the canonical expression of what code is required (c.f. writeup; we follow option 2). So, rather than just naming a dep and where to get it from, we have to specify a pattern that can be used to match import paths that we encounter and map them back to their repository root.

Plus, composer constrains names to have two parts, but we have no such guarantee - launchpad.net/someproject has two, but github.com/golang/dep has three. Variable-length root names make it impossible to designate a general rule that, say, "all custom rules must have two path parts," which would make ensuring that we don't have any conflicting rules much easier.

Also, as one final kick in the pants, because we're ultimately talking about a system to decide what names (import paths) mean, it would have to be incorporated into the input hashing system we use to determine when you need to regenerate your lock. (That's the memo field in your lock.json - this is a very experimental idea that we may drop)

Again, I do think there are reasonable solutions here. But I have yet to figure out a good, balanced compromise that ratchets down control to avoid rendering the entire system unsafe; I've been enmeshed in this problem for a long time, and have lost a lot of the perspective necessary to weigh tradeoffs. Feedback is super welcome :)

@sdboyer

This comment has been minimized.

Show comment
Hide comment
@sdboyer

sdboyer Jan 27, 2017

Member

Oh! I forgot to mention...

All of this ugliness is my own single strongest motivation for creating a registry (#175): we can impose meta-rules on a subset of import paths. For example, imagine this pattern:

    import "p/gopherhoard.com/user/project"
  • The leading p indicates it's a registry source type
  • The second part is the base domain where the registry is hosted. This would allow the crucial feature of allowing people to host their own private (or public) registries; we might optionally have a shorter alias for the main, public registry.
  • There's a fixed number of trailing elements. (We might also just do project name, without namespacing by user).

This solves so many problems.

  • There's no more "unknown rule" problem, because there's a meta-pattern followed by all registries
  • Root deduction is dead-simple
  • The accuracy of input hashing is no longer at risk
  • ...all of the other problems that registries solve, independent of this
Member

sdboyer commented Jan 27, 2017

Oh! I forgot to mention...

All of this ugliness is my own single strongest motivation for creating a registry (#175): we can impose meta-rules on a subset of import paths. For example, imagine this pattern:

    import "p/gopherhoard.com/user/project"
  • The leading p indicates it's a registry source type
  • The second part is the base domain where the registry is hosted. This would allow the crucial feature of allowing people to host their own private (or public) registries; we might optionally have a shorter alias for the main, public registry.
  • There's a fixed number of trailing elements. (We might also just do project name, without namespacing by user).

This solves so many problems.

  • There's no more "unknown rule" problem, because there's a meta-pattern followed by all registries
  • Root deduction is dead-simple
  • The accuracy of input hashing is no longer at risk
  • ...all of the other problems that registries solve, independent of this
@zevdg

This comment has been minimized.

Show comment
Hide comment
@zevdg

zevdg Jan 27, 2017

Thanks for the background! Glad I'm not the only person thinking about this.

The problem that vanity imports don't solve for us is having a centralized cache of 3rd party libraries. We use artifactory to do this from nearly every other language, but since go uses the import path to uniquely identify both the package and its source url, the existing go tool simply cannot handle changing the source url without also changing the package's identifier. We could re-write import paths to point to artifactory, but we'd have to also do that inside of every 3rd party package we use as well so that we don't end up referring to the same package with by different names.

The alternate urls described in dep ensure -h looks like exactly what we need. I was assuming that those alternate urls would be stored in your manifest.json individually for every dependency. Something like

{
    "dependencies": {
        "github.com/golang/protobuf": {
            "branch": "master",
            "source": "artifactory.corp-intranet.com/github.com/golang/protobuf"
        },
        "github.com/pkg/errors": {
            "branch": "master",
            "source": "artifactory.corp-intranet.com/github.com/pkg/errors"
        },
        "ghe.corp-intranet.com/myteam/lib": {
            "branch": "master"
        }
    }
}

So while that leaves us with a lot of seemingly redundant text in the manifest file, it keeps any complex deduction rules out of it. Anyone would be able to pull down this project's deps from the right place assuming they had access to the intranet.

That alone seems "sufficient" to solve the problem, but it requires a lot of discipline for devs to add the correct alternate location to each and every 3rd party dependency. So on some level, I do want a "custom rule" of some sort, but more generally, I'm imagining some kind of onManifestEdit hook there I can run some arbitrary script AFTER dep updates the manifest but BEFORE dep uses manifest to resolve dependencies and update the lockfile. This hook would have the opportunity to also add more updates to the manifest and if it did, dep would re-read and use the new info in generating the lock file. Conceptually, this would be similar to setting up your editor to run goimports followed by go build on save.

In particular, the hook I'd personally want would be along the lines of (in psedo code)

for each package p:
    if p is 3rd party:
        p.source = "artifactory.corp-intranet.com/"+p.importpath

So that on a properly configured machine, dep ensure github.com/golang/protobuf would inject my alternate source but dep ensure ghe.corp-intranet.com/myteam/lib would not. On an unconfigured machine, everything would still work fine, but one would have to manually specify alternate sources for any new dependencies they add.

I imagine there would be other legit use cases for having an automated edit to the manifest here. I haven't thought too deeply about the implication of allowing a hook like this, but since the manifest is supposed to be editable by humans, never mind other tools, this doesn't seem too dangerous at first glance.

Also, we may have gone very far off topic from the original issue. Maye this should be separated into a separate issue or 2?

zevdg commented Jan 27, 2017

Thanks for the background! Glad I'm not the only person thinking about this.

The problem that vanity imports don't solve for us is having a centralized cache of 3rd party libraries. We use artifactory to do this from nearly every other language, but since go uses the import path to uniquely identify both the package and its source url, the existing go tool simply cannot handle changing the source url without also changing the package's identifier. We could re-write import paths to point to artifactory, but we'd have to also do that inside of every 3rd party package we use as well so that we don't end up referring to the same package with by different names.

The alternate urls described in dep ensure -h looks like exactly what we need. I was assuming that those alternate urls would be stored in your manifest.json individually for every dependency. Something like

{
    "dependencies": {
        "github.com/golang/protobuf": {
            "branch": "master",
            "source": "artifactory.corp-intranet.com/github.com/golang/protobuf"
        },
        "github.com/pkg/errors": {
            "branch": "master",
            "source": "artifactory.corp-intranet.com/github.com/pkg/errors"
        },
        "ghe.corp-intranet.com/myteam/lib": {
            "branch": "master"
        }
    }
}

So while that leaves us with a lot of seemingly redundant text in the manifest file, it keeps any complex deduction rules out of it. Anyone would be able to pull down this project's deps from the right place assuming they had access to the intranet.

That alone seems "sufficient" to solve the problem, but it requires a lot of discipline for devs to add the correct alternate location to each and every 3rd party dependency. So on some level, I do want a "custom rule" of some sort, but more generally, I'm imagining some kind of onManifestEdit hook there I can run some arbitrary script AFTER dep updates the manifest but BEFORE dep uses manifest to resolve dependencies and update the lockfile. This hook would have the opportunity to also add more updates to the manifest and if it did, dep would re-read and use the new info in generating the lock file. Conceptually, this would be similar to setting up your editor to run goimports followed by go build on save.

In particular, the hook I'd personally want would be along the lines of (in psedo code)

for each package p:
    if p is 3rd party:
        p.source = "artifactory.corp-intranet.com/"+p.importpath

So that on a properly configured machine, dep ensure github.com/golang/protobuf would inject my alternate source but dep ensure ghe.corp-intranet.com/myteam/lib would not. On an unconfigured machine, everything would still work fine, but one would have to manually specify alternate sources for any new dependencies they add.

I imagine there would be other legit use cases for having an automated edit to the manifest here. I haven't thought too deeply about the implication of allowing a hook like this, but since the manifest is supposed to be editable by humans, never mind other tools, this doesn't seem too dangerous at first glance.

Also, we may have gone very far off topic from the original issue. Maye this should be separated into a separate issue or 2?

@fnouama

This comment has been minimized.

Show comment
Hide comment
@fnouama

fnouama commented Jan 30, 2017

+1

@sdboyer

This comment has been minimized.

Show comment
Hide comment
@sdboyer

sdboyer Jan 30, 2017

Member

The alternate urls described in dep ensure -h looks like exactly what we need. I was assuming that those alternate urls would be stored in your manifest.json individually for every dependency.

So while that leaves us with a lot of seemingly redundant text in the manifest file, it keeps any complex deduction rules out of it. Anyone would be able to pull down this project's deps from the right place assuming they had access to the intranet.

Yep yep!

That alone seems "sufficient" to solve the problem, but it requires a lot of discipline for devs to add the correct alternate location to each and every 3rd party dependency.

Oh for sure - but I'll go two steps stronger.

Step 1: it's not just that it requires discipline for devs. It's that you effectively leave to individual developers what may, to meet the workflow/legal/whatever reqs of your organization, need to be questions of policy.

Step 2: when you explicitly record the source values in manifest.json, you're creating a new vector for conflicts - all of your projects relying on github.com/golang/protobuf, for example, need to agree that they are sourced from artifactory.corp-intranet.com/github.com/golang/protobuf. If any don't agree, it's considered a conflict by the solver - not a version constraint conflict, but a source conflict - because how could it not be? We can't source one thing from two different places. (I described a little more how this class of conflict works in #191 (comment) - the mechanism is the same there)

This creates a potentially nasty situation: let's say your company gets acquired, and the artifactory base URL has to change from artifactory.corp-intranet.com to artifactory.newcorp-intranet.com, because 🐙. All of your projects will have to independently cut over to the new artifactory URL at the same time, roll new releases with that change, and none of them will be able to work with older versions. That's exactly the kind of high-risk change scenario that we should all be seeking to avoid.

(The source property is rife with possible outcomes like this. That's why I don't really like it that much as-is, and would ideally like to see it replaced with a few, more narrowly-defined properties that accomplish the same sort of goals.)

I'm imagining some kind of onManifestEdit hook there I can run some arbitrary script AFTER dep updates the manifest but BEFORE dep uses manifest to resolve dependencies and update the lockfile.

Yes, I've been imagining something like this since some folks described a similar need in Masterminds/glide#372. A script would probably work well; my plan was that it would take either a project root or a URL (so, github.com/golang/protobuf or https://github.com/golang/protobuf) and transform them into the URL(s) that may be used to interact with the source.

If we went that route, it would be transparent to the manifest, and the onus would be on the implementor of the script to ensure that the mirror is properly reflecting the data from upstream. Failure to do that would compromise the portability of the lock outside of the alternate sourcing universe imposed by the script...which is maybe fine in some cases, but has its risks :)

Also, we may have gone very far off topic from the original issue. Maye this should be separated into a separate issue or 2?

Sure - my thinking on these topics is all pretty jumbled up together, so if you can see a way to tease out more specific issues, please do!

Member

sdboyer commented Jan 30, 2017

The alternate urls described in dep ensure -h looks like exactly what we need. I was assuming that those alternate urls would be stored in your manifest.json individually for every dependency.

So while that leaves us with a lot of seemingly redundant text in the manifest file, it keeps any complex deduction rules out of it. Anyone would be able to pull down this project's deps from the right place assuming they had access to the intranet.

Yep yep!

That alone seems "sufficient" to solve the problem, but it requires a lot of discipline for devs to add the correct alternate location to each and every 3rd party dependency.

Oh for sure - but I'll go two steps stronger.

Step 1: it's not just that it requires discipline for devs. It's that you effectively leave to individual developers what may, to meet the workflow/legal/whatever reqs of your organization, need to be questions of policy.

Step 2: when you explicitly record the source values in manifest.json, you're creating a new vector for conflicts - all of your projects relying on github.com/golang/protobuf, for example, need to agree that they are sourced from artifactory.corp-intranet.com/github.com/golang/protobuf. If any don't agree, it's considered a conflict by the solver - not a version constraint conflict, but a source conflict - because how could it not be? We can't source one thing from two different places. (I described a little more how this class of conflict works in #191 (comment) - the mechanism is the same there)

This creates a potentially nasty situation: let's say your company gets acquired, and the artifactory base URL has to change from artifactory.corp-intranet.com to artifactory.newcorp-intranet.com, because 🐙. All of your projects will have to independently cut over to the new artifactory URL at the same time, roll new releases with that change, and none of them will be able to work with older versions. That's exactly the kind of high-risk change scenario that we should all be seeking to avoid.

(The source property is rife with possible outcomes like this. That's why I don't really like it that much as-is, and would ideally like to see it replaced with a few, more narrowly-defined properties that accomplish the same sort of goals.)

I'm imagining some kind of onManifestEdit hook there I can run some arbitrary script AFTER dep updates the manifest but BEFORE dep uses manifest to resolve dependencies and update the lockfile.

Yes, I've been imagining something like this since some folks described a similar need in Masterminds/glide#372. A script would probably work well; my plan was that it would take either a project root or a URL (so, github.com/golang/protobuf or https://github.com/golang/protobuf) and transform them into the URL(s) that may be used to interact with the source.

If we went that route, it would be transparent to the manifest, and the onus would be on the implementor of the script to ensure that the mirror is properly reflecting the data from upstream. Failure to do that would compromise the portability of the lock outside of the alternate sourcing universe imposed by the script...which is maybe fine in some cases, but has its risks :)

Also, we may have gone very far off topic from the original issue. Maye this should be separated into a separate issue or 2?

Sure - my thinking on these topics is all pretty jumbled up together, so if you can see a way to tease out more specific issues, please do!

This was referenced Feb 1, 2017

@sdboyer

This comment has been minimized.

Show comment
Hide comment
@sdboyer

sdboyer Feb 10, 2017

Member

Just bumping this to see if you've had any more thoughts on separating out some issues, @zevdg 😄

Member

sdboyer commented Feb 10, 2017

Just bumping this to see if you've had any more thoughts on separating out some issues, @zevdg 😄

@mikkeloscar

This comment has been minimized.

Show comment
Hide comment
@mikkeloscar

mikkeloscar Feb 11, 2017

Contributor

I'm glad I'm not the only one who have thought about these things :)

At my company, we solve this exact problem using go's vanity imports.

I had this thought maybe 8 months ago but went away from it for reasons I can't quite remember. I think I imagined it would be inconvenient to use a different import url. But looking at it now I see more benefits than trade-offs.

This is part of our spec (https://docs.google.com/document/d/1qnmjwfMmvSCDaY4jxPmLAccaaUI5FfySNE90gB0pTKQ/edit#). Search for "alt location".

One minor thing that this doesn't solve as far as I can tell is when you want to vendor for the first time with dep init, then you would have to run dep ensure <package> for all the dependencies instead, which is a bit annoying if you already defined them as imports in your code.

If you're working on a private, internal company system, then this sort of portability isn't really a concern. The problem with a feature like this is that it wouldn't just be used for that - how many people would write their own little custom rule for, say, gitlab (for which we don't currently have a built-in rule)?

I can understand wanting to prevent users from misusing a feature like this, but if this would be the only way to get dependencies from gitlab, then it would be better than dep not working with gitlab at all IMO (assuming that not having a built-in rule == it doesn't currently work).

Contributor

mikkeloscar commented Feb 11, 2017

I'm glad I'm not the only one who have thought about these things :)

At my company, we solve this exact problem using go's vanity imports.

I had this thought maybe 8 months ago but went away from it for reasons I can't quite remember. I think I imagined it would be inconvenient to use a different import url. But looking at it now I see more benefits than trade-offs.

This is part of our spec (https://docs.google.com/document/d/1qnmjwfMmvSCDaY4jxPmLAccaaUI5FfySNE90gB0pTKQ/edit#). Search for "alt location".

One minor thing that this doesn't solve as far as I can tell is when you want to vendor for the first time with dep init, then you would have to run dep ensure <package> for all the dependencies instead, which is a bit annoying if you already defined them as imports in your code.

If you're working on a private, internal company system, then this sort of portability isn't really a concern. The problem with a feature like this is that it wouldn't just be used for that - how many people would write their own little custom rule for, say, gitlab (for which we don't currently have a built-in rule)?

I can understand wanting to prevent users from misusing a feature like this, but if this would be the only way to get dependencies from gitlab, then it would be better than dep not working with gitlab at all IMO (assuming that not having a built-in rule == it doesn't currently work).

@zevdg

This comment has been minimized.

Show comment
Hide comment
@zevdg

zevdg Feb 22, 2017

Ok, so my main concerns around rewrite rules for alternate sources seem to be much less pressing due to the paradigm shift that seems to be happening around #213 . If the manifests will not be edited by dep, then we can just make our own manifest generation tooling that writes to the manifest and not have to worry about conflicting with or hooking into dep. If implemented, #221 could help ensure this tooling is installed correctly. The UX could be improved by making some sort of source rewrite rules a first class citizen in dep, but it's a less pressing issue now and it seems like a contentious feature that could be added in a later version of dep. I'll open a ticket for reference and discussion around this issue.

The other main issue is authentication. Even though github has the right go-import meta tags, they are not accessible for any repository (public or private) in my company's GHE necessitating our vanity import server. If I could configure dep with a github access token or with a git-like credential manager, that wouldn't be necessary. To be clear though, AFAIK, this isn't just an issue for GHE, but also for private repositories on regular old github.com We could continue to use this issue for that, but it's now pretty cluttered with other stuff so I'll open a separate issue for that as well.

zevdg commented Feb 22, 2017

Ok, so my main concerns around rewrite rules for alternate sources seem to be much less pressing due to the paradigm shift that seems to be happening around #213 . If the manifests will not be edited by dep, then we can just make our own manifest generation tooling that writes to the manifest and not have to worry about conflicting with or hooking into dep. If implemented, #221 could help ensure this tooling is installed correctly. The UX could be improved by making some sort of source rewrite rules a first class citizen in dep, but it's a less pressing issue now and it seems like a contentious feature that could be added in a later version of dep. I'll open a ticket for reference and discussion around this issue.

The other main issue is authentication. Even though github has the right go-import meta tags, they are not accessible for any repository (public or private) in my company's GHE necessitating our vanity import server. If I could configure dep with a github access token or with a git-like credential manager, that wouldn't be necessary. To be clear though, AFAIK, this isn't just an issue for GHE, but also for private repositories on regular old github.com We could continue to use this issue for that, but it's now pretty cluttered with other stuff so I'll open a separate issue for that as well.

@nathany

This comment has been minimized.

Show comment
Hide comment
@nathany

nathany May 29, 2017

Contributor

Giving dep v0.1.0 a try with an existing work project. Since the project is hosted on GitHub Enterprise, I knew it wasn't going to work perfectly given this issue and #286.

When running dep init, I see the following error:

sm.DeduceProjectRoot: unable to deduce repository and source type for 
"github.myorg.com/myorg/myrepo": unable to read metadata: go-import metadata not found

In this case myrepo is a reusable package I wrote.

There's nothing tricky about it. This is the remote for that repository:

git remote show -n origin
* remote origin
  Fetch URL: https://github.myorg.com/myorg/myrepo.git

There is a 1:1 mapping here -- it's just https://{import path}.git. However, I recognize it's not one of the "blessed" locations and authentication could be "interesting" as well.

Our previous tool was gvt. It doesn't support private repos either.

However, I was able to work around that issue by manually dumping files in vendor and not letting gvt know about them. I don't know that there is any temporary workaround with dep? In fact, dep init didn't generate the starting manifest.

Contributor

nathany commented May 29, 2017

Giving dep v0.1.0 a try with an existing work project. Since the project is hosted on GitHub Enterprise, I knew it wasn't going to work perfectly given this issue and #286.

When running dep init, I see the following error:

sm.DeduceProjectRoot: unable to deduce repository and source type for 
"github.myorg.com/myorg/myrepo": unable to read metadata: go-import metadata not found

In this case myrepo is a reusable package I wrote.

There's nothing tricky about it. This is the remote for that repository:

git remote show -n origin
* remote origin
  Fetch URL: https://github.myorg.com/myorg/myrepo.git

There is a 1:1 mapping here -- it's just https://{import path}.git. However, I recognize it's not one of the "blessed" locations and authentication could be "interesting" as well.

Our previous tool was gvt. It doesn't support private repos either.

However, I was able to work around that issue by manually dumping files in vendor and not letting gvt know about them. I don't know that there is any temporary workaround with dep? In fact, dep init didn't generate the starting manifest.

@nathany

This comment has been minimized.

Show comment
Hide comment
@nathany

nathany May 30, 2017

Contributor

Just noticed I wasn't completely honest in my last comment. There is something a bit tricky.

import "github.myorg.com/myorg/myrepo/mypacakge"

->

https://github.myorg.com/myorg/myrepo.git/mypackage

That could explain it. Of course there's no way of really knowing where the .git repository ends and the paths within it begin.

Contributor

nathany commented May 30, 2017

Just noticed I wasn't completely honest in my last comment. There is something a bit tricky.

import "github.myorg.com/myorg/myrepo/mypacakge"

->

https://github.myorg.com/myorg/myrepo.git/mypackage

That could explain it. Of course there's no way of really knowing where the .git repository ends and the paths within it begin.

@nathany

This comment has been minimized.

Show comment
Hide comment
@nathany

nathany May 31, 2017

Contributor

If I change my import paths to:

import "github.myorg.com/myorg/myrepo.git/mypacakge"

I saw the following message with dep init (dep v0.1.0 0e8607e)

https://github.myorg.com/myorg/myrepo.git does not exist in the local cache and fetching failed: context canceled
Unable to cache github.myorg.com/myorg/myrepo.git
main.getProjectData.func1
	/Users/nathany/src/github.com/golang/dep/cmd/dep/init.go:351
runtime.goexit
	/Users/nathany/src/go.googlesource.com/go/src/runtime/asm_amd64.s:2197

But then the next time I ran dep init it worked! 👍

Not a big fan of having to put .git into my import paths though. But to get around that, dep would need to know or be told that github.myorg.com/myorg/myrepo.git is the repo so that import paths below that work. And be told in such a way that dep init works, before there is a manifest file.

Contributor

nathany commented May 31, 2017

If I change my import paths to:

import "github.myorg.com/myorg/myrepo.git/mypacakge"

I saw the following message with dep init (dep v0.1.0 0e8607e)

https://github.myorg.com/myorg/myrepo.git does not exist in the local cache and fetching failed: context canceled
Unable to cache github.myorg.com/myorg/myrepo.git
main.getProjectData.func1
	/Users/nathany/src/github.com/golang/dep/cmd/dep/init.go:351
runtime.goexit
	/Users/nathany/src/go.googlesource.com/go/src/runtime/asm_amd64.s:2197

But then the next time I ran dep init it worked! 👍

Not a big fan of having to put .git into my import paths though. But to get around that, dep would need to know or be told that github.myorg.com/myorg/myrepo.git is the repo so that import paths below that work. And be told in such a way that dep init works, before there is a manifest file.

@aajtodd

This comment has been minimized.

Show comment
Hide comment
@aajtodd

aajtodd Jun 22, 2017

Looking at private mirroring of our dependencies currently.

So we have the ability to fetch a dependency from a different location via
dep ensure github.com/pkg/foo:git.internal.com/alt/foo

This seems mostly sufficient, just mirror the repositories you want to host internally and I'm assuming everything will work out fine. I have question with this workflow though regarding transitive dependencies.

Let's say we want to add go.uber.org/zap to our manifest.

dep ensure go.uber.org/zap pulls in the transitive dependency github.com/davecgh/go-spew with it and writes it to the lock file.

Let's say we know we are going to pull this dependency in and we mirror it internally.

So what should happen if we do dep ensure go.uber.org/zap:git.internal.com/alt/zap.

  1. Should transitive dependencies also be pulled from the alternative location?
  2. What is a good workflow for discovering said transitive dependencies to make sure they also get mirrored?
    • I suppose you could do a dry run of the ensure step to inspect the resulting lock file to see the transitive dependencies. Then go and make sure you mirror the dependency you care about and all of it's dependencies.

Any thoughts on what this workflow ought to look like?

aajtodd commented Jun 22, 2017

Looking at private mirroring of our dependencies currently.

So we have the ability to fetch a dependency from a different location via
dep ensure github.com/pkg/foo:git.internal.com/alt/foo

This seems mostly sufficient, just mirror the repositories you want to host internally and I'm assuming everything will work out fine. I have question with this workflow though regarding transitive dependencies.

Let's say we want to add go.uber.org/zap to our manifest.

dep ensure go.uber.org/zap pulls in the transitive dependency github.com/davecgh/go-spew with it and writes it to the lock file.

Let's say we know we are going to pull this dependency in and we mirror it internally.

So what should happen if we do dep ensure go.uber.org/zap:git.internal.com/alt/zap.

  1. Should transitive dependencies also be pulled from the alternative location?
  2. What is a good workflow for discovering said transitive dependencies to make sure they also get mirrored?
    • I suppose you could do a dry run of the ensure step to inspect the resulting lock file to see the transitive dependencies. Then go and make sure you mirror the dependency you care about and all of it's dependencies.

Any thoughts on what this workflow ought to look like?

@nathany

This comment has been minimized.

Show comment
Hide comment
@nathany

nathany Jun 22, 2017

Contributor

Good questions.

Though I'm starting to wonder if we could split up this issue? Personally I'd just like a way to support GitHub Enterprise for dependencies written within the organization.

Transitively mirroring third-party deps is absolutely an important feature, but not one I need where I'm at.

I just don't want one to slow down the implementation of the other... and it becomes a bit difficult to follow the conversation when the issue covers to many different topics.

Contributor

nathany commented Jun 22, 2017

Good questions.

Though I'm starting to wonder if we could split up this issue? Personally I'd just like a way to support GitHub Enterprise for dependencies written within the organization.

Transitively mirroring third-party deps is absolutely an important feature, but not one I need where I'm at.

I just don't want one to slow down the implementation of the other... and it becomes a bit difficult to follow the conversation when the issue covers to many different topics.

@aajtodd

This comment has been minimized.

Show comment
Hide comment
@aajtodd

aajtodd Jun 23, 2017

@nathany I'm happy to move/split however maintainers see fit. The question seemed to fit under "private repository" usage patterns to me.

As an update though on actually trying this out:

dep ensure golang.org/x/sys:git.internal.com/external/golang.org/x/sys
Did not quite work for me, I hit the following

$ dep ensure golang.org/x/sys:http://git.internal.org/external/golang.org/x/sys
ensure Solve(): unable to deduce repository and source type for "http://git.internal.org/external/golang.org/x/sys": unable to read metadata: go-import metadata not found

$ dep ensure golang.org/x/sys:http://git.internal.org/external/golang.org/x/sys.git
ensure Solve(): No versions of golang.org/x/sys met constraints:
	master: Could not introduce golang.org/x/sys (from http://git.internal.org/external/golang.org/x/sys.git)@master, as it is not allowed by constraint * from project demo.

Appears I would need to setup a page to serve the correct metadata?

Modifying Gopkg.toml directly did seem to work though:

 [[constraint]]
   branch = "master"
   name = "golang.org/x/sys"
+  source = "http://git.internal.org/external/golang.org/x/sys.git"

dep ensure -v shows it was pulled from the internal mirrored repository.

As I suspected though transitive dependencies continue to be pulled from their original location which is not what we (or presumably other organizations) would want when trying to vendor all third party dependencies. @sdboyer I am inclined to agree that perhaps this part of the issue ought to be split but I'm not particularly sure how you want it organized (or rather what even the right question ought to be).

aajtodd commented Jun 23, 2017

@nathany I'm happy to move/split however maintainers see fit. The question seemed to fit under "private repository" usage patterns to me.

As an update though on actually trying this out:

dep ensure golang.org/x/sys:git.internal.com/external/golang.org/x/sys
Did not quite work for me, I hit the following

$ dep ensure golang.org/x/sys:http://git.internal.org/external/golang.org/x/sys
ensure Solve(): unable to deduce repository and source type for "http://git.internal.org/external/golang.org/x/sys": unable to read metadata: go-import metadata not found

$ dep ensure golang.org/x/sys:http://git.internal.org/external/golang.org/x/sys.git
ensure Solve(): No versions of golang.org/x/sys met constraints:
	master: Could not introduce golang.org/x/sys (from http://git.internal.org/external/golang.org/x/sys.git)@master, as it is not allowed by constraint * from project demo.

Appears I would need to setup a page to serve the correct metadata?

Modifying Gopkg.toml directly did seem to work though:

 [[constraint]]
   branch = "master"
   name = "golang.org/x/sys"
+  source = "http://git.internal.org/external/golang.org/x/sys.git"

dep ensure -v shows it was pulled from the internal mirrored repository.

As I suspected though transitive dependencies continue to be pulled from their original location which is not what we (or presumably other organizations) would want when trying to vendor all third party dependencies. @sdboyer I am inclined to agree that perhaps this part of the issue ought to be split but I'm not particularly sure how you want it organized (or rather what even the right question ought to be).

@ibrasho

This comment has been minimized.

Show comment
Hide comment
@ibrasho

ibrasho Jun 25, 2017

Collaborator

@aajtodd Won't an [[override]] allow you to do that?

[[override]]
  branch = "master"
  name = "golang.org/x/sys"
  source = "http://git.internal.org/external/golang.org/x/sys.git"

This should ensure that golang.org/x/sys (regardless if it's direct or transitive dep) uses http://git.internal.org/external/golang.org/x/sys.git as its source.

Collaborator

ibrasho commented Jun 25, 2017

@aajtodd Won't an [[override]] allow you to do that?

[[override]]
  branch = "master"
  name = "golang.org/x/sys"
  source = "http://git.internal.org/external/golang.org/x/sys.git"

This should ensure that golang.org/x/sys (regardless if it's direct or transitive dep) uses http://git.internal.org/external/golang.org/x/sys.git as its source.

@ibrasho

This comment has been minimized.

Show comment
Hide comment
@ibrasho

ibrasho Jun 25, 2017

Collaborator

Also, the error you are getting is to be expected since dep cannot tell where the project repository exist since the URL is not serving the required go-import metadata:

<meta name="go-import" content="golang.org/x/sys git http://git.internal.org/external/golang.org/x/sys.git">

@sdboyer It seems we have more cases where people might not be able to run dep init in a current project (when it's not possible to solve with default params). Most cases require creating the manifest and filling in the required constraints or overrides. How should we deal with this?

Collaborator

ibrasho commented Jun 25, 2017

Also, the error you are getting is to be expected since dep cannot tell where the project repository exist since the URL is not serving the required go-import metadata:

<meta name="go-import" content="golang.org/x/sys git http://git.internal.org/external/golang.org/x/sys.git">

@sdboyer It seems we have more cases where people might not be able to run dep init in a current project (when it's not possible to solve with default params). Most cases require creating the manifest and filling in the required constraints or overrides. How should we deal with this?

@aajtodd

This comment has been minimized.

Show comment
Hide comment
@aajtodd

aajtodd Jun 26, 2017

@ibrasho

Won't an [[override]] allow you to do that?

I don't know haven't tried it, I will attempt to it sometime this week (admittedly golang.org/x/sys was a bad example as it has no transitive dependencies). The documentation on the manifest format and what constraints vs override behavior was not 100% clear to me.

It seems we have more cases where people might not be able to run dep init in a current project (when it's not possible to solve with default params)

I do find init behavior slightly over presumptuous. Perhaps dep init should just create the empty manifest and allow you to fill it in (this is how npm and cargo work for JS/Rust respectively). dep ensure would create the lock file if it is not present. I'm not too worried about it though, it isn't difficult to modify the manifest after the fact (although documentation on this type of workflow will certainly help).

aajtodd commented Jun 26, 2017

@ibrasho

Won't an [[override]] allow you to do that?

I don't know haven't tried it, I will attempt to it sometime this week (admittedly golang.org/x/sys was a bad example as it has no transitive dependencies). The documentation on the manifest format and what constraints vs override behavior was not 100% clear to me.

It seems we have more cases where people might not be able to run dep init in a current project (when it's not possible to solve with default params)

I do find init behavior slightly over presumptuous. Perhaps dep init should just create the empty manifest and allow you to fill it in (this is how npm and cargo work for JS/Rust respectively). dep ensure would create the lock file if it is not present. I'm not too worried about it though, it isn't difficult to modify the manifest after the fact (although documentation on this type of workflow will certainly help).

@ibrasho

This comment has been minimized.

Show comment
Hide comment
@ibrasho

ibrasho Jul 25, 2017

Collaborator

I'm under the assumption that dep ensure golang.org/x/sys:git.internal.com/external/golang.org/x/sys.git should work (note the .git at the end) even if no go-get metadata is available. Could you try that?

what constraints vs override behavior was not 100% clear to me.

Basically, [[constraint]] is used to constraint how dep determine which version to use for one of your direct dependencies. While [[override]] is used to override the constraints defined downstream for one of your transitive dependencies.

Collaborator

ibrasho commented Jul 25, 2017

I'm under the assumption that dep ensure golang.org/x/sys:git.internal.com/external/golang.org/x/sys.git should work (note the .git at the end) even if no go-get metadata is available. Could you try that?

what constraints vs override behavior was not 100% clear to me.

Basically, [[constraint]] is used to constraint how dep determine which version to use for one of your direct dependencies. While [[override]] is used to override the constraints defined downstream for one of your transitive dependencies.

@justinfx

This comment has been minimized.

Show comment
Hide comment
@justinfx

justinfx Sep 1, 2017

I've just started trying out dep in an internal project that is currently using glide. Similar to glide, I really can't use the "init" command. It causes issues between the internal gitlab server deps and the external deps in combination with auth/proxies/etc. So I accepted that and just wrote an equivalent Gopkg.toml for dep. But now I am finding that the ensure command wants to hit the "?go-get=1" http endpoints for the internal gitlab projects, even though I have specified the "source" for all of these. Why is that? I was hoping to answer the question for the tool, but now I seem to be stuck between getting it to work for the external deps vs the internal deps, but never the both.

Is there a way to specify enough information for the internal dependencies that can get this working?

justinfx commented Sep 1, 2017

I've just started trying out dep in an internal project that is currently using glide. Similar to glide, I really can't use the "init" command. It causes issues between the internal gitlab server deps and the external deps in combination with auth/proxies/etc. So I accepted that and just wrote an equivalent Gopkg.toml for dep. But now I am finding that the ensure command wants to hit the "?go-get=1" http endpoints for the internal gitlab projects, even though I have specified the "source" for all of these. Why is that? I was hoping to answer the question for the tool, but now I seem to be stuck between getting it to work for the external deps vs the internal deps, but never the both.

Is there a way to specify enough information for the internal dependencies that can get this working?

afiune added a commit to afiune/dep that referenced this issue Sep 7, 2017

How to use a Github Token using .netrc file
Adds documentation about a second option to interact with
private repositories, in particular for those who wants to use
a Github Token either locally or in a CI System.

Might help issues:
- golang#174
- golang#286

afiune added a commit to afiune/dep that referenced this issue Sep 7, 2017

How to use a Github Token in .netrc file
Adds documentation about a second option to interact with
private repositories, in particular for those who wants to use
a Github Token either locally or in a CI System.

Might help issues:
- golang#174
- golang#286

Signed-off-by: Salim Afiune <afiune@chef.io>
@wadelee1986

This comment has been minimized.

Show comment
Hide comment
@wadelee1986

wadelee1986 Oct 3, 2017

[[constraint]]
name = "golang.org/x/crypto"
branch = "master"
source = "https://github.com/golang/crypto.git"
#source = "https://github.com/golang/crypto.git"
#source = "https://github.com/myfork/package.git"
#source = "https://github.com/golang/crypto"
#source = "github.com/golang/crypto"

[[override]]
branch = "master"
name = "golang.org/x/crypto"
source = "https://github.com/golang/crypto.git"
#source = "https://github.com/golang/crypto.git"

error:

master: unable to deduce repository and source type for "golang.org/x/crypto/ocsp": unable to read metadata: unable to fetch raw metadata: failed HTTP request to URL "http://golang.org/x/crypto/ocsp?go-get=1": Get http://golang.org/x/crypto/ocsp?go-get=1: dial tcp 216.239.37.1:80: i/o timeout

who can help me ?
thank you .

wadelee1986 commented Oct 3, 2017

[[constraint]]
name = "golang.org/x/crypto"
branch = "master"
source = "https://github.com/golang/crypto.git"
#source = "https://github.com/golang/crypto.git"
#source = "https://github.com/myfork/package.git"
#source = "https://github.com/golang/crypto"
#source = "github.com/golang/crypto"

[[override]]
branch = "master"
name = "golang.org/x/crypto"
source = "https://github.com/golang/crypto.git"
#source = "https://github.com/golang/crypto.git"

error:

master: unable to deduce repository and source type for "golang.org/x/crypto/ocsp": unable to read metadata: unable to fetch raw metadata: failed HTTP request to URL "http://golang.org/x/crypto/ocsp?go-get=1": Get http://golang.org/x/crypto/ocsp?go-get=1: dial tcp 216.239.37.1:80: i/o timeout

who can help me ?
thank you .

@sagikazarmark

This comment has been minimized.

Show comment
Hide comment
@sagikazarmark

sagikazarmark Oct 3, 2017

You sure it's the correct place for that question?

sagikazarmark commented Oct 3, 2017

You sure it's the correct place for that question?

@justinfx

This comment has been minimized.

Show comment
Hide comment
@justinfx

justinfx Oct 3, 2017

@freeformz , what about my question? I have asked that question not only here, but on golang-nuts and the Go Package Management forum, with no replies. I currently can't use the dep tool at work.

justinfx commented Oct 3, 2017

@freeformz , what about my question? I have asked that question not only here, but on golang-nuts and the Go Package Management forum, with no replies. I currently can't use the dep tool at work.

@morganhein

This comment has been minimized.

Show comment
Hide comment
@morganhein

morganhein Oct 5, 2017

This may not be an option for everyone, but we just use SSH keys to authenticate, and utilize git url rewriting:

git config --global url.git://github.com/.insteadOf https://github.com/

Then dep and glide both work fine. You have to have your ssh key set up locally and on your git repo, but it's working fine atm.

morganhein commented Oct 5, 2017

This may not be an option for everyone, but we just use SSH keys to authenticate, and utilize git url rewriting:

git config --global url.git://github.com/.insteadOf https://github.com/

Then dep and glide both work fine. You have to have your ssh key set up locally and on your git repo, but it's working fine atm.

zknill added a commit to zknill/dep that referenced this issue Oct 6, 2017

How to use a Github Token in .netrc file
Adds documentation about a second option to interact with
private repositories, in particular for those who wants to use
a Github Token either locally or in a CI System.

Might help issues:
- golang#174
- golang#286

Signed-off-by: Salim Afiune <afiune@chef.io>
@sdboyer

This comment has been minimized.

Show comment
Hide comment
@sdboyer

sdboyer Oct 7, 2017

Member

@wadelee1986 not relevant to this issue - please make a new issue.

@justinfx an explanation for why adding source is not sufficient is here: #860

Member

sdboyer commented Oct 7, 2017

@wadelee1986 not relevant to this issue - please make a new issue.

@justinfx an explanation for why adding source is not sufficient is here: #860

@magicalbanana

This comment has been minimized.

Show comment
Hide comment
@magicalbanana

magicalbanana Oct 31, 2017

@morganhein but isn't git:// less secure?

magicalbanana commented Oct 31, 2017

@morganhein but isn't git:// less secure?

@nathany

This comment has been minimized.

Show comment
Hide comment
@nathany

nathany Nov 17, 2017

Contributor

Not sure about git:// but SSH to GitHub is pretty secure.

A comparison here: https://gist.github.com/grawity/4392747

Contributor

nathany commented Nov 17, 2017

Not sure about git:// but SSH to GitHub is pretty secure.

A comparison here: https://gist.github.com/grawity/4392747

@nathany

This comment has been minimized.

Show comment
Hide comment
@nathany

nathany Nov 17, 2017

Contributor

If using github.myenterprise.com, it's not entirely clear to me how Git URL rewriting would satisfy dep, go get, or other tools in the Go ecosystem.

Has anyone got this to work, and have an example to share?

https://blog.devzero.com/2014/08/29/useful-git-commands-url-rewriting/

Contributor

nathany commented Nov 17, 2017

If using github.myenterprise.com, it's not entirely clear to me how Git URL rewriting would satisfy dep, go get, or other tools in the Go ecosystem.

Has anyone got this to work, and have an example to share?

https://blog.devzero.com/2014/08/29/useful-git-commands-url-rewriting/

@mikkeloscar

This comment has been minimized.

Show comment
Hide comment
@mikkeloscar

mikkeloscar Nov 17, 2017

Contributor

@nathany The problem is most likely that your github enterprise installation doesn't expose the url https://github.myenterprise.com/org/repo?go-get=1 without authentication, so when go get or dep calls this url it will fail and not know how to fetch the repository. This is at least the problem we have with our Github Enterprise setup.

In this case go get will go to the HTTP url before calling git and thus your rewrite rule will have no effect since git is never executed.

It would be great if Github Enterprise could just expose the go metadata without protecting it behind an auth page since there are no secrets in the metadata.

Contributor

mikkeloscar commented Nov 17, 2017

@nathany The problem is most likely that your github enterprise installation doesn't expose the url https://github.myenterprise.com/org/repo?go-get=1 without authentication, so when go get or dep calls this url it will fail and not know how to fetch the repository. This is at least the problem we have with our Github Enterprise setup.

In this case go get will go to the HTTP url before calling git and thus your rewrite rule will have no effect since git is never executed.

It would be great if Github Enterprise could just expose the go metadata without protecting it behind an auth page since there are no secrets in the metadata.

@dsvensson

This comment has been minimized.

Show comment
Hide comment
@dsvensson

dsvensson Nov 17, 2017

@nathany As for other tools, Glide works correctly in this situation. You just point your dependency at any git repository and it will do the right thing, no matter what package path you use in your source code.

@mikkeloscar or if dep would just do the ssh dance like Glide does.

dsvensson commented Nov 17, 2017

@nathany As for other tools, Glide works correctly in this situation. You just point your dependency at any git repository and it will do the right thing, no matter what package path you use in your source code.

@mikkeloscar or if dep would just do the ssh dance like Glide does.

@nathany

This comment has been minimized.

Show comment
Hide comment
@nathany

nathany Nov 21, 2017

Contributor

It would be nice if dep did the ssh dance 💃 🕺 @dsvensson mentioned, or if GitHub implemented what @mikkeloscar suggests.

I'd rather not move work projects from gvt to glide to dep -- and I don't imagine my co-workers would be super happy about switching tools multiple times either.

Plus I'd like to get more experience using dep in "real world" scenarios.

Contributor

nathany commented Nov 21, 2017

It would be nice if dep did the ssh dance 💃 🕺 @dsvensson mentioned, or if GitHub implemented what @mikkeloscar suggests.

I'd rather not move work projects from gvt to glide to dep -- and I don't imagine my co-workers would be super happy about switching tools multiple times either.

Plus I'd like to get more experience using dep in "real world" scenarios.

@sdboyer

This comment has been minimized.

Show comment
Hide comment
@sdboyer

sdboyer Nov 21, 2017

Member

this is really just about #860 - if we were to trust the source declaration fully, it would obviate the need for the HTTP go-get request.

Member

sdboyer commented Nov 21, 2017

this is really just about #860 - if we were to trust the source declaration fully, it would obviate the need for the HTTP go-get request.

@zaquestion

This comment has been minimized.

Show comment
Hide comment
@zaquestion

zaquestion Jan 2, 2018

@nathany @mikkeloscar git URL rewriting should work fine with go get and dep to clone private repos. I've used the below for the past couple years with no issues. It seems like you're actually running into http://grokbase.com/t/gg/golang-nuts/13c5jx3g79/go-nuts-problem-with-go-get-with-github-enterprise-repo perhaps you can further rewrite the URL to append the .git back on.

git config --global url."git@github.com:".insteadOf "https://github.com/"

This works because go get does use git/hg to clone repositories.
Ref: https://golang.org/doc/articles/go_command.html#tmp_3

zaquestion commented Jan 2, 2018

@nathany @mikkeloscar git URL rewriting should work fine with go get and dep to clone private repos. I've used the below for the past couple years with no issues. It seems like you're actually running into http://grokbase.com/t/gg/golang-nuts/13c5jx3g79/go-nuts-problem-with-go-get-with-github-enterprise-repo perhaps you can further rewrite the URL to append the .git back on.

git config --global url."git@github.com:".insteadOf "https://github.com/"

This works because go get does use git/hg to clone repositories.
Ref: https://golang.org/doc/articles/go_command.html#tmp_3

@mikkeloscar

This comment has been minimized.

Show comment
Hide comment
@mikkeloscar

mikkeloscar Jan 2, 2018

Contributor

It works for github.com because it's hardcoded in go get (https://github.com/golang/go/blob/9a13f8e11ce487fccef071303164b3d963e6ede6/src/cmd/go/internal/get/vcs.go#L956), it does not work for custom GH enterprise installations because go get goes to the meta-data HTTP endpoint before it calls git, and this fails when the GHE is protected by some login page.

Contributor

mikkeloscar commented Jan 2, 2018

It works for github.com because it's hardcoded in go get (https://github.com/golang/go/blob/9a13f8e11ce487fccef071303164b3d963e6ede6/src/cmd/go/internal/get/vcs.go#L956), it does not work for custom GH enterprise installations because go get goes to the meta-data HTTP endpoint before it calls git, and this fails when the GHE is protected by some login page.

@zaquestion

This comment has been minimized.

Show comment
Hide comment
@zaquestion

zaquestion Jan 2, 2018

Ha, I just noticed this myself as you posted. Here's the link in the tools repo fwiw.
https://github.com/golang/tools/blob/master/go/vcs/vcs.go#L623

Further discussion here https://groups.google.com/forum/m/#!msg/golang-nuts/AURCoVLjNyc/2Uw7A_-LRfQJ seems to imply that suffixing your imports with .git should work. The first go nuts forum I posted seemed to indicate this worked with git ls-remote

Pretty sure I use rewriting with GitLab instances which Id expect to have the same problem. Beginning to doubt that now though.

zaquestion commented Jan 2, 2018

Ha, I just noticed this myself as you posted. Here's the link in the tools repo fwiw.
https://github.com/golang/tools/blob/master/go/vcs/vcs.go#L623

Further discussion here https://groups.google.com/forum/m/#!msg/golang-nuts/AURCoVLjNyc/2Uw7A_-LRfQJ seems to imply that suffixing your imports with .git should work. The first go nuts forum I posted seemed to indicate this worked with git ls-remote

Pretty sure I use rewriting with GitLab instances which Id expect to have the same problem. Beginning to doubt that now though.

@sdboyer

This comment has been minimized.

Show comment
Hide comment
@sdboyer

sdboyer Jan 2, 2018

Member

to clarify - while the go get behavior is largely equivalent to dep's in this specific regard, go get code should not be attributed explanatory power, as dep has its own implementation and never calls go get.

Member

sdboyer commented Jan 2, 2018

to clarify - while the go get behavior is largely equivalent to dep's in this specific regard, go get code should not be attributed explanatory power, as dep has its own implementation and never calls go get.

@zaquestion

This comment has been minimized.

Show comment
Hide comment
@zaquestion

zaquestion Jan 2, 2018

Thanks for clarifying, I misunderstood your previous comment and did get that impression. Having now also read 830.

if we were to trust the source declaration fully, it would obviate the need for the HTTP go-get request.

This seems like the way to do it, the "ping" seems to create more problems while just letting the vcs error on a bad url seems manageable.

zaquestion commented Jan 2, 2018

Thanks for clarifying, I misunderstood your previous comment and did get that impression. Having now also read 830.

if we were to trust the source declaration fully, it would obviate the need for the HTTP go-get request.

This seems like the way to do it, the "ping" seems to create more problems while just letting the vcs error on a bad url seems manageable.

@amenzhinsky

This comment has been minimized.

Show comment
Hide comment
@amenzhinsky

amenzhinsky Feb 6, 2018

I've tried many workarounds but non of them worked.
I'm trying to dep ensure a package from a private gitlab repo but it's asking for username and then just gets stuck after I hit enter.

$ dep ensure
Username for 'https://git.mygitlab.com:443':

Git clone and go get work as expected.

$ dep version
dep:
 version     : v0.4.1
 build date  : 2018-01-24
 git hash    : 37d9ea0a
 go version  : go1.9.1
 go compiler : gc
 platform    : linux/amd64

amenzhinsky commented Feb 6, 2018

I've tried many workarounds but non of them worked.
I'm trying to dep ensure a package from a private gitlab repo but it's asking for username and then just gets stuck after I hit enter.

$ dep ensure
Username for 'https://git.mygitlab.com:443':

Git clone and go get work as expected.

$ dep version
dep:
 version     : v0.4.1
 build date  : 2018-01-24
 git hash    : 37d9ea0a
 go version  : go1.9.1
 go compiler : gc
 platform    : linux/amd64
@outcoldman

This comment has been minimized.

Show comment
Hide comment
@outcoldman

outcoldman Feb 25, 2018

I believe I have a solution for private repos with ssh access, see PR #1717

outcoldman commented Feb 25, 2018

I believe I have a solution for private repos with ssh access, see PR #1717

@jan-g jan-g referenced a pull request that will close this issue Mar 18, 2018

Open

External file-based registry #1759

@jan-g

This comment has been minimized.

Show comment
Hide comment
@jan-g

jan-g Mar 18, 2018

#1759 - this is a quick sketch of a local registry implementation that doesn't require the editing of any import paths. Let me know if this is useful.

jan-g commented Mar 18, 2018

#1759 - this is a quick sketch of a local registry implementation that doesn't require the editing of any import paths. Let me know if this is useful.

@chehsunliu

This comment has been minimized.

Show comment
Hide comment
@chehsunliu

chehsunliu Aug 4, 2018

This is the workaround in our team:

  1. Make Dep fetch dependencies via SSH:

    $ git config --global url."git@github.mycompany.com:".insteadOf "https://github.mycompany.com/"
  2. Add the .git suffix for every related project in code:

    package xxx
    
    import (
        "github.mycompany.com/chehsunliu/example.git/pkg/calc"
    )
    
    func DoSomething(a, b int) int {
        return calc.Add(a, b)
    }

If one project is likely to be imported, clone it with the .gitsuffix from the beginning.

$ git clone git@github.mycompany.com:chehsunliu/example.git $GOPATH/src/github.mycompany.com/chehsunliu/example.git

Otherwise, it would be quite painful to rename every import statement in the project.

chehsunliu commented Aug 4, 2018

This is the workaround in our team:

  1. Make Dep fetch dependencies via SSH:

    $ git config --global url."git@github.mycompany.com:".insteadOf "https://github.mycompany.com/"
  2. Add the .git suffix for every related project in code:

    package xxx
    
    import (
        "github.mycompany.com/chehsunliu/example.git/pkg/calc"
    )
    
    func DoSomething(a, b int) int {
        return calc.Add(a, b)
    }

If one project is likely to be imported, clone it with the .gitsuffix from the beginning.

$ git clone git@github.mycompany.com:chehsunliu/example.git $GOPATH/src/github.mycompany.com/chehsunliu/example.git

Otherwise, it would be quite painful to rename every import statement in the project.

@EugenMayer

This comment has been minimized.

Show comment
Hide comment
@EugenMayer

EugenMayer Sep 8, 2018

A little sidenote for clarifcation / addition:

meaning of .git and where it is mandatory
you can either go the way of @chehsunliu which has one good point to it, you do not longer need to define constraints manually in this manner:

[[constraint]]
   name=code.mycompany.com/whatever/myproject
   source=ssh://git@mycompany.com/whatever/myproject.git

So the important one here is, you do not need .git in the package name, you need it in the source. Since @chehsunliu is already importing using .git, this constrain to "remap" name to source is no longer needed.

This said, you could also import the package without .git and do the constraint above .. but this would need a constrain per internal project to remap you non .git import to a .git based repo source

global git url rewrite
You do not need the global configuration @chehsunliu was proposing unless you want to go with "auto imports without constraints", so if you want to skip the constraint definition and want dep to auto-resolve it, you have to import with .git and also remap the non-scheme based import to a ssh based source, since by default, dep will use https://<yourimport .. so if you want to use ssh whatever, you remap https://mycompany.com to git@mycompany.com

You can fix a lot more with the git url rewrites .. even paths and path remapping if the anchor of your repos e.g. is /scm/whatever/project you could remap that https://mycompany.com/whatever/project . from the import will be sourced from git@mycompany/scm/whatever/project and so on. This helps for go projects where you remap / need to fix it / have legacy or when your SCM server has odd paths and you start to golang :) thats why i mention it

git urls
git@mycompany.com my be not enough when you run a non default ssh port, in this case you do
ssh://gitolite@mycompany.com:3302 - this would be needed in the global rewrite or in the constraint - whatever you stick with

EugenMayer commented Sep 8, 2018

A little sidenote for clarifcation / addition:

meaning of .git and where it is mandatory
you can either go the way of @chehsunliu which has one good point to it, you do not longer need to define constraints manually in this manner:

[[constraint]]
   name=code.mycompany.com/whatever/myproject
   source=ssh://git@mycompany.com/whatever/myproject.git

So the important one here is, you do not need .git in the package name, you need it in the source. Since @chehsunliu is already importing using .git, this constrain to "remap" name to source is no longer needed.

This said, you could also import the package without .git and do the constraint above .. but this would need a constrain per internal project to remap you non .git import to a .git based repo source

global git url rewrite
You do not need the global configuration @chehsunliu was proposing unless you want to go with "auto imports without constraints", so if you want to skip the constraint definition and want dep to auto-resolve it, you have to import with .git and also remap the non-scheme based import to a ssh based source, since by default, dep will use https://<yourimport .. so if you want to use ssh whatever, you remap https://mycompany.com to git@mycompany.com

You can fix a lot more with the git url rewrites .. even paths and path remapping if the anchor of your repos e.g. is /scm/whatever/project you could remap that https://mycompany.com/whatever/project . from the import will be sourced from git@mycompany/scm/whatever/project and so on. This helps for go projects where you remap / need to fix it / have legacy or when your SCM server has odd paths and you start to golang :) thats why i mention it

git urls
git@mycompany.com my be not enough when you run a non default ssh port, in this case you do
ssh://gitolite@mycompany.com:3302 - this would be needed in the global rewrite or in the constraint - whatever you stick with

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment