Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support SSH Git URLs for authenticated connections to private repositories #1851

Open
jnicholls opened this issue Jul 28, 2015 · 35 comments

Comments

@jnicholls
Copy link

commented Jul 28, 2015

Right now there is no way for cargo to connect to a private repository, whether or HTTPS or SSH. I believe if the git Cargo.toml dependency option would support a non-URL string such as git@github.com:user/repo then the Git client in cargo would likely successfully connect.

This is a major hole in cargo right now; I'm forced to submodule everything which is tedious.

Thanks in advance.

@alexcrichton

This comment has been minimized.

Copy link
Member

commented Jul 28, 2015

This is actually supported, Cargo just doesn't prompt for a password (see #1306). You can use git credentials to store information (which Cargo reads).

@jnicholls

This comment has been minimized.

Copy link
Author

commented Jul 28, 2015

I've tried this before only a few weeks ago and it was not working as
advertised, despite that issue having been closed and me running the latest
cargo. I'll try again.

On Tuesday, July 28, 2015, Alex Crichton notifications@github.com wrote:

This is actually supported, Cargo just doesn't prompt for a password (see
#1306 #1306). You can use git
credentials http://git-scm.com/docs/gitcredentials to store information
(which Cargo reads).


Reply to this email directly or view it on GitHub
#1851 (comment).

Sent from Gmail Mobile

@alexcrichton

This comment has been minimized.

Copy link
Member

commented Jul 28, 2015

Ok, but if you run into any problems feel free to open an issue! It's a difficult code path to test and it's not exercised that much, so there may be a bug or two lurking.

@jnicholls

This comment has been minimized.

Copy link
Author

commented Jul 31, 2015

This is in fact an issue. I have my osxkeychain git credential helper setup, and my credentials are cached. All fresh terminals authenticate seamlessly to HTTPS GitHub URLs of private repositories. However, cargo continues to report Unable to update https://github.com/blah/blah both on stable and nightly cargo. The credential helper is setup in the global config (~/.gitconfig)

[credential]
        helper = osxkeychain

Please advise. Thanks in advance!

EDIT: I have also tried a file store helper, which also does not work.

[credential]
        helper = store

Do I need to setup credentials in some special way other than using a credential helper?

@alexcrichton

This comment has been minimized.

Copy link
Member

commented Jul 31, 2015

Sounds like some investigation is warranted!

@alexcrichton alexcrichton reopened this Jul 31, 2015

@jnicholls

This comment has been minimized.

Copy link
Author

commented Jul 31, 2015

As stated in the OP, SSH URLs in the form of git@server.com:user/repo are not supported by Cargo. When parsing the .toml file, it does not understand that as being a legitimate URL (because it isn't).

This is a suggestion. Apparently one can use ssh://user@server.com/repo and Cargo is happy with that, since it is in fact a legitimate URL and libgit2 understands it just fine.

@alexcrichton

This comment has been minimized.

Copy link
Member

commented Jul 31, 2015

Specifically, it sounds like you had a submodule which used a URL of the form git@example.com:user/repo and when Cargo tried to check that out it failed because it wasn't able to parse the URL and set the right credentials?

@jnicholls

This comment has been minimized.

Copy link
Author

commented Jul 31, 2015

That's correct! Whether it is specified directly as the git argument of a [dependency] entry, or if it's present as a submodule url in .gitmodules, it will not work.

@simon-nicholls

This comment has been minimized.

Copy link

commented Dec 5, 2015

I actually thought this was working until I switched locations.

I set up a project with a git ssh://Si@blah.org:/repo/blah.git dependency whilst on OSX, and thought it was picking up my .ssh config just fine - certainly my id_rsa, but perhaps also a custom ssh port number.

Now that I'm on Linux, the same project ignores all .ssh config for git dependencies, though it does work when including the password, port number, and what I had for breakfast in the URL.

@jnicholls

This comment has been minimized.

Copy link
Author

commented Dec 5, 2015

Are you running through vagrant or something? In either case, is ssh-agent
running?

I've had no problems on OS X or Linux on vagrant with ssh-agent enabled in
my Vagrantfile.

On Saturday, December 5, 2015, Si notifications@github.com wrote:

I actually thought this was working until I switched locations.

I set up a project with a git ssh://Si@blah.org:/repo/blah.git dependency
whilst on OSX, and thought it was picking up my .ssh config just fine -
certainly my id_rsa, but perhaps also a custom ssh port number.

Now that I'm on Linux, the same project ignores all .ssh config for git
dependencies, though it does work when including the password, port number,
and what I had for breakfast in the URL.


Reply to this email directly or view it on GitHub
#1851 (comment).

Sent from Gmail Mobile

@simon-nicholls

This comment has been minimized.

Copy link

commented Dec 5, 2015

Physical machines, unfortunately in different countries.

My best guess is that ssh-agent was running on the Mac without my realising it. As soon as I use ssh-agent here on Linux, it works as expected. I still need to be explicit about the port, but that may also have been the case on the Mac. The repo is just a personal one, so I was relying on ssh auth negotiation beforehand.

@alexcrichton

This comment has been minimized.

Copy link
Member

commented Dec 5, 2015

@simon-nicholls you may also be running into #2078, right now Cargo doesn't take a look at .ssh/config

@simon-nicholls

This comment has been minimized.

Copy link

commented Dec 5, 2015

Thanks. That clears things up for me.

@Binero

This comment has been minimized.

Copy link

commented Apr 5, 2016

I would like to bump this issue. I am trying to connect to a remote, private ssh repository on Gitlab. I change the scp syntax git@gitlab.com:project/supercool.git into ssh://gitlab.com/project/supercool.git. When running cargo update it gives me an error saying it cannot authenticate. I do have the key inside my ~/.ssh, and it is registered using ssh-add. It is also part of my GNOME Keyring, which my git credential helper is set to.

I can clone the repository manually, as well as ssh into the server.

@jnicholls

This comment has been minimized.

Copy link
Author

commented Apr 6, 2016

@Binero change your URL to ssh://git@gitlab.com/project/supercool.git (notice the git@ user addition) and try again?

@Binero

This comment has been minimized.

Copy link

commented Apr 6, 2016

@jnicholls Gold. Worked nicely.

@mkollaro

This comment has been minimized.

Copy link

commented Sep 1, 2016

I'm having a similar issue:

reponame = { git = "ssh://git@gitlab.com:22/mkollaro/reponame.git" }

but cargo fails with:

 $ cargo test --verbose
    Updating git repository `ssh://git@gitlab.com:22/mkollaro/reponame.git`
error: Unable to update ssh://git@gitlab.com:22/mkollaro/reponame.git

Caused by:
  failed to fetch into /home/mkollaro/.cargo/git/db/reponame-e51b056051cf84
Caused by:
  failed to authenticate when downloading repository
attempted ssh-agent authentication, but none of the usernames `git` succeeded

Caused by:
  [23/-1] error authenticating: no auth sock variable

Doing a git clone with the very same URL works fine. I'm using cargo 0.12.0-nightly (6b98d1f 2016-07-04).

Btw, I had to add an explicit port number because it's complaining about an invalid port number without it - maybe another bug?

@alexcrichton

This comment has been minimized.

Copy link
Member

commented Sep 1, 2016

@mkollaro that actually looks like it's a different error perhaps?

I believe git clone will transitively invoke ssh itself which will read ~/.ssh/config for keys and such. Cargo, however, which uses libgit2 which uses libssh2, will not read ~/.ssh/config yet (not implemented). The only authentication method method supported by Cargo right now is connecting to the ssh-agent. Do you have the agent running with your keys added?

@Zteve

This comment has been minimized.

Copy link

commented Mar 23, 2017

On Mac OS X (Sierra) I had to create a .ssh/config file like this:

Host *
   UseKeychain yes
   AddKeysToAgent yes
   IdentityFile ~/.ssh/id_rsa

with the (private) rsa file pointed to, and then issue the command:

ssh-add -K ~/.ssh/id_rsa

which (finally!) allowed an entry like:

git = "ssh://git@github.com/skipjaq/loda.git"

to work perfectly. (Now I only have to cure the coding bugs.)

I do not know how often I will have to repeat the ssh-add command, but it appears this ought to hold at least until the next reboot, which (on MAC at least) is pretty rare.

This anomaly is apparently a feature of ssh-agent on Mac OS X Sierra.

@pronebird

This comment has been minimized.

Copy link

commented Jul 31, 2017

@Zteve I am seeing quite weird behavior when https://github.com/username/XXXX.git rejects to work, while switching prefix to ssh:// makes it work.

@Zteve

This comment has been minimized.

Copy link

commented Aug 1, 2017

@pronebird Yes, the scheme https: fails for me also. I can only get it to work if I use the ssh: scheme with the git@ prefix on the host part.

Have you tried it with https://git@github.com/username/XXXX.git? I haven't.

@pronebird

This comment has been minimized.

Copy link

commented Aug 1, 2017

@Zteve tried with https://git@... but unfortunately it doesn't work. Only ssh:// works for private repos.

@alexcrichton

This comment has been minimized.

Copy link
Member

commented Aug 1, 2017

@pronebird using the https URL requires password authentication which Cargo doesn't implement. Using the ssh URL uses ssh-agent authentication which is implemented.

@Zteve

This comment has been minimized.

Copy link

commented Aug 1, 2017

@pronebird OK. Thanks for trying. Incidentally, I've just rebooted (for a macOS update) and didn't have to re-issue the ssh-add command. This had been necessary after previous reboots.

Thanks for the clarification @alexcrichton .

@emk

This comment has been minimized.

Copy link

commented Aug 8, 2017

Thank you for showing how to make this work!

Here are some rambling thoughts on use cases...

My employer already has a heavy investment in Ruby and Node.js, and we're rapidly expanding our use of Rust. This issue and #3917, unfortunately, mean that the Rust workflow for private repositories is clunkier than for Ruby or Node, and this will require some modifications throughout our development, CI and deployment systems.

In a Ruby Gemfile, we can write:

gem 'my_gem', :git => 'git@github.com:<company>/<repo>.git'

In a Node package.json file, we can write:

"package-name": "git+ssh://git@github.com:<company>/<repo>.git"

Alternatively, we can run private package server or subscribe to a hosted one, such as the one provided by npm. (The relevant Cargo issue for that is #3917.) Normally, we lean towards a private package server and we only use private git URLs when working with unreleased branches.

From a security perspective, we're very careful about long-lived credentials on developer machines. We use 2FA heavily and issue temporary credentials for as many services as possible. We do assume that developers have GitHub SSH keys, but I don't know if we assume ssh-agent is set up correctly.

It looks like our only real choice here is to make ssh-agent work for everybody, including the CI system (ugh). It would be very nice if Cargo could read ~/.ssh/config, which would avoid the need for booting up an ssh-agent inside a Docker container on the CI server. Alternatively, it would be marvelous to have the ability to host our own crates.io server for private packages.

But right now, these limitations of cargo seems like a major speedbumps for Rust adoption inside companies, at least for anything that extends beyond a single private repository.

@alexcrichton

This comment has been minimized.

Copy link
Member

commented Aug 9, 2017

Thanks for the comment @emk! Some points of note:

  • It looks like node's and Ruby's support for SSH urls is exactly the same as Cargo's, modulo a few details. A repository fetched over SSH just needs to use a URL like ssh://github.com/company/repo.git. Cargo doesn't currently support git@github.com:... style urls (that's what this issue is).

  • The authentication is then the next phase, and Cargo primarily supports ssh-agent which works through libssh2. What doesn't work is that Cargo doesn't read .ssh/config. My guess is that npm and rubygems literally shell out to ssh, whereas Cargo uses a bundled libssh2 library.

  • Private registries are definitely desired! That's the purpose of #3917.

@emk

This comment has been minimized.

Copy link

commented Aug 9, 2017

OK, here are my notes on using private Git repos with cargo and our CI system.

So far, this is mostly a list of things which don't work, in case somebody else needs to make private crates work with a CI system.

Attempt 1: ssh-agent authentication

  • Figure out who has the 2FA tokens for GitHub deployment account with read-only access.
  • Create new SSH keys and add them to the GitHub deployment account.
  • Store SSH keys in our secret management service.
  • Try to fetch secrets from secret management service at build time and pass to cargo via ssh-agent inside docker build. FAILED because passing secrets to docker build is actually pretty terrible. We have some workarounds for this with other package managers, but it assumes that we have a working private package repository available over HTTP (details elided).

Attempt 2: cargo vendor

  • Run cargo vendor in hopes of selectively vendoring just the private sub-module. FAILED because cargo vendor vendors all the publicly available packages from crates.io, but not anything from GitHub. So it's basically the opposite of what we need.

OK, for attempt 3, I'm going to read about [source.*] overrides and see if I can make them work with GitHub.

@emk

This comment has been minimized.

Copy link

commented Aug 9, 2017

@alexcrichton As far as I know, both bundler and npm allow https URLs with embedded passwords, which makes it pretty easy to use GitHub deployment tokens. These aren't ideal from a security perspective, but they're a lot easier than getting ssh-agent running inside a docker build command on a CI system.

There relevant section of our Dockerfile looks like:

WORKDIR /app
ADD ./ /app
RUN cargo build --release

Trying to get ssh-agent running in this context is moderately annoying, but not impossible.

I think the ideal outcome here would be:

  1. Implement #3917 (but that's a bigger, longer-term project).
  2. Support the same kinds of URLs and configuration npm and bundler. Specifically, read .ssh/config and support HTTPS with username + password. This would allow existing workarounds for Ruby or Node to be re-used with cargo with less fuss.

Oh, and:

Attempt 3: Try to manually vendor the private crate

After reading the docs, it looks like this really only works for crates from crates.io. I don't want to replace crates.io, I just want to override a single crate.

At this point, I think I may need to write a helper utility for my CI system which manually checks out private repositories and provides an override. The ergonomics for this are fairly painful, and it seems like "private Rust crates" would be a common use-case for companies adopting Rust.

Anyway, thank you for your advice! I'll try to figure something out.

(EDIT: I'm going to write this up as either a separate issue or a blog post, but the TL;DR is that private crate dependencies are surprisingly hard to get working with a build system, and that most of the cargo features that look like they might help have surprising limitations, such as only working for crates published to crates.io.)

@Boscop

This comment has been minimized.

Copy link

commented Oct 25, 2017

I tried to use my_crate = { git = 'ssh://git@foo.com:my-repo' } but it said invalid port number.
Then I tried my_crate = { git = 'ssh://git@foo.com/my-repo' } but it said authentication failure, even though it's NOT an auth failure.
Turns out I had to write my_crate = { git = 'ssh://git@foo.com/~/my-repo' } which is not obvious. It should give a more helpful error with /, not auth failure.

Btw, why doesn't my_crate = { git = 'https://github.com/user/my-repo' } work for private repos anymore?

@emk

This comment has been minimized.

Copy link

commented Oct 25, 2017

OK, just a quick follow up: We do finally have private git repositories working with a credential helper. In practice, it's a significant nuisance to set up across developer machines and CI, but once it's automated, it actually works fairly well.

Thank you to the cargo team for explaining this!

@aturon

This comment has been minimized.

Copy link
Member

commented Jan 22, 2018

The Cargo team discussed this issue briefly in our meeting today, and we'd be happy to take a PR for it!

@akshayknarayan

This comment has been minimized.

Copy link

commented May 1, 2018

Cargo doesn't currently support git@github.com:... style urls (that's what this issue is).

This is an issue for repositories that include submodules. A .gitmodules written by git submodule add will look like this: url = git@github.com:akshayknarayan/test-submodule.

Then, attempting to build a crate with the following dependency:

[dependencies]
cargo-submodules-test = { git = "ssh://git@github.com/akshayknarayan/cargo-submodules-test" }

will give the following error message:

⇒  cargo b
    Updating git repository `ssh://git@github.com/akshayknarayan/cargo-submodules-test`
error: failed to load source for a dependency on `cargo-submodules-test`

Caused by:
  Unable to update ssh://git@github.com/akshayknarayan/cargo-submodules-test

Caused by:
  failed to update submodule `test-submodule`

Caused by:
  invalid url `git@github.com:akshayknarayan/test-submodule`: relative URL without a base
@kenhuang

This comment has been minimized.

Copy link

commented Dec 27, 2018

ssh-agent method seems not working on windows, there is my steps. Similar method works fine on Mac and Linux.

  1. install git for windows
$ git --version
git version 2.20.1.windows.1
  1. use default git bash, generate ssh key
  2. empty config for ~/.gitconfig
  3. have the following for ~/.ssh/config
Host *
   UseKeychain yes
   AddKeysToAgent yes
   IdentityFile ~/.ssh/id_rsa
  1. lauch ssh agent
eval `ssh-agent -s`
ssh-add

ssh-add -L showing key was added
  1. cargo build with error
...
Caused by:
  Unable to update ssh://git@bitbucket.org/xxx/xxx.git#964db79b

Caused by:
  failed to fetch into C:\Users\xxx\.cargo\git\db\substrate-79104a8293a5645e

Caused by:
  failed to authenticate when downloading repository
attempted ssh-agent authentication, but none of the usernames `git` succeeded

Caused by:
  error authenticating: failed connecting agent; class=Ssh (23)

Any suggestion? Thanks for your help!

@kenhuang

This comment has been minimized.

Copy link

commented Dec 27, 2018

able to get this working on windows10 by update ~/.cargo/config

 [net]
 git-fetch-with-cli = true
@mouse07410

This comment has been minimized.

Copy link

commented Mar 5, 2019

Here's what worked for me on MacOS Mojave.

  1. URL in Cargo.toml should be in form of ssh://git@github.server.com/my-user-id/repo
  2. You must configure your Enterprise GitHub for SSH access and upload the right keys
  3. Get ssh-agent running
  4. Add your private key (corresponding to the uploaded SSH public key) via
    ssh-add ~/.ssh/my-private-key
    note: flag -K is no longer accepted by ssh-add. Correction: MacOS-included ssh-add does support -K, but the mainstream ssh-add installed by Macports - does not.

Then Cargo was able to pull the crate from there.

It worked with and without the following addition to the ~/.cargo/config:

[net]
git-fetch-with-cli = true

but I put it in, so it will probably stay there, unless I find a reason to remote it.

I also had

[credential "https://github.server.com"] 
         username = my-user-id
         helper = store

added to ~/.gitconfig, but don't know how relevant it was.

CruizeMissile added a commit to CruizeMissile/repo that referenced this issue Jun 4, 2019

add use_cli to project and settings
- add --cli and --no-cli flags to edit and add commands
- git functions now switch on using cli
- Note: rust-lang/cargo#1851 is the refernece
  I used to make sure that my ssh urls can convert into Url's
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.