Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support SSH Git URLs for authenticated connections to private repositories #1851

Open
jnicholls opened this issue Jul 28, 2015 · 58 comments
Open

Comments

@jnicholls
Copy link

@jnicholls jnicholls commented Jul 28, 2015

Right now there is no way for cargo to connect to a private repository, whether or HTTPS or SSH. I believe if the git Cargo.toml dependency option would support a non-URL string such as git@github.com:user/repo then the Git client in cargo would likely successfully connect.

This is a major hole in cargo right now; I'm forced to submodule everything which is tedious.

Thanks in advance.

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Jul 28, 2015

This is actually supported, Cargo just doesn't prompt for a password (see #1306). You can use git credentials to store information (which Cargo reads).

@jnicholls
Copy link
Author

@jnicholls jnicholls commented Jul 28, 2015

I've tried this before only a few weeks ago and it was not working as
advertised, despite that issue having been closed and me running the latest
cargo. I'll try again.

On Tuesday, July 28, 2015, Alex Crichton notifications@github.com wrote:

This is actually supported, Cargo just doesn't prompt for a password (see
#1306 #1306). You can use git
credentials http://git-scm.com/docs/gitcredentials to store information
(which Cargo reads).


Reply to this email directly or view it on GitHub
#1851 (comment).

Sent from Gmail Mobile

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Jul 28, 2015

Ok, but if you run into any problems feel free to open an issue! It's a difficult code path to test and it's not exercised that much, so there may be a bug or two lurking.

@jnicholls
Copy link
Author

@jnicholls jnicholls commented Jul 31, 2015

This is in fact an issue. I have my osxkeychain git credential helper setup, and my credentials are cached. All fresh terminals authenticate seamlessly to HTTPS GitHub URLs of private repositories. However, cargo continues to report Unable to update https://github.com/blah/blah both on stable and nightly cargo. The credential helper is setup in the global config (~/.gitconfig)

[credential]
        helper = osxkeychain

Please advise. Thanks in advance!

EDIT: I have also tried a file store helper, which also does not work.

[credential]
        helper = store

Do I need to setup credentials in some special way other than using a credential helper?

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Jul 31, 2015

Sounds like some investigation is warranted!

@alexcrichton alexcrichton reopened this Jul 31, 2015
@jnicholls
Copy link
Author

@jnicholls jnicholls commented Jul 31, 2015

As stated in the OP, SSH URLs in the form of git@server.com:user/repo are not supported by Cargo. When parsing the .toml file, it does not understand that as being a legitimate URL (because it isn't).

This is a suggestion. Apparently one can use ssh://user@server.com/repo and Cargo is happy with that, since it is in fact a legitimate URL and libgit2 understands it just fine.

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Jul 31, 2015

Specifically, it sounds like you had a submodule which used a URL of the form git@example.com:user/repo and when Cargo tried to check that out it failed because it wasn't able to parse the URL and set the right credentials?

@jnicholls
Copy link
Author

@jnicholls jnicholls commented Jul 31, 2015

That's correct! Whether it is specified directly as the git argument of a [dependency] entry, or if it's present as a submodule url in .gitmodules, it will not work.

@simon-nicholls
Copy link

@simon-nicholls simon-nicholls commented Dec 5, 2015

I actually thought this was working until I switched locations.

I set up a project with a git ssh://Si@blah.org:/repo/blah.git dependency whilst on OSX, and thought it was picking up my .ssh config just fine - certainly my id_rsa, but perhaps also a custom ssh port number.

Now that I'm on Linux, the same project ignores all .ssh config for git dependencies, though it does work when including the password, port number, and what I had for breakfast in the URL.

@jnicholls
Copy link
Author

@jnicholls jnicholls commented Dec 5, 2015

Are you running through vagrant or something? In either case, is ssh-agent
running?

I've had no problems on OS X or Linux on vagrant with ssh-agent enabled in
my Vagrantfile.

On Saturday, December 5, 2015, Si notifications@github.com wrote:

I actually thought this was working until I switched locations.

I set up a project with a git ssh://Si@blah.org:/repo/blah.git dependency
whilst on OSX, and thought it was picking up my .ssh config just fine -
certainly my id_rsa, but perhaps also a custom ssh port number.

Now that I'm on Linux, the same project ignores all .ssh config for git
dependencies, though it does work when including the password, port number,
and what I had for breakfast in the URL.


Reply to this email directly or view it on GitHub
#1851 (comment).

Sent from Gmail Mobile

@simon-nicholls
Copy link

@simon-nicholls simon-nicholls commented Dec 5, 2015

Physical machines, unfortunately in different countries.

My best guess is that ssh-agent was running on the Mac without my realising it. As soon as I use ssh-agent here on Linux, it works as expected. I still need to be explicit about the port, but that may also have been the case on the Mac. The repo is just a personal one, so I was relying on ssh auth negotiation beforehand.

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Dec 5, 2015

@simon-nicholls you may also be running into #2078, right now Cargo doesn't take a look at .ssh/config

@simon-nicholls
Copy link

@simon-nicholls simon-nicholls commented Dec 5, 2015

Thanks. That clears things up for me.

@Binero
Copy link

@Binero Binero commented Apr 5, 2016

I would like to bump this issue. I am trying to connect to a remote, private ssh repository on Gitlab. I change the scp syntax git@gitlab.com:project/supercool.git into ssh://gitlab.com/project/supercool.git. When running cargo update it gives me an error saying it cannot authenticate. I do have the key inside my ~/.ssh, and it is registered using ssh-add. It is also part of my GNOME Keyring, which my git credential helper is set to.

I can clone the repository manually, as well as ssh into the server.

@jnicholls
Copy link
Author

@jnicholls jnicholls commented Apr 6, 2016

@Binero change your URL to ssh://git@gitlab.com/project/supercool.git (notice the git@ user addition) and try again?

@Binero
Copy link

@Binero Binero commented Apr 6, 2016

@jnicholls Gold. Worked nicely.

@mkollaro
Copy link

@mkollaro mkollaro commented Sep 1, 2016

I'm having a similar issue:

reponame = { git = "ssh://git@gitlab.com:22/mkollaro/reponame.git" }

but cargo fails with:

 $ cargo test --verbose
    Updating git repository `ssh://git@gitlab.com:22/mkollaro/reponame.git`
error: Unable to update ssh://git@gitlab.com:22/mkollaro/reponame.git

Caused by:
  failed to fetch into /home/mkollaro/.cargo/git/db/reponame-e51b056051cf84
Caused by:
  failed to authenticate when downloading repository
attempted ssh-agent authentication, but none of the usernames `git` succeeded

Caused by:
  [23/-1] error authenticating: no auth sock variable

Doing a git clone with the very same URL works fine. I'm using cargo 0.12.0-nightly (6b98d1f 2016-07-04).

Btw, I had to add an explicit port number because it's complaining about an invalid port number without it - maybe another bug?

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Sep 1, 2016

@mkollaro that actually looks like it's a different error perhaps?

I believe git clone will transitively invoke ssh itself which will read ~/.ssh/config for keys and such. Cargo, however, which uses libgit2 which uses libssh2, will not read ~/.ssh/config yet (not implemented). The only authentication method method supported by Cargo right now is connecting to the ssh-agent. Do you have the agent running with your keys added?

@Zteve
Copy link

@Zteve Zteve commented Mar 23, 2017

On Mac OS X (Sierra) I had to create a .ssh/config file like this:

Host *
   UseKeychain yes
   AddKeysToAgent yes
   IdentityFile ~/.ssh/id_rsa

with the (private) rsa file pointed to, and then issue the command:

ssh-add -K ~/.ssh/id_rsa

which (finally!) allowed an entry like:

git = "ssh://git@github.com/skipjaq/loda.git"

to work perfectly. (Now I only have to cure the coding bugs.)

I do not know how often I will have to repeat the ssh-add command, but it appears this ought to hold at least until the next reboot, which (on MAC at least) is pretty rare.

This anomaly is apparently a feature of ssh-agent on Mac OS X Sierra.

@pronebird
Copy link

@pronebird pronebird commented Jul 31, 2017

@Zteve I am seeing quite weird behavior when https://github.com/username/XXXX.git rejects to work, while switching prefix to ssh:// makes it work.

@Zteve
Copy link

@Zteve Zteve commented Aug 1, 2017

@pronebird Yes, the scheme https: fails for me also. I can only get it to work if I use the ssh: scheme with the git@ prefix on the host part.

Have you tried it with https://git@github.com/username/XXXX.git? I haven't.

@pronebird
Copy link

@pronebird pronebird commented Aug 1, 2017

@Zteve tried with https://git@... but unfortunately it doesn't work. Only ssh:// works for private repos.

@kenhuang
Copy link

@kenhuang kenhuang commented Dec 27, 2018

able to get this working on windows10 by update ~/.cargo/config

 [net]
 git-fetch-with-cli = true

@mouse07410
Copy link

@mouse07410 mouse07410 commented Mar 5, 2019

Here's what worked for me on MacOS Mojave.

  1. URL in Cargo.toml should be in form of ssh://git@github.server.com/my-user-id/repo
  2. You must configure your Enterprise GitHub for SSH access and upload the right keys
  3. Get ssh-agent running
  4. Add your private key (corresponding to the uploaded SSH public key) via
    ssh-add ~/.ssh/my-private-key
    note: flag -K is no longer accepted by ssh-add. Correction: MacOS-included ssh-add does support -K, but the mainstream ssh-add installed by Macports - does not.

Then Cargo was able to pull the crate from there.

It worked with and without the following addition to the ~/.cargo/config:

[net]
git-fetch-with-cli = true

but I put it in, so it will probably stay there, unless I find a reason to remote it.

I also had

[credential "https://github.server.com"] 
         username = my-user-id
         helper = store

added to ~/.gitconfig, but don't know how relevant it was.

@hrydgard
Copy link

@hrydgard hrydgard commented Oct 15, 2019

@akshayknarayan 's issue above is #7202, which is fixed by #7238 . Not sure how to figure out which release that fix will be in though? Seems at least 0.40:

$ git merge-base 130e11c --is-ancestor master && echo yes || echo no
yes
$ git merge-base 130e11c --is-ancestor 0.39.0 && echo yes || echo no
no

@mathstuf
Copy link
Contributor

@mathstuf mathstuf commented Oct 15, 2019

Yes. If you go to the commit that merged the PR, GitHub will show the tags it is part of. For that PR, see 0400879. Example that shows tags (since none exist for that commit today): e853aa9

@kaimast
Copy link

@kaimast kaimast commented Feb 26, 2020

So I had an issue with using a forwarded ssh agent (through pythons paramiko SSH implementation) with Cargo.

The git-fetch-with-cli = true fixed it for me but I do not really understand why. Doesn't the git cli also use libgit2? Do the rust bindings somehow mess up environment variables?

Any hint on how to debug this would be appreciated. There's some more info on my problem here: paramiko/paramiko#1626

@sfackler
Copy link
Member

@sfackler sfackler commented Feb 26, 2020

The git cli does not use libgit2.

@senyaak
Copy link

@senyaak senyaak commented Apr 23, 2020

I could not fix the issue. So I cloned repository and used direct path in config package = { path = ... }. It is useful if you just need to install dependency once.

@rcastill
Copy link

@rcastill rcastill commented Jul 17, 2020

Related to "invalid port" error, I have urls of this style (thanks to azure devops):

git@ssh.dev.azure.com:v3/organization/project/repository

If I use that format, it fails with "relative URL without a base". If I prepend "ssh://" it fails with "invalid port" (:v3).

Of course git cloning works, so the url should be valid.

@gibfahn
Copy link

@gibfahn gibfahn commented Oct 25, 2020

Struggled with this, turns out that I needed to add my ssh keys to the ssh-agent with ssh-add (ssh-add -A worked for me).

The reason I don't need to do this for the git CLI is that it uses OpenSSH, which parses ~/.ssh/config and picks up the IdentityFile lines I have there. libssh2 doesn't, so they need to be added manually.

You can check whether you have non-default ssh keys by running:

$ grep IdentityFile ~/.ssh/config | sort -u
   IdentityFile ~/.ssh/gibfahn_id_ed25519
   IdentityFile ~/.ssh/gibfahn_id_rsa

and then add them with:

$ ssh-add ~/.ssh/gibfahn_id_ed25519 ~/.ssh/gibfahn_id_rsa

@ahmadseleem
Copy link

@ahmadseleem ahmadseleem commented Nov 23, 2020

able to get this working on windows10 by update ~/.cargo/config

 [net]
 git-fetch-with-cli = true

That's the one. Thanks!

@dpryg
Copy link

@dpryg dpryg commented Dec 2, 2020

If I prepend "ssh://" it fails with "invalid port" (:v3).

Try to replace ":" with "/". It works for gitlab.

@gibfahn
Copy link

@gibfahn gibfahn commented Dec 5, 2020

I ended up writing down all the different ways to auth and the gotchas here: https://fahn.co/posts/cargo-auth-for-private-git-repos.html

It got a bit long 😅 . Not sure if any of it is worth adding to the Cargo docs.

@xuqingkuang
Copy link

@xuqingkuang xuqingkuang commented Apr 29, 2021

It seems git changed the behavior of URL, all of upon solutions doesn't work anymore.

$ cargo install --git 'ssh://git.lab.com/rust/example-module.git'

    Updating git repository `ssh://git.lab.com/rust/example-module.git`
Enter passphrase for key '/home/xqkuang/.ssh/id_rsa': 
error: failed to fetch into: /home/xqkuang/.cargo/git/db/trpc-rust-ec60571bf845afd2

Caused by:
  process didn't exit successfully: `git fetch --force --update-head-ok 'ssh://git.lab.com/rust/example-module.git' 'refs/heads/master:refs/remotes/origin/master' 'HEAD:refs/remotes/origin/HEAD'` (exit code: 128)
  --- stderr
  fatal: remote error: Git:Project error, please check URL.

$ git fetch --force --update-head-ok 'ssh://git.lab.com/rust/example-module.git' 

'refs/heads/master:refs/remotes/origin/master' 'HEAD:refs/remotes/origin/HEAD'
Enter passphrase for key '/home/xqkuang/.ssh/id_rsa': 
fatal: remote error: Git:Project error, please check url.

But changing the url argument of git fetch to the correct URL git@git.lab.com:rust/example-module.git is working.

$ git fetch --force --update-head-ok 'git@git.lab.com:rust/example-module.git' 

'refs/heads/master:refs/remotes/origin/master' 'HEAD:refs/remotes/origin/HEAD'
Enter passphrase for key '/home/xqkuang/.ssh/id_rsa': 
remote: Finding sources: 100% (3124/3124)
remote: Total 3124 (delta 1718), reused 3055 (delta 1718)
Receiving objects: 100% (3124/3124), 567.77 KiB | 19.58 MiB/s, done.
Resolving deltas: 100% (1718/1718), done.
From git.lab.com:rust/example-module
 * [new branch]      master     -> origin/master
 * [new ref]                    -> origin/HEAD

Is it possible for the cargo to translate the URL to the correct one?

@mathstuf
Copy link
Contributor

@mathstuf mathstuf commented Apr 29, 2021

I think reporting to Git developers that something changed would be better here. Can you, perhaps, bisect where GIt changed this behavior?

@xuqingkuang
Copy link

@xuqingkuang xuqingkuang commented Apr 30, 2021

I think reporting to Git developers that something changed would be better here. Can you, perhaps, bisect where GIt changed this behavior?

The problem is git fetch supports the URL schema like git@git.lab.com:rust/example-module.git, but not the cargo schema ssh://git.lab.com/rust/example-module.git

I just think cargo should adapt the URL resolve mechanism of git, and solve the problem.

@mathstuf
Copy link
Contributor

@mathstuf mathstuf commented Apr 30, 2021

Git supports ssh:// just fine; I use it all the time. Something else is up here.

@mathstuf
Copy link
Contributor

@mathstuf mathstuf commented Apr 30, 2021

I think I see the issue. You have ssh://host/ vs. user@host:. Try ssh://user@host/ instead.

oowekyala added a commit to lf-lang/lingua-franca that referenced this issue Jul 13, 2021
Now we use an ssh connection to access the github repo.
Be aware that you need a credentials helper for this to
work as the repo is private and cargo will not prompt for
a password.

rust-lang/cargo#1851
@onatm
Copy link

@onatm onatm commented Sep 28, 2021

Note to all other hipster fish users.

it doesn't seem cargo install --git ssh://git@github.com/org/crate command work on fish. You need to switch to bash, add your ssh key and then you'll be able to download the crate

@mathstuf
Copy link
Contributor

@mathstuf mathstuf commented Sep 30, 2021

It seems very odd that the shell would affect something like that. Are there any other symptoms?

@onatm
Copy link

@onatm onatm commented Oct 13, 2021

It seems very odd that the shell would affect something like that. Are there any other symptoms?

There isn't much except the error output:

❯ cargo install --git ssh://git@github.com/xxx/xxxxx --branch main
    Updating git repository `ssh://git@github.com/xxx/xxxxx`
error: failed to fetch into: /Users/onat.mercan/.cargo/git/db/xxxxx-db26b348fdd70ca7

Caused by:
  failed to authenticate when downloading repository

  * attempted ssh-agent authentication, but no usernames succeeded: `git`

  if the git CLI succeeds then `net.git-fetch-with-cli` may help here
  https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli

Caused by:
  no authentication available

@mathstuf
Copy link
Contributor

@mathstuf mathstuf commented Oct 13, 2021

It seems like something is wrong with the libgit2 used internally. I'm not sure what it thinks using the remote url's username has to do with contacting the local ssh-agent.

The only thing fish could be doing is some magic on that argument, but it looks like it got through successfully. I would recommend some strace to see what is happening under the hood and whether any strings are being manip'd improperly.

@sfackler
Copy link
Member

@sfackler sfackler commented Oct 13, 2021

Your bash config is probably setting up an ssh-agent and your fish config is not.

@onatm
Copy link

@onatm onatm commented Oct 13, 2021

Your bash config is probably setting up an ssh-agent and your fish config is not.

It's the other way around. My bash config is not setting up ssh-agent but fish does.

The only thing fish could be doing is some magic on that argument, but it looks like it got through successfully. I would recommend some strace to see what is happening under the hood and whether any strings are being manip'd improperly.

I am on Mac and I'll try to figure out if I could get any meaningful trace with dtruss. I tried verbose display option to get more info from cargo but the output is the same. I'll also try to debug a local cargo version.

@sfackler
Copy link
Member

@sfackler sfackler commented Oct 13, 2021

If one shell has an ssh-agent and the other doesn't, that is the difference. It is not related to fish or bash specifically.

@onatm
Copy link

@onatm onatm commented Oct 13, 2021

I found out the reason for the failure. It is due to missing identity on ssh-agent. I am not sure how I managed to clone and work on private repos so far without having an identity on ssh-agent. Thank you @sfackler and @mathstuf

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet