Renovate is unable to pull Cargo artifact over ssh in Github

[vmaerten]

What would you like help with?

I think I found a bug

How are you running Renovate?

Mend Renovate hosted app on github.com

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

No response

Please tell us more about your question or problem

Hi !

Since #24704 has been merged, we could pull private crate with Renovate
But it stopped working, we run Renovate one time a month, last run was the 1th may and it worked.

Example :
First line 26th april = OK , second line 2nd of may = Artifact issues

I've opened a new discussion but we are running with the same setup as for this issue : #26269

Both repo are private, in the same orgs, both are managed by Renovate (so the GITHUB_TOKEN should be good)

I can see a new way to pass secret in the UI, maybe should I put a token here ?

Thanks a lot for the tool!

Let me know if you need more information or if I can help

Logs (if relevant)

Logs

{
  "artifactErrors": [
    {
      "lockFile": "code/rust/Cargo.lock",
      "stderr": "Command failed: cargo update --config net.git-fetch-with-cli=true --manifest-path code/rust/mithrandir/Cargo.toml --workspace\n    Updating git repository `ssh://**redacted**@github.com/pbstck/aws-lib.git`\nerror: failed to get `aws-lib` as a dependency of package `mithrandir v0.1.0 (/tmp/renovate/repos/github/pbstck/intake/code/rust/mithrandir)`\n\nCaused by:\n  failed to load source for dependency `aws-lib`\n\nCaused by:\n  Unable to update ssh://**redacted**@github.com/pbstck/aws-lib.git?tag=v1.20240517.1535#1a3ce817\n\nCaused by:\n  failed to clone into: /home/ubuntu/.cargo/git/db/aws-lib-66fd83fbc17bc4b6\n\nCaused by:\n  failed to authenticate when downloading repository\n\n  * attempted ssh-agent authentication, but no usernames succeeded: `git`\n\n  if the git CLI succeeds then `net.git-fetch-with-cli` may help here\n  https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli\n\nCaused by:\n  no authentication methods succeeded\n"
    }
  ]
}

[rarkins]

If you can create a new org for this reproduction with two private repos which reproduces the problem then it would help a lot (assuming you can grant access to myself and any other which needs).

We might also be able to roll back to an old version of Renovate to see if there's any change in behavior.

Here is the list of env from your logs:

        "GIT_CONFIG_KEY_0": "url.https://**redacted**@github.com/.insteadOf",
        "GIT_CONFIG_VALUE_0": "ssh://**redacted**@github.com/",
        "GIT_CONFIG_KEY_1": "url.https://**redacted**@github.com/.insteadOf",
        "GIT_CONFIG_VALUE_1": "git@github.com:",
        "GIT_CONFIG_KEY_2": "url.https://**redacted**@github.com/.insteadOf",
        "GIT_CONFIG_VALUE_2": "https://github.com/",

I would have assumed that "GIT_CONFIG_VALUE_0": "ssh://**redacted**@github.com/" is "GIT_CONFIG_VALUE_0": "ssh://git@github.com/", but if so then I would have assumed it to work.

[vmaerten]

I've created an org with two repos (vmaerten-renovate) :

• private-lib : a private crate, managed by renovate, which will be included in other project
• private-project: simple rust project that include tracing (public and needed to be updated) and private-lib

You can see there is an error in the cargo update (in the logs and/or in the PR)

I also forked renovate to try it locally. So i've setup another project : private-project2 (same as private-project, but I did not want polluate the first one).

I've made some test, I don't know if there are relevant :

• Create a "classic token" and run : COREPACK_ENABLE_STRICT=0 LOG_LEVEL=debug pn start --token=<TOKEN> --recreate-when=always vmaerten-renovate/private-project2 = works as expected because my token has access permissions to all repos
• Create a fine grained token (PAT, beta), with access to only private-project2, and run COREPACK_ENABLE_STRICT=0 LOG_LEVEL=debug pn start --token=<TOKEN> --recreate-when=always vmaerten-renovate/private-project2 = Does not work but it's excepted as the token does not have permission to access to private-lib
• Create a fine grained token (PAT, beta), with access to private-project2 and private-lib, and run COREPACK_ENABLE_STRICT=0 LOG_LEVEL=debug pn start --token=<TOKEN> --recreate-when=always vmaerten-renovate/private-project2 = Works as expected. I think this is the configuration I expect to have with the Github Apps

My final thought is it could be a token issue. There is a way to get the token and/or see the token permissions ?

I've added you as an admin in the org, feel free to add whoever you want. You can also modify the configuration of any project as you like.
Do not hesite if I can help to troubleshoot this.
Thanks a lot

[rarkins]

@nabeelsaabna the debugging above implies that the problem is with the app token permissions, so I'm adding you in.

On the other hand the failing PR has this:

Caused by:
  failed to authenticate when downloading repository

  * attempted ssh-agent authentication, but no usernames succeeded: `git`

  if the git CLI succeeds then `net.git-fetch-with-cli` may help here
  https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli

And this implies that it tried using ssh auth, not https. @vmaerten instead of running locally, which could then use your personal git/ssh credentials, can you run the same test using the Renovate image? e.g. docker run --rm -e ...... renovate/renovate vmaerten-renovate/private-project2

[vmaerten]

Hi @rarkins,

I managed to (almost) figure out what happens here. I reproduced with either the Docker run or with the --binary-mode=docker
I tried to rollback Renovate, it was not working either.

I figure out that rust 1.78.0 was released the 2nd of May (which was in the timeline issue). I added :

 "constraints": {
    "rust": "1.77.2"
  }

In the config.
And It works. I do not have the time to troubleshoot why this implementation does not work with the latest Cargo version right now but I'll give a try later.

I wanted to explain you, so if you have other questions like that, maybe the issue is with Cargo itself

Thanks a lot for your patience and your help.

[rarkins]

Nothing is jumping out at me from the changelog in https://github.com/rust-lang/cargo/blob/master/CHANGELOG.md#cargo-178-2024-05-02

FYI @epage @Turbo87

[Turbo87]

hmm, I don't see anything in that changelog either, but @epage is more familiar with cargo itself and should hopefully be able to help :)

[vmaerten]

The only thing I can see is those two updates :

[epage]

gix isn't used yet for fetching and cargo itself doesn't deal with this much, so its most likely in the git2 upgrade, more specifically libgit2. We upgraded git2 from 0.18.1 to 0.18.2.

Looking at git2 0.18.2, it does seem like the only relevant change is from libgit2 1.7.1 to 1.7.2
https://github.com/rust-lang/git2-rs/blob/master/CHANGELOG.md#0182---2024-02-06

Even looking at libgit2 1.7.2 doesn't seem too relevant
https://github.com/libgit2/libgit2/releases/tag/v1.7.2

I'm also not seeing changes in future releases that would imply a regression was introduced in 1.7.2
https://github.com/libgit2/libgit2/releases

We don't have bindings for 1.8.0 or 1.8.1 yet so we can't test to see if one of those fixed it.

Has someone tried setting net.git-fetch-with-cli? That can help narrow down the root cause as well.