Send authentication credentials to private registry to fetch config.json

[cpaika]

Problem

Currently, cargo will always send an unauthenticated request to a private registry to initially fetch the config.json. When it receives a 401, it will then retry fetching the config.json including the authentication headers the second time.

However Artifactory (a common private registry) will rate limit users based on a 401 - so in my case, if we have two jobs running with the same user, and they both fetch the config.json, and both receive a 401 - that user will be rate limited in Artifactory due to too many 401's, which then causes further Artifactory cargo fetches to fail.

Steps

1. Setup a artifactory registry with a cargo repository
2. Authenticate with the cargo repository
3. Run multiple curl requests against the cargo repository to simulate multiple cargo build running simultaneously:
   seq inf | xargs -n1 -P10 -I{} curl "https://username@private-registry.com/index/config.json"

You will observe 401's, and then 403's as your user is rate limited.

Possible Solution(s)

Users have authentication setup for private registry already - can there be a flag in your authentication to always send the auth header while fetching the config.json? That way you can avoid triggering 401's and user lockout. This also improves the user experience, since we're not wasting a round trip on a 401 when auth is already configured for a private repository.

Notes

No response

Version

cargo 1.76.0 (c84b36747 2024-01-18)
release: 1.76.0
commit-hash: c84b367471a2db61d2c2c6aab605b14130b8a31b
commit-date: 2024-01-18
host: aarch64-apple-darwin
libgit2: 1.7.1 (sys:0.18.1 vendored)
libcurl: 8.4.0 (sys:0.4.70+curl-8.5.0 system ssl:(SecureTransport) LibreSSL/3.3.6)
ssl: OpenSSL 1.1.1w  11 Sep 2023
os: Mac OS 14.2.1 [64-bit]

[cpaika]

For anyone seeing this issue in the future there are three workarounds:

1. Remove the rate limit at the artifactory level
2. Send a curl command to fetch the config.json from your artifact repository (including auth headers).
   • Store that config.json in the cargo cache folder $CARGO_HOME/registry/cache//config.json
   • When you run cargo it will see the cached credentials and skip the initial unauthenticated request
3. Disable authentication on search - this is the recommended JFrog approach at the moment: https://jfrog.com/help/r/jfrog-artifactory-documentation/resolve-cargo-packages-using-the-rust-command-line

[rohansx]

@cpaika I would like to solve this bug

[weihanglo]

@rohansx wroth noting that this one doesn't have S-accepted or S-needs-mentor label. It means that the Cargo team hasn't yet triaged it.

[rohansx]

@weihanglo Oh i see.. thanks for clarifying!

[weihanglo]

@arlosi you might be interested in this

[arlosi]

Cargo already caches the config.json to minimize the number of HTTP requests that get an HTTP 401.

Since the request is unauthenticated, it shouldn't be possible to link it to a specific user account to rate-limit.

• If the user account is being identified via the URL alone, then this is an Artifactory DOS vulnerability. Sending two unauthenticated requests at the same time shouldn't be able to lock-out a user.
• If this is an IP-based rate limit, the limit seems overly strict. One unauthenticated HTTP request per session run seems reasonable, especially since the request does not contain credentials.

I'm going to close this since it seems like an Artifactory issue. If Cargo is observed sending more than one uncached, unauthenticated request, please re-open.

cc @nadav-y

[cpaika]

IIRC the username is required in the URL in order to request the config.json at all, which then creates the lock-out problem. Agreed this is an Artifactory issue since this behavior opens them up to DOS, if they allowed complete anonymous download of just the config.json that would fix this issue.