feat(crates-io): expose HTTP headers and Error type #12310

weihanglo · 2023-06-25T15:20:51Z

What does this PR try to resolve?

This is part of #11521.

RFC 3231 mentions the authentication process could have an additional challenge-response mechanism to prevent potential replay attacks. The challenge usually comes from HTTP www-authenticate header as a opaque string. When a client gets a 401/403 response with such a challenge, it may attach the challenge to the payload and request again to anwser the challenge.

➡️ cargo  requests
⬅️ server responds with `www-authenticate` containing some opaque challenge string
➡️ cargo  automatically requests again without any user perception
⬅️ server responds ok

However, crates-io crate doesn't expose HTTP headers. There is no access to www-authenticate header.

This PR make it expose HTTP headers and the custom Error type, so cargo can access and do further on the authentication process.

How should we test and review this PR?

By commit.

This removes the dependency anyhow and uses our own custom Error enum, so that crates-io consumer can access Error::API::challenge field. It optionally uses thiserror to reduce boilerplates but this part can be dropped if we don't want.

rustbot · 2023-06-25T15:20:56Z

r? @ehuss

(rustbot has picked a reviewer for you, use r? to override)

weihanglo · 2023-06-25T15:21:07Z

cc @Eh2406

crates/crates-io/lib.rs

weihanglo · 2023-06-27T18:57:26Z

Cargo.lock

 "percent-encoding",
 "serde",
 "serde_json",
+ "thiserror",


thiserror is already in the dependency graph. Just do we want it to appear as a direct dependency or not.

weihanglo · 2023-06-27T19:02:22Z

crates/crates-io/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "crates-io"
-version = "0.36.1"
+version = "0.37.0"


Side note: the version bump check in CI didn't work, since the new version in beta hasn't yet published. But in reality, the script should treat any version in beta as “published”.

crates/crates-io/lib.rs

Eh2406 · 2023-06-29T16:07:54Z

This looks good to me. @arlosi, what are your thoughts?

arlosi · 2023-06-30T21:20:58Z

Will this be supported for sparse registry (fetching metadata) as well?
How should this interact with credential providers? The way I've currently implemented it is that we'll pass the whole www-authenticate header to the provider. The credential provider would then re-parse the header to extract the nonce. Is that acceptable? Or should we do something different?

weihanglo · 2023-06-30T23:08:25Z

Will this be supported for sparse registry (fetching metadata) as well?

I don't think so. At least not at this moment? This PR doesn't implement the flow mentioned above. It just pokes the return type of crates-io crate. crates-io is only for requesting registry API. It doesn't do anything more than that. But for supporting sparse registry, maybe yes in the future.

BTW, in this PR from_www_authenticate_values() is something shared with sparse registry.

How should this interact with credential providers? The way I've currently implemented it is that we'll pass the whole www-authenticate header to the provider. The credential provider would then re-parse the header to extract the nonce. Is that acceptable? Or should we do something different?

Sorry I am forgetting the details of the fantastic overhaul of credential providers, could you share the doc or the working branch again? BTW if cargo unifies the interface by providing the Challenge struct, is that a bit more convenience for credential provider implementor?

Eh2406 · 2023-07-05T20:49:43Z

Will this be supported for sparse registry (fetching metadata) as well?

According to the asymmetric tokens RFC as written, yes. (Which is not to say that it will happen in this PR.) Of course, if we discover during implementation that this is a bad idea... Implementation and RFC do not have to match. I would expect that only the most paranoid registries will intentionally use a challenge for read/index endpoints. On the other hand, implementing auth would only be made more complicated by requiring registries to have a different scheme for GET vs PUSH.

How should this interact with credential providers? The way I've currently implemented it is that we'll pass the whole www-authenticate header to the provider. The credential provider would then re-parse the header to extract the nonce. Is that acceptable? Or should we do something different?

Passing along the entire header allows registries to use standards cargo doesn't know about, or invent their own. It seems like the most flexible way to provide the data. It is entirely compatible with cargo passing the data it parsed in separate fields for credential providers that are using asymmetric tokens and do not want to do their own parsing.

weihanglo · 2023-07-05T21:32:59Z

Passing along the entire header allows registries to use standards cargo doesn't know about, or invent their own. It seems like the most flexible way to provide the data. It is entirely compatible with cargo passing the data it parsed in separate fields for credential providers that are using asymmetric tokens and do not want to do their own parsing.

Does this imply that we should just add a new field www_authenticate or even a simple header on ResponseError::Api? Then we can move header handling into cargo lib.

Eh2406 · 2023-07-05T21:35:52Z

That seams reasonable.

weihanglo · 2023-07-06T13:42:56Z

@Eh2406 thanks for the reply. This is now ready for another review!

ehuss · 2023-07-08T17:02:03Z

@Eh2406 Do you want to take assignment for this review? I'm happy to do a final review if you want.

Eh2406 · 2023-07-18T21:06:54Z

Sorry for the delay. This looks good. Happy to have Eric do the honours. (We need to keep the API brake in mind.)

In response to RFC 3231 [^1], our registry client need to return headers to caller, so that the caller (cargo binary) can continue parsing challenge headers. [^1]: https://rust-lang.github.io/rfcs/3231-cargo-asymmetric-tokens.html#the-authentication-process

From https://github.com/rust-lang/cargo/blob/5febbe5587b74108165f748e79a4f8badbdf5e0e/src/cargo/sources/registry/http_remote.rs#L234-L237

This removes the dependency `anyhow` and uses our own custom Error enum, so that crates-io consumer can access `Error::API::challenge` field.

Optionally use `thiserror` to reduce boilerplates but this part can be dropped if we don't want.

weihanglo · 2023-07-18T21:24:14Z

Thanks for the reminder! Bumped to 0.38.0.

ehuss · 2023-07-20T22:19:04Z

@bors r=Eh2406

bors · 2023-07-20T22:19:06Z

📌 Commit 31b500c has been approved by Eh2406

It is now in the queue for this repository.

bors · 2023-07-20T22:19:12Z

⌛ Testing commit 31b500c with merge b40be8b...

bors · 2023-07-20T23:17:47Z

☀️ Test successful - checks-actions
Approved by: Eh2406
Pushing b40be8b to master...

Update cargo 8 commits in 1b15556767f4b78a64e868eedf4073c423f02b93..7ac9416d82cd4fc5e707c9ec3574d22dff6466e5 2023-07-18 14:44:47 +0000 to 2023-07-24 14:29:38 +0000 - fix(cargo-credential): should enable feature `serde/derive` (rust-lang/cargo#12396) - fix: encode URL params correctly for SourceId in Cargo.lock (rust-lang/cargo#12280) - docs: format config override caveat as a note (rust-lang/cargo#12392) - credential provider implementation (rust-lang/cargo#12334) - feat(crates-io): expose HTTP headers and Error type (rust-lang/cargo#12310) - chore: Don't update test data (rust-lang/cargo#12380) - fix: only skip mtime check on `~/.cargo/{git,registry}` (rust-lang/cargo#12369) - Update docs for artifact JSON debuginfo levels. (rust-lang/cargo#12376) Since rust-lang/cargo#12334 makes built-in credential providers part of the cargo binary, it's no longer needed to build them in bootstrap.

rustbot assigned ehuss Jun 25, 2023

rustbot added A-interacts-with-crates.io Area: interaction with registries A-registries Area: registries A-sparse-registry Area: http sparse registries S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 25, 2023

weihanglo commented Jun 25, 2023

View reviewed changes

crates/crates-io/lib.rs Outdated Show resolved Hide resolved

crates/crates-io/lib.rs Show resolved Hide resolved

weihanglo changed the title ~~feat(crates-io) Expose www-authenticate challenge~~ feat(crates-io): expose Cargo challenge from www-authenticate header Jun 25, 2023

weihanglo marked this pull request as ready for review June 27, 2023 10:42

weihanglo commented Jun 27, 2023

View reviewed changes

Eh2406 reviewed Jun 27, 2023

View reviewed changes

crates/crates-io/lib.rs Outdated Show resolved Hide resolved

weihanglo force-pushed the asymmetric-challenge branch from 397b41d to 53521cd Compare June 28, 2023 08:25

weihanglo force-pushed the asymmetric-challenge branch from 53521cd to 966a89b Compare July 6, 2023 13:35

weihanglo changed the title ~~feat(crates-io): expose Cargo challenge from www-authenticate header~~ feat(crates-io): expose HTTP headers and Error type Jul 6, 2023

weihanglo removed the A-sparse-registry Area: http sparse registries label Jul 6, 2023

This was referenced Jul 9, 2023

Tracking Issue for alternative registry authentication (RFC 3139) #10474

Closed

version-bump check doesn't know a subcrate hasn't bumped on nightly if we've bumped it on beta #12347

Closed

weihanglo added 2 commits July 18, 2023 22:22

fix(crates-io): don't let server sneak extra lines anywhere

00a8727

From https://github.com/rust-lang/cargo/blob/5febbe5587b74108165f748e79a4f8badbdf5e0e/src/cargo/sources/registry/http_remote.rs#L234-L237

weihanglo added 4 commits July 18, 2023 22:22

refactor(crates-io): rename ResponseError to Error

ed6692b

feat(crates-io): use our own Result wrapper everywhere

a561e8b

refactor(crates-io): remove anyhow

b4499af

This removes the dependency `anyhow` and uses our own custom Error enum, so that crates-io consumer can access `Error::API::challenge` field.

refactor(crates-io): use thiserror

31b500c

Optionally use `thiserror` to reduce boilerplates but this part can be dropped if we don't want.

weihanglo force-pushed the asymmetric-challenge branch from 966a89b to 31b500c Compare July 18, 2023 21:23

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 20, 2023

bors merged commit b40be8b into rust-lang:master Jul 20, 2023
21 checks passed

This was referenced Jul 20, 2023

credential provider implementation #12334

Merged

fix: Set MSRV for internal packages #12381

Merged

arlosi mentioned this pull request Jul 24, 2023

Update cargo rust-lang/rust#114027

Merged

ehuss added this to the 1.73.0 milestone Jul 30, 2023

weihanglo deleted the asymmetric-challenge branch August 3, 2023 13:15

weihanglo mentioned this pull request Nov 27, 2023

refactor: use custom error instead of anyhow #13050

Merged

weihanglo mentioned this pull request Dec 19, 2023

refactor: custom error types for cargo-util-schemas #13186

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(crates-io): expose HTTP headers and Error type #12310

feat(crates-io): expose HTTP headers and Error type #12310

weihanglo commented Jun 25, 2023 •

edited

rustbot commented Jun 25, 2023

weihanglo commented Jun 25, 2023

weihanglo Jun 27, 2023

weihanglo Jun 27, 2023

Eh2406 commented Jun 29, 2023

arlosi commented Jun 30, 2023

weihanglo commented Jun 30, 2023 •

edited

Eh2406 commented Jul 5, 2023

weihanglo commented Jul 5, 2023

Eh2406 commented Jul 5, 2023

weihanglo commented Jul 6, 2023

ehuss commented Jul 8, 2023

Eh2406 commented Jul 18, 2023

weihanglo commented Jul 18, 2023

ehuss commented Jul 20, 2023

bors commented Jul 20, 2023

bors commented Jul 20, 2023

bors commented Jul 20, 2023

feat(crates-io): expose HTTP headers and Error type #12310

feat(crates-io): expose HTTP headers and Error type #12310

Conversation

weihanglo commented Jun 25, 2023 • edited

What does this PR try to resolve?

How should we test and review this PR?

rustbot commented Jun 25, 2023

weihanglo commented Jun 25, 2023

weihanglo Jun 27, 2023

Choose a reason for hiding this comment

weihanglo Jun 27, 2023

Choose a reason for hiding this comment

Eh2406 commented Jun 29, 2023

arlosi commented Jun 30, 2023

weihanglo commented Jun 30, 2023 • edited

Eh2406 commented Jul 5, 2023

weihanglo commented Jul 5, 2023

Eh2406 commented Jul 5, 2023

weihanglo commented Jul 6, 2023

ehuss commented Jul 8, 2023

Eh2406 commented Jul 18, 2023

weihanglo commented Jul 18, 2023

ehuss commented Jul 20, 2023

bors commented Jul 20, 2023

bors commented Jul 20, 2023

bors commented Jul 20, 2023

weihanglo commented Jun 25, 2023 •

edited

weihanglo commented Jun 30, 2023 •

edited