Add post to call for testing of Cargo's -Z sparse-registry

[arlosi]

We've recently landed all the components necessary to test accessing crates.io over http in Cargo. This blog post should help get more testing of the feature.

[Eh2406]

Looks good thank you!
We should get Cargo and crates.io approval.

[Mark-Simulacrum]

This should probably go on the main blog, as it's explicitly written with an external audience in mind -- we want the general Rust user population to test, right?

[eminence]

Maybe also include a link to https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#http-registry (which has links to the RFC and tracking issue) ?

[arlosi]

This should probably go on the main blog

Makes sense. The other recent crates.io-related post was on the main blog: https://blog.rust-lang.org/2022/02/14/crates-io-snapshot-branches.html I'll move it.

[jsha]

Some feedback as a potential tester: I'd like to turn this on in CI for my projects. But in CI (GitHub Actions), I use the "matrix" feature to select different versions (stable, beta, nightly, MSRV). I'm currently trying to figure out a tidy way to add -Zhttp-registry to my cargo invocations only when building with nightly. I suspect others will be in the same boat. I'll probably figure out a way to do this with shell scripting, but if there's a handy one-liner it might be useful to share in this post.

[Eh2406]

From some quick testing if you set it as an environment variable CARGO_UNSTABLE_HTTP_REGISTRY=true then nightly uses the new behavior and cargo stable does not complain. This is a significant enough improvement I would recommend it as the way to test.

env is more ergonomic then .cargo/config.toml then -Zhttp-registry IMO.

[weihanglo]

Looks great!

Might not be needed, but do we want to collect data? For power users, we may provide a simple guide in order to collect a more structural data for further performance/bug analysis.

[jsha]

From some quick testing if you set it as an environment variable CARGO_UNSTABLE_HTTP_REGISTRY=true then nightly uses the new behavior and cargo stable does not complain.

Great trick, thanks! I incorporated it in rustls/rustls-ffi#266, and now my nightly job is ~2 min vs ~4 min for stable.

[Eh2406]

If this is going to be a big deal blog post, we should probably provide a timeline and thank the people involved. This has been in the works for over 2.5 years. @kornelski worked very hard on the RFC and initial performance proof of concept. @jonhoo did the first impl for Cargo, and dealt with over 160 comments on his PR. @arlosi has already made 5 significant PR's to Cargo to get us here, and also did mager work on the crates.io implementation, and worked tirelessly to push this over the finish line! Also thanks to the crates.io and infra teams, this couldn't have been done without your help!
(I know I am forgetting people, please remind me!)

[kornelski]

🚲🏠 Is it possible to call the new index, and all options related to it, a sparse index? There's git over HTTP. And if ever this index format is replaced, the next one may use HTTP transport too, so http-registry is a generic name.

[arlosi]

@jsha, the name of the unstable flag is changing, so going forward you'll want to use CARGO_UNSTABLE_SPARSE_REGISTRY=true
cc rustls/rustls-ffi#266

[arlosi]

nightly-2022-06-20 has the renamed flag 🎉. I've updated the post with the new nightly date.

[Eh2406]

I think this is looking good. Thank you!
What are the next steps? How do we get this published?

[arlosi]

Should I go ahead and publish this to the internals forum?

Then we can start getting initial feedback, and I can update the [internals] link in the post.

[Eh2406]

I updated the testing instructions in the tracking issue to include the more articulate form you created for this blog post. I think were ready for an internals post!

[arlosi]

Posted to internals: https://internals.rust-lang.org/t/call-for-testing-cargo-sparse-registry/16862

[joshtriplett]

Looks good. We should wait to merge this until we're ready for the influx of testing, but I'm happy to merge this at any time.

[rylev]

Looks good. The date of the post needs to be changed to reflect today (or whatever date you want to publish). Once that's done, we can merge.

[arlosi]

Thanks everyone for all the review and getting this post ready! @rylev date is updated to today.

[rylev]

🎉

[rylev]

@arlosi @joshtriplett I just wanted to triple check that you are ready for the feedback, and so you would like this to be merged.

[Eh2406]

Feedback on internals is suggesting that the version date is confusing given that the cargo and rustc dates do not match.

[arlosi]

Let's wait until we do another Cargo update in the rust repo. Then we can give a nightly date that works for both rustup and cargo.

Edit decided to update the post instead.

[Eh2406]

I think we are ready!

[Eh2406]

The longer we leave this open, the more things we will find to tinker with.
However, this is more than good enough to release.

[arlosi]

Agreed. I think we're good to release.