Alternative Registries Stabilization Proposal

[ehuss]

@rfcbot fcp merge

This issue is to track the proposal to stabilize alternative registries.

RFC: 2141
Tracking Issue: rust-lang/rust#44931
Stabilization Target: 1.34 — Release date Apr 11 2019

What is being stabilized

Alternate registries allows Cargo to use registries other than crates.io. The following is an overview of the surface changes:

• Alternate registries are defined in .cargo/config under the registries table, example:

  [registries]
  my-registry = { index = "https://my-intranet:8080/git/index" }

• Dependencies in Cargo.toml can reference other registries with the registry key, example:

  [dependencies]
  other-crate = { version = "1.0", registry = "my-registry"}

• Commands may take the --registry flag to specify the server to interact with. This includes install, login, owner, publish, search, and yank. new and init also take the same flag to initialize the publish field. Example:

  cargo publish --registry my-registry

  • The default for --registry may be specified in .cargo/config with the registry.default key (except new and init, see cargo publish should be opt-in #6153). Example:

  [registry]
  default = "my-registry"

• Manifests may restrict which registry they may interact with the publish key, example:

  [package]
  publish = ["my-registry"]

• Authentication information saved by cargo login is stored in .cargo/credentials under the registries table:

  [registries.my-registry]
  token = "854DvwSlUwEHtIo3kWy6x7UCPKHfzCmy"

• Ancillary feature support:

  • cargo metadata exposes the registry field for dependency declarations. source and PackageID fields also point to the registry.
  • [patch] tables can be a registry name or index URL.
  • Registries may opt-out of Web API support, in which case commands like publish display an appropriate error message.

• Misc post-RFC decisions:

  • Registries as URLs in Cargo.toml are currently not allowed.
  • Registry names are restricted to the same format as package names (alphanumeric characters, dash, and underscore only).
  • Registry URLs in [registries] may contain usernames, but not passwords.
  • The crates.io codebase won't currently cater towards being used for alternative registries. Cargo refuses to publish alternative dependencies to crates.io, and crates.io double-checks to prevent it.
  • generate-index-metadata will currently not be implemented directly in Cargo. cargo-index is a demonstration of how it can be implemented using cargo metadata.
  • cargo-reg is a third-party util to assist adding registries to .cargo/config.

• Documentation: I have written a new chapter for the Cargo Reference that goes into depth about registries. This includes how to use them, how to run one, the index format, and the minimum required Web API. I will be posting this soon.

[sfackler]

We've been using an internal registry for over a year without issue (cough __CARGO_TEST_CHANNEL_OVERRIDE_DO_NOT_USE_THIS cough), so I'm really happy to see this land!

[ehuss]

@rfcbot merge

[rfcbot]

Team member @ehuss has proposed to merge this. The next step is review by the rest of the tagged team members:

• @Eh2406
• @alexcrichton
• @dwijnand
• @ehuss
• @joshtriplett
• @nrc
• @withoutboats
• @wycats

Concerns:

• registry-and-registries-key resolved by Alternative Registries Stabilization Proposal #6589 (comment)

Once a majority of reviewers approve (and none object), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[joshtriplett]

s/other-create/other-crate/

I'd ideally like to see the new documentation before we merge this.

[Centril]

So ostensibly, this will make it easier to hold large amounts of code in companies privately. That's fine...

However, it seems to me that it will negatively impact the library, compiler, and language teams abilities to perform crater runs to ascertain the scope of breakages. iirc, @pietroalbini had some ideas about letting private registries run crater on their machines and report back to us when we need it... However, they also note that this required a huge amount of implementation work.

Thus, my question... What impact will this have on semver and our ability to fix soundness holes, fix other bugs, make type inference changes, or add trait methods?

Should this be delayed until the necessary capabilities in crater are developed? Should we instead state points of information wrt. private registries and our language semver bugs (e.g. "its your problem if you have a private registry and we have no ability to crater...")?

cc @rust-lang/lang @rust-lang/compiler

[pietroalbini]

Tons of companies write private code right now, and that code is already out of reach for Crater. I don't think there are companies out there that care about their code being secret while hosting it on crates.io because there is no alternate registries support.

[sfackler]

Private code is private regardless of if it's published in a private registry or not. I don't understand what this has to do with crater's inherent inability to find all Rust code that exists in the world.

[cramertj]

+1-- not all Rust code in the world lives on crates.io, nor will it, nor should it. We can't prevent people from having code that needs to be private, and it's a reality that "crater run and fix all the breakages" isn't sufficient to ensure that no code is broken.

[Eh2406]

How about this (only a somewhat tongue and cheek): "If you, a big company, want Crater to be run on your private code, then higher someone to add that functionality to Crater. The community will be excited to explore how to make this possible. If you don't want to support the work then live with the standard level of support or make your code publick."

From what I have heard about this feature I am excited to see it stabilized, but as someone who has not used it I would like to look over the docks before I check my box.

[cramertj]

No, it's not acceptable to ask private companies to work with the crater team to set up an endpoint for testing in order to guarantee that the stability guarantees Rust provides continue to be upheld. This is all entirely orthogonal to this issue in any case-- private code already exists without custom registries and it will continue to exist in the future.

[ehuss]

I have posted a preview of some of the documentation at #6594.

[dwijnand]

If my understanding of the RFC procedure is correct, with one more sign off from either @nrc, @withoutboats, or @wycats this would go into final comment period.

[withoutboats]

@rfcbot concern registry-and-registries-key

I don't love that we are adding a key called "registry" and a key called "registries." Is there a better syntax for specifying the default registry we could add? Until I saw the stabilization post I thought the registry.default syntax was already stable, if thats not true I think a different syntax would be preferable.

[ehuss]

The [registry] table already exists for the token and index keys. For reference the discussion of where to set the default is here.

Do you have a preference for what it should look like? I don't have any strong preferences. The default could be moved to [registries.my-registry] default = true. Or is having both [registry] and [registries] the main objection? It would be possible to use [registry.my-registry] instead of the plural form. You just wouldn't be able to name your registry token or index, and would make it more difficult to add global [registry] keys in the future (since the namespace would overlap). Cargo isn't really consistent about table names being singular or plural.

[sfackler]

The registry vs registries dichotomy came up in this PR: #4839. I can't remember the exact issues, but I think it has to do with preserving backwards compatibility with the existing config layout and dealing with a separate credentials file in a reasonable way.

[withoutboats]

Thanks for refreshing my memory: the reason i hought registry.defaultpre-existed was I remembered registry.index. Since it predates this feature and we can't change it, I don't think its worthwhile to try to do something different here.

@rfcbot resolve registry-and-registries-key

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

psst @ehuss, I wasn't able to add the final-comment-period label, please do so.

[alexcrichton]

We're not quite done yet, but I've gone ahead and r+'d #6654 and we can always revert if the next few days have any blocking concerns come up!

[Stargateur]

This doesn't allow to use a private git index, I try to put the token in the url but

error: invalid index URL for registry `ktra` defined in ~\.cargo\config.toml

Caused by:
  registry URLs may not contain passwords

AFAIK there is no way to do this.

[Stargateur]

According to the RFC it should be possible to use credenticials file like:

[registries.my-registry]
username = "myusername"
password = "mypassword"

Unfortunately it's not documented in https://doc.rust-lang.org/cargo/reference/registries.html#using-an-alternate-registry and doesn't look to handle auth2 authentication (what use gitlab token)

[Stargateur]

I make it work with a lot of search, gitlab is lacking of documentation about how to use token see https://gitlab.com/gitlab-org/gitlab/-/issues/335869 but I manage to make it work using the username of the "account" gitlab automatically create for the token feature and the token as password so in credenticials:

[registries.ktra]
token = "SECRET" # ktra account
username = "project_42_bot1337" # user auto create by gitlab
password = "VERYSECRET" # token of gitlab

So to use token it's require to know how the git server expect you to use the giving token, I guess cargo could also have auth2 feature added in the future.

[Stargateur]

nevermind it was my git config:

[credential]
        helper = store

that allow to use a private repository not cargo credentials.