RFC for support of alternate registries in cargo

[cswindle]

This RFC proposes adding support for alternative crates.io servers which could be used alongside the public crates.io server.

Rendered

[Mark-Simulacrum]

Could you change the RFC file name so every line is "new"? Otherwise it's a little hard to read in the diff format.

[Mark-Simulacrum]

Could you change the RFC file name so every line is "new"? Otherwise it's a little hard to read in the diff format.

[cswindle]

@Mark-Simulacrum, I have sorted that now.

[cswindle]

@Mark-Simulacrum, I have sorted that now.

[Phrohdoh]

In my opinion (please don't take this as a personal slight) you as a user do not want to (and should not have to) define which registry a dependency comes from (for the common case) so I feel this RFC should be expanded a bit.

IMO ideally you would do something like this in a global location (such as ~/.cargo/config):

[registry.'crates-io']
url = "..."
token = "..."

[registry.'companyname-private']
url = "..." # This could be a network-internal URL or a hosted instance
token = "..."

If the above is done and a registry is not explicitly provided for the your-private-package dep then, after not finding your package on crates.io (if you have it setup as a registry) we could get the niceness of:

$ cargo build
Downloading your-private-package 0.1.7 found in 'companyname-private' registry
   Building your-private-package 0.1.7

But if a registry is provided (using the named registry (or a url if we don't go with named registries)):

[dependencies]
your-private-package = { version = "=0.1.7", registry = "companyname-private" }

$ cargo build
Downloading your-private-package 0.1.7
   Building your-private-package 0.1.7

Sorry I feel I have somewhat derailed this and have attempted to push it in the direction I think is best, but I do think we're at a point of making an important decision.

Unfortunately I don't have much time this morning to discuss this but should be available on IRC (#rust, maybe #rust-infra and #cargo) closer to 6pm UTC-5.

[Phrohdoh]

In my opinion (please don't take this as a personal slight) you as a user do not want to (and should not have to) define which registry a dependency comes from (for the common case) so I feel this RFC should be expanded a bit.

IMO ideally you would do something like this in a global location (such as ~/.cargo/config):

[registry.'crates-io']
url = "..."
token = "..."

[registry.'companyname-private']
url = "..." # This could be a network-internal URL or a hosted instance
token = "..."

If the above is done and a registry is not explicitly provided for the your-private-package dep then, after not finding your package on crates.io (if you have it setup as a registry) we could get the niceness of:

$ cargo build
Downloading your-private-package 0.1.7 found in 'companyname-private' registry
   Building your-private-package 0.1.7

But if a registry is provided (using the named registry (or a url if we don't go with named registries)):

[dependencies]
your-private-package = { version = "=0.1.7", registry = "companyname-private" }

$ cargo build
Downloading your-private-package 0.1.7
   Building your-private-package 0.1.7

Sorry I feel I have somewhat derailed this and have attempted to push it in the direction I think is best, but I do think we're at a point of making an important decision.

Unfortunately I don't have much time this morning to discuss this but should be available on IRC (#rust, maybe #rust-infra and #cargo) closer to 6pm UTC-5.

[cswindle]

@Phrohdoh, thanks for the feedback on the RFC. I do however have concerns about your approaches, I feel that if you are wanting to use an alternate registry for crates then you would want to explicitly specify that in the dependencies, without that I think you would run into problems where a crate with the same name happens to exist in both registries and cargo may chose the wrong version to use.

I can recognise some use of having a global config file defining the URL that you want to use for the crates, but I think that the registry to use belongs in the crate that you are defining, not in the separate config on the users machine.

I am going to update the RFC to add that an alternate crates.io registry would be updated to include the registry in the fragment of Cargo.toml displayed for each package, thus allowing users just to copy and paste the fragment. In the instance where they wish to upload to the server it seems essential that they provide a server to upload to.

For reference, below is an example of the output when using cargo build using the private registries:

cargo build
Compiling unicode-normalization v0.1.4
Compiling matches v0.1.4
Compiling private-crate v0.3.1 (registry https://private.repo/)
Compiling unicode-bidi v0.2.5
Compiling idna v0.1.1
Compiling url v1.4.0
Compiling test v0.1.0 (file:///my-great-project/test)
Finished dev [unoptimized + debuginfo] target(s) in 7.99 secs

[cswindle]

@Phrohdoh, thanks for the feedback on the RFC. I do however have concerns about your approaches, I feel that if you are wanting to use an alternate registry for crates then you would want to explicitly specify that in the dependencies, without that I think you would run into problems where a crate with the same name happens to exist in both registries and cargo may chose the wrong version to use.

I can recognise some use of having a global config file defining the URL that you want to use for the crates, but I think that the registry to use belongs in the crate that you are defining, not in the separate config on the users machine.

I am going to update the RFC to add that an alternate crates.io registry would be updated to include the registry in the fragment of Cargo.toml displayed for each package, thus allowing users just to copy and paste the fragment. In the instance where they wish to upload to the server it seems essential that they provide a server to upload to.

For reference, below is an example of the output when using cargo build using the private registries:

cargo build
Compiling unicode-normalization v0.1.4
Compiling matches v0.1.4
Compiling private-crate v0.3.1 (registry https://private.repo/)
Compiling unicode-bidi v0.2.5
Compiling idna v0.1.1
Compiling url v1.4.0
Compiling test v0.1.0 (file:///my-great-project/test)
Finished dev [unoptimized + debuginfo] target(s) in 7.99 secs

[Phrohdoh]

if you are wanting to use an alternate registry for crates then you would want to explicitly specify that in the dependencies

Why do this if you don't have to?

If there is a conflict then it is of course up to the human to resolve (possibly with a --source cargo flag) and cargo should not be making any choices for the user (unless we say the order of registry entries also provides rank (where the lowest rank is chosen)).

I think that the registry to use belongs in the crate that you are defining
I would like to explore and discuss this further.

The large organizations I've worked at are mostly .NET based which means they use NuGet so this is where a lot of my inspiration draws from (the nuget client being a pain is another store and is separate).

It is extremely easy to spin up a private/public/personal/org NuGet server because of some choices they made (but I do think there is room for improvement there too).

[Phrohdoh]

if you are wanting to use an alternate registry for crates then you would want to explicitly specify that in the dependencies

Why do this if you don't have to?

If there is a conflict then it is of course up to the human to resolve (possibly with a --source cargo flag) and cargo should not be making any choices for the user (unless we say the order of registry entries also provides rank (where the lowest rank is chosen)).

I think that the registry to use belongs in the crate that you are defining
I would like to explore and discuss this further.

The large organizations I've worked at are mostly .NET based which means they use NuGet so this is where a lot of my inspiration draws from (the nuget client being a pain is another store and is separate).

It is extremely easy to spin up a private/public/personal/org NuGet server because of some choices they made (but I do think there is room for improvement there too).

[cswindle]

I do not like the idea of needing to get the human involved to resolve by passing a flag to cargo as it will get very complicated very quickly if you need to specify the source for two different crates. I also do not like having an ordered list of registries as if the same name crate is available on the public and private registry then in some cases you may want it one way and in others the other. I feel that providing this information in the Cargo.toml and having it checked in with the project seems most sensible to me as the registries that you want to use is part of the data which belongs to the project in my eyes.

[cswindle]

I do not like the idea of needing to get the human involved to resolve by passing a flag to cargo as it will get very complicated very quickly if you need to specify the source for two different crates. I also do not like having an ordered list of registries as if the same name crate is available on the public and private registry then in some cases you may want it one way and in others the other. I feel that providing this information in the Cargo.toml and having it checked in with the project seems most sensible to me as the registries that you want to use is part of the data which belongs to the project in my eyes.

[Blacktiger]

Many organizations that want this feature will also want to make sure they can control exactly how the crates get into the organization. This usually means they want to know that when someone downloads the dependencies for a project they only come from their repository manager (Nexus, Artifactory, etc) and nowhere else. So developers need a way to make sure their local cargo only reaches out to the repository manager. Maybe that just means that they don't provide a specific registry at the project level?

[Blacktiger]

Many organizations that want this feature will also want to make sure they can control exactly how the crates get into the organization. This usually means they want to know that when someone downloads the dependencies for a project they only come from their repository manager (Nexus, Artifactory, etc) and nowhere else. So developers need a way to make sure their local cargo only reaches out to the repository manager. Maybe that just means that they don't provide a specific registry at the project level?

[Phrohdoh]

@Blacktiger indeed, this is why I want a registry list in a global place so crates.io can be taken out of the picture entirely if necessary.

At my current place of work we host everything we need on-prem and don't use nuget.org for anything.

If we need a package from there we ask for a specific version (down to patch number) and that gets pulled, inspected, and if approved pushed to our private registry.

[Phrohdoh]

@Blacktiger indeed, this is why I want a registry list in a global place so crates.io can be taken out of the picture entirely if necessary.

At my current place of work we host everything we need on-prem and don't use nuget.org for anything.

If we need a package from there we ask for a specific version (down to patch number) and that gets pulled, inspected, and if approved pushed to our private registry.

[cswindle]

@Phrohdoh, what you are suggesting there is just to provide a replacement for crates.io, rather than something that works alongside a public registry. There are no cargo changes that would be required to support such a thing, all that would be required is only allowing a specific group of people to upload crates to your private registry, then update cargo/config to override using your new registry in place of crates.io.

[cswindle]

@Phrohdoh, what you are suggesting there is just to provide a replacement for crates.io, rather than something that works alongside a public registry. There are no cargo changes that would be required to support such a thing, all that would be required is only allowing a specific group of people to upload crates to your private registry, then update cargo/config to override using your new registry in place of crates.io.

[Phrohdoh]

I am not suggesting that I merely stated that we use that process at work.

An alternate crates.io (mirror) and an alternate registry altogether are different things IMO.

Now I am not sure which one you are proposing here (your PR title says alternate registry but the RFC says alternate crates.io).

Replacing crates.io with an on-prem / private registry naturally falls out of supporting non-crates.io registries (assuming the APIs are the same of course) but doesn't mean you have to replace crates.io.

[Phrohdoh]

I am not suggesting that I merely stated that we use that process at work.

An alternate crates.io (mirror) and an alternate registry altogether are different things IMO.

Now I am not sure which one you are proposing here (your PR title says alternate registry but the RFC says alternate crates.io).

Replacing crates.io with an on-prem / private registry naturally falls out of supporting non-crates.io registries (assuming the APIs are the same of course) but doesn't mean you have to replace crates.io.

[Phrohdoh]

I don't think broad "we believe" statements like this are as truthful as they could be.

Future readers will no doubt read this as something everyone in this discussion agreed on which isn't the case.

[cswindle]

Sorry, that was a reflection of discussions that I had within our company prior to submitting the RFC, not intended as a full reflection of the discussion here as I thought it would be useful to include.

[alexcrichton]

cc @rust-lang/cargo

[alexcrichton]

cc @rust-lang/cargo

[Phrohdoh]

Thinking about this a bit more I have come to the conclusion that named registries are not a simple as I would like to think they are and perhaps living in each Cargo.toml is what we actually want (instead of pushing that off on a global config which is just the 'easy' route).

@cswindle what do you think of this?

[package]
name = "example"

[dependencies]
libc = "=0.1.7"
foobar = { registry = "internal", version = "^0.1" }

[registries]
default = "<url to crates.io>" # maybe this isn't necessary
internal = "<url to internal / on-prem / private / etc registry host>"

[Phrohdoh]

Thinking about this a bit more I have come to the conclusion that named registries are not a simple as I would like to think they are and perhaps living in each Cargo.toml is what we actually want (instead of pushing that off on a global config which is just the 'easy' route).

@cswindle what do you think of this?

[package]
name = "example"

[dependencies]
libc = "=0.1.7"
foobar = { registry = "internal", version = "^0.1" }

[registries]
default = "<url to crates.io>" # maybe this isn't necessary
internal = "<url to internal / on-prem / private / etc registry host>"

[cswindle]

I did originally consider having the registries pulled out into a separate registries section in Cargo.toml, but thought that it was added complexity for very little gain. Also with my proposal to include the registry in the Cargo.toml fragment on the web server, this would then not be a simple copy/paste to use the library and I agree with the point you made about making it easier for the users. There is scope though for both methods to be used, but I would not be planning on adding that as part of this RFC, maybe it is something that could be proposed as a subsequent RFC if you feel there is enough justification for it.

[cswindle]

I did originally consider having the registries pulled out into a separate registries section in Cargo.toml, but thought that it was added complexity for very little gain. Also with my proposal to include the registry in the Cargo.toml fragment on the web server, this would then not be a simple copy/paste to use the library and I agree with the point you made about making it easier for the users. There is scope though for both methods to be used, but I would not be planning on adding that as part of this RFC, maybe it is something that could be proposed as a subsequent RFC if you feel there is enough justification for it.

[Phrohdoh]

That's totally fair, I'll cede the registries table in favor of { registry = "<url>" }.

[Phrohdoh]

That's totally fair, I'll cede the registries table in favor of { registry = "<url>" }.

[mark-i-m]

We still want to support private crates having dependencies on the public crates.io server, so we propose relaxing a check which ensures that the source for a dependency matches the registry. We propose that this only performs the check only if the dependency is not the default registry, thus allowing private crates to reference public crates on crates.io.

IIUC, this RFC makes it possible to have two versions of a library with the same name and version that are not the same... For example, suppose I have something like this:

[dependencies]
libc = { version = "*", registry = "https://github.com/my_awesome_fork/crates.io-index" }
foo = "*"

where foo depends on libc = "*"... Which libc does foo get? If it gets libc from crates.io then it is possible for libc (crates.io) to see types from the other libc or vice versa. But if foo gets libc from my registry, in theory it is possible for my libc to be different from what foo expects (so it doesn't compile or misbehaves).

One solution is to ensure that in the entire dependency graph all crates with the same name and version have the same registry too.

[mark-i-m]

We still want to support private crates having dependencies on the public crates.io server, so we propose relaxing a check which ensures that the source for a dependency matches the registry. We propose that this only performs the check only if the dependency is not the default registry, thus allowing private crates to reference public crates on crates.io.

IIUC, this RFC makes it possible to have two versions of a library with the same name and version that are not the same... For example, suppose I have something like this:

[dependencies]
libc = { version = "*", registry = "https://github.com/my_awesome_fork/crates.io-index" }
foo = "*"

where foo depends on libc = "*"... Which libc does foo get? If it gets libc from crates.io then it is possible for libc (crates.io) to see types from the other libc or vice versa. But if foo gets libc from my registry, in theory it is possible for my libc to be different from what foo expects (so it doesn't compile or misbehaves).

One solution is to ensure that in the entire dependency graph all crates with the same name and version have the same registry too.

[carols10cents]

What if I want to be able to publish a crate to both a private registry and crates.io? For example, what if the workflow is that I publish prerelease versions to the private registry for internal testing, and then I publish polished versions to crates.io? I don't know if this is something that's wanted/needed, but it seems like it would be made inconvenient, if not impossible, by this implementation.

[mark-i-m]

Hmm... that's true. I can imagine such a feature being useful in CI environment.

[cswindle]

It would be possible to do such a thing by not including the registry field and using a flag on your local crates.io server to allow a blank registry. I would not be intending to make such a change with this RFC though, I am trying to make this the minimum to unblock further work in this area.

[Phrohdoh]

IMO the simple solution here is to supply a --source (or --registry) flag which tells cargo which registry to publish to.

[Nemo157]

supply a --source (or --registry) flag which tells cargo which registry to publish to

👍, in the given example the package could not specify a registry as the primary release channel is the default (or it could explicitly specify crates.io if it wanted), then setup the CI to do cargo publish --registry https://internal.registry/ to override it if the tag is a prerelease version.

[carols10cents]

If I'm installing all crates from a private registry, it seems annoying to have to specify that registry for every dependency. I think it would be better if there was a way to provide a default registry and then override that default per-crate if necessary.

[cswindle]

My position is that there is a default registry and that is crates.io. I think keeping the registry with the dependency is a sensible approach as that makes it easier to copy dependencies from one Cargo.toml to another, but there is no stopping this from being done in a later stage.

[matklad]

I am also in favor of "per dependency" config because of the "optimize for the reader" advice. When reading Cargo.toml, I would like to know from which registry each of the deficiencies comes, and per-dep config seems to be the easiest way to achieve that.

Adding a "default" registry would be backwards compatible, so we can skip it anyway at this stage :)

[sfackler]

I agree that the default is something you'll want to override on a per-dependency basis, but the current syntax seems pretty verbose and prone to typos. Something like this might be nicer?

[registries]
sone_name = "https://my_internal_registry.foobar"

[dependencies]
"some_name/foo" = "0.1"
libc = "0.3"

[carols10cents]

This proposed change to crates.io means that the crates.io codebase needs to know the host where it is running, which it does not currently. That might be something you were implying here by "change to crates.io", but I just wanted to make sure you knew that currently the codebase doesn't have knowledge of where it's running.

[cswindle]

It needs to know the git repository and compare that, which it already has, so I don't think that this is an issue.

[carols10cents]

rust-lang/cargo#3365 is relevant here :)

[cswindle]

I agree and that is something which I would like as well, but I wanted to try and get something in place, which can then be built on to add support for things like this.

[matklad]

FYI, we may get this rather soon because we are moving credentials anyway: rust-lang/cargo#3978

[cswindle]

Nice to know there is progress on that front as well :)

[carols10cents]

Cargo will look up the directory tree for .cargo/config files, so it is possible to have, for example, ~/some/path/my-project/.cargo/config rather than globally.

I think Cargo.toml is a better place for this information since it's intended to be checked in and shared while .cargo/config files are meant to be ignored and personal, but I wanted to make sure you knew about .cargo/config being an option.

[cswindle]

I know they can, but I just think that logically they belong in the Cargo.toml for the use cases I am trying to solve.

[withoutboats]

I'm concerned about the implications this RFC would have for the evolvability of cargo. Right now, we control the source for both the client and the server (cargo and crates.io). Importantly, this means that we can roll out features to cargo with the knowledge that the server already supports them. We can't assume that all clients are up to date, but we can assume that crates.io is.

With multiple registries, most of which won't be run by the Rust project, we will need to worry about what happens when you make a request involving new feature X to a registry which has not been updated to support that feature. Today we have a lot more flexibility.

I'm also a bit concerned about the increased possibility of ecosystem incoherence. Someone running a public secondary registry could start introducing new features to it & release new cargo subcommands based on those features (which only work on their registry). Essentially this would enable motivated subgroups to ignore the consensus-building RFC process & partially fork the ecosystem without the extremely high cost of hard forking the ecoystem.

Both of these boil down to this: we currently have a "single server, many clients" model, and get all the benefits of that. Moving to a "many servers, many clients" model introduces significant consistency issues. This is a large drawback of this RFC.

---

Is the use case you describe of setting up a private registry hypothetical for you @cswindle, or are you currently trying to integrate cargo in a closed source environment & its not meeting your needs? Making it easier to use cargo in environments like this, where you have your own requirements, is a roadmap goal for this year, so if that's the case, we want to figure out what's needed to support your use case entirely. Feel free to leave comments on the roadmap tracking issue rust-lang/rust-roadmap-2017#12.

I think there is a more moderate approach that still treats crates.io as ultimate, but allows for a kind of 'lesser/intermediate' source. Ruby's gemstash seems like possibly a model for this. We already have a concept of source replacement, though its mainly used for local vendoring, not for setting up another server like this would proposal. I think an approach along those lines is worth pursuing.

[withoutboats]

I'm concerned about the implications this RFC would have for the evolvability of cargo. Right now, we control the source for both the client and the server (cargo and crates.io). Importantly, this means that we can roll out features to cargo with the knowledge that the server already supports them. We can't assume that all clients are up to date, but we can assume that crates.io is.

With multiple registries, most of which won't be run by the Rust project, we will need to worry about what happens when you make a request involving new feature X to a registry which has not been updated to support that feature. Today we have a lot more flexibility.

I'm also a bit concerned about the increased possibility of ecosystem incoherence. Someone running a public secondary registry could start introducing new features to it & release new cargo subcommands based on those features (which only work on their registry). Essentially this would enable motivated subgroups to ignore the consensus-building RFC process & partially fork the ecosystem without the extremely high cost of hard forking the ecoystem.

Both of these boil down to this: we currently have a "single server, many clients" model, and get all the benefits of that. Moving to a "many servers, many clients" model introduces significant consistency issues. This is a large drawback of this RFC.

---

Is the use case you describe of setting up a private registry hypothetical for you @cswindle, or are you currently trying to integrate cargo in a closed source environment & its not meeting your needs? Making it easier to use cargo in environments like this, where you have your own requirements, is a roadmap goal for this year, so if that's the case, we want to figure out what's needed to support your use case entirely. Feel free to leave comments on the roadmap tracking issue rust-lang/rust-roadmap-2017#12.

I think there is a more moderate approach that still treats crates.io as ultimate, but allows for a kind of 'lesser/intermediate' source. Ruby's gemstash seems like possibly a model for this. We already have a concept of source replacement, though its mainly used for local vendoring, not for setting up another server like this would proposal. I think an approach along those lines is worth pursuing.

[withoutboats]

Actually, reading the source replacement documentation again, I see that it does support non-local registries:

A "registry source" is one that is the same as crates.io itself. That is, it has an index served in a git repository which matches the format of the crates.io index. That repository then has configuration indicating where to download crates from.

Currently there is not an already-available project for setting up a mirror of crates.io. Stay tuned though!

This seems like it supports the use case described in the motivation. I suppose the concern is about making these "registry replacements" support including crates that aren't in the crates.io index (currently every source in the replacement needs to be present in the replaced source).

[withoutboats]

Actually, reading the source replacement documentation again, I see that it does support non-local registries:

A "registry source" is one that is the same as crates.io itself. That is, it has an index served in a git repository which matches the format of the crates.io index. That repository then has configuration indicating where to download crates from.

Currently there is not an already-available project for setting up a mirror of crates.io. Stay tuned though!

This seems like it supports the use case described in the motivation. I suppose the concern is about making these "registry replacements" support including crates that aren't in the crates.io index (currently every source in the replacement needs to be present in the replaced source).

[cswindle]

@withoutboats, I am not sure that I agree that you currently have full control of the server, it is already possible to setup mirrors and to override to use those, so I think that you already have the scenario where you should care about other server. I would agree that this is a step further along that road, but I do not see any alternative for enterprises which wish to have closed source integrated with open source that is a nice experience for the user. One of the great benefits of rust is that crates.io/cargo is really good at discovering crates, without that for closed source projects it makes rust pretty annoying. I agree that there are ways to work around it by using git repositories for everything, however that becomes very tedious to use.

Regarding using source replacement, you could update it to allow replacement when they aren't present on crates.io, but that feels like the wrong thing to me. I would personally like rust to be as nice an experience for closed source software development as it is for open source software and have all the same tools available and a normal rust way to use them. I accept that in doing that you will then lose some control over the server component.

[cswindle]

@withoutboats, I am not sure that I agree that you currently have full control of the server, it is already possible to setup mirrors and to override to use those, so I think that you already have the scenario where you should care about other server. I would agree that this is a step further along that road, but I do not see any alternative for enterprises which wish to have closed source integrated with open source that is a nice experience for the user. One of the great benefits of rust is that crates.io/cargo is really good at discovering crates, without that for closed source projects it makes rust pretty annoying. I agree that there are ways to work around it by using git repositories for everything, however that becomes very tedious to use.

Regarding using source replacement, you could update it to allow replacement when they aren't present on crates.io, but that feels like the wrong thing to me. I would personally like rust to be as nice an experience for closed source software development as it is for open source software and have all the same tools available and a normal rust way to use them. I accept that in doing that you will then lose some control over the server component.

[internetionals]

In our corporate environment we deal with a few external repositories and maintain custom repositories on the side. Here are some of our motivations.

In our current dealings with remote repositories for other languages, we often want to:

• Add custom version of specific packages (for non-maintainer updates)
• Force certain packages to always come from our repositories (for packages that we need to patch)
• Provide internal packages (these must always come from our repositories)
• Provide pre-release versions that are only accessible for certain environments

So in 3 of these cases, we explicitly want to receive the packages from our own repository only. Should our own repository, for whatever reason (malicious, human error, ..) serve the wrong replies, then we don't want to suddenly receive some other package or version.

The other case, where we just want to release a hot fix version of some package, we want to be able to do that for all possible packages available from upstream.

We currently handle different repositories and toolchains. The solutions we currently use:

• Provide selective repositories and configure the toolchains to prefer our packages above the others and to only lookup specially named packages in our repositories (mostly based on a prefix).
• Mirror the entire upstream repository index and apply our custom packages on top of that. The toolchains only ever talk to our version of the repositories.

The first one is generally easier to setup and maintain. We always try to use some form of tracking the changes in the upstream repositories, eg. through mirroring. We often setup multiple internal repositories with released versions and testing/CI versions.

For cargo to fit this use case, we'd want to be able to specify where the preferred repository is, where the fallback repositories are and a way to explicitly set certain dependencies to always come from on of those.

The second is a lot more intrusive, in that you need better tools to maintain it and go with the changes in the upstream repository management.

In this case we'd only need cargo to specify which upstream repository it must use.

The choice between those two solutions is mainly dominated by the setup and maintenance effort:

• The first is often the easiest to implement as many toolchains support that and the repository management tools are often lacking. This is often the "cheap but pragmatic" solution and used as long as the toolchain supports it.
• The second is the most preferred, but only if there is readily available software for the repository management. In that case the policy is fully enforced at the repository level and tracking updates is generally also easy. It also has the benefit of allowing you to have full control over the availability and the amount of information you "leak".

One thing that helps in both cases: Have an option to say, preferably system wide, what the default upstream repository is. This allows for the environment to control where to fetch the packages from when they are not explicitly pulled from a specific repository. This can be used to prevent unintended "leakage" and allows environment specific repositories to be used.

Hopefully this gives some insights into some of the use cases users might have.

[internetionals]

In our corporate environment we deal with a few external repositories and maintain custom repositories on the side. Here are some of our motivations.

In our current dealings with remote repositories for other languages, we often want to:

• Add custom version of specific packages (for non-maintainer updates)
• Force certain packages to always come from our repositories (for packages that we need to patch)
• Provide internal packages (these must always come from our repositories)
• Provide pre-release versions that are only accessible for certain environments

So in 3 of these cases, we explicitly want to receive the packages from our own repository only. Should our own repository, for whatever reason (malicious, human error, ..) serve the wrong replies, then we don't want to suddenly receive some other package or version.

The other case, where we just want to release a hot fix version of some package, we want to be able to do that for all possible packages available from upstream.

We currently handle different repositories and toolchains. The solutions we currently use:

• Provide selective repositories and configure the toolchains to prefer our packages above the others and to only lookup specially named packages in our repositories (mostly based on a prefix).
• Mirror the entire upstream repository index and apply our custom packages on top of that. The toolchains only ever talk to our version of the repositories.

The first one is generally easier to setup and maintain. We always try to use some form of tracking the changes in the upstream repositories, eg. through mirroring. We often setup multiple internal repositories with released versions and testing/CI versions.

For cargo to fit this use case, we'd want to be able to specify where the preferred repository is, where the fallback repositories are and a way to explicitly set certain dependencies to always come from on of those.

The second is a lot more intrusive, in that you need better tools to maintain it and go with the changes in the upstream repository management.

In this case we'd only need cargo to specify which upstream repository it must use.

The choice between those two solutions is mainly dominated by the setup and maintenance effort:

• The first is often the easiest to implement as many toolchains support that and the repository management tools are often lacking. This is often the "cheap but pragmatic" solution and used as long as the toolchain supports it.
• The second is the most preferred, but only if there is readily available software for the repository management. In that case the policy is fully enforced at the repository level and tracking updates is generally also easy. It also has the benefit of allowing you to have full control over the availability and the amount of information you "leak".

One thing that helps in both cases: Have an option to say, preferably system wide, what the default upstream repository is. This allows for the environment to control where to fetch the packages from when they are not explicitly pulled from a specific repository. This can be used to prevent unintended "leakage" and allows environment specific repositories to be used.

Hopefully this gives some insights into some of the use cases users might have.

[carols10cents]

I'm interested in hearing from @alexcrichton @wycats @matklad about the technical implications of this proposal-- is this proposal compatible with how cargo works today? Is this proposal compatible with the future use cases mentioned in the proposal? What other considerations are there that we might be missing?

[carols10cents]

I'm interested in hearing from @alexcrichton @wycats @matklad about the technical implications of this proposal-- is this proposal compatible with how cargo works today? Is this proposal compatible with the future use cases mentioned in the proposal? What other considerations are there that we might be missing?

[matklad]

@mark-i-m raised a very interesting concern: #2006 (comment)

I think we need to cover in the RFC how dependencies resolution would work in the presence of registry key. The core question seems to be:

If crate X depends on crate Y from the default registry, could this dependency be satisfied by the crate Y from a private registry?

I think the safe answer is "No". But there is perhaps an interesting case where you want to patch some dependency across the ecosystem.

[matklad]

@mark-i-m raised a very interesting concern: #2006 (comment)

I think we need to cover in the RFC how dependencies resolution would work in the presence of registry key. The core question seems to be:

If crate X depends on crate Y from the default registry, could this dependency be satisfied by the crate Y from a private registry?

I think the safe answer is "No". But there is perhaps an interesting case where you want to patch some dependency across the ecosystem.

[cswindle]

@matklad, my position would be that if a dependency does not include a registry it comes from the default registry. For the case of overriding I would prefer for the source replacement to be used as that seems a cleaner way to do it. I will look at updating the rfc to provide further details on this area (although it may be a few days before I get a chance to do so).

[cswindle]

@matklad, my position would be that if a dependency does not include a registry it comes from the default registry. For the case of overriding I would prefer for the source replacement to be used as that seems a cleaner way to do it. I will look at updating the rfc to provide further details on this area (although it may be a few days before I get a chance to do so).

[mark-i-m]

@withoutboats Another major motivation I can think of is licensing. At a company I worked for before, we had a very rigorous vetting/curation process and ONLY approved stuff was added to the local apt-get mirror or code repo. Of course, a significant part was quality of dependencies, but there was also very strict legal requirements too...

@matklad To be honest, @Ericson2314, @alexcrichton, and @aturon first brought up a related issue in #1977 (comment) about public and private dependencies. IIRC, the invariant they are planning to enforce there is that no crate should see items from a different version of itself (it results in an error). That solution is consistent with what you said, but I agree that it is possibly a large limitation of the feature.

But there is perhaps an interesting case where you want to patch some dependency across the ecosystem.

I can easily see this happening in a corporate setting... e.g. many smartphone makers maintain forks of Linux and Android with various in-house features added in.

[mark-i-m]

@withoutboats Another major motivation I can think of is licensing. At a company I worked for before, we had a very rigorous vetting/curation process and ONLY approved stuff was added to the local apt-get mirror or code repo. Of course, a significant part was quality of dependencies, but there was also very strict legal requirements too...

@matklad To be honest, @Ericson2314, @alexcrichton, and @aturon first brought up a related issue in #1977 (comment) about public and private dependencies. IIRC, the invariant they are planning to enforce there is that no crate should see items from a different version of itself (it results in an error). That solution is consistent with what you said, but I agree that it is possibly a large limitation of the feature.

But there is perhaps an interesting case where you want to patch some dependency across the ecosystem.

I can easily see this happening in a corporate setting... e.g. many smartphone makers maintain forks of Linux and Android with various in-house features added in.

[withoutboats]

@withoutboats Another major motivation I can think of is licensing. At a company I worked for before, we had a very rigorous vetting/curation process and ONLY approved stuff was added to the local apt-get mirror or code repo. Of course, a significant part was quality of dependencies, but there was also very strict legal requirements too...

This is a clear use case for source replacement though, not what this RFC proposes.

[withoutboats]

@withoutboats Another major motivation I can think of is licensing. At a company I worked for before, we had a very rigorous vetting/curation process and ONLY approved stuff was added to the local apt-get mirror or code repo. Of course, a significant part was quality of dependencies, but there was also very strict legal requirements too...

This is a clear use case for source replacement though, not what this RFC proposes.

[cswindle]

I have just updated the document to include information about dependency resolution, plus I have added an option to allow an alternate server administrator to insist that crates names all have a specific prefix.

[cswindle]

I have just updated the document to include information about dependency resolution, plus I have added an option to allow an alternate server administrator to insist that crates names all have a specific prefix.

[alexcrichton]

I'm interested in hearing from @alexcrichton @wycats @matklad about the technical implications of this proposal

AFAIK Cargo is nearly 100% ready for this in terms of internal plumbing. Since day 1 Cargo has "supported multiple registries" in the sense that it's not hardcoded to crates.io everywhere. The main technical hurdle in my mind is cross-registry dependencies. This is mentioned in the RFC as it should be supported (which I complete agree with!) but the index format for a registry currently has no support for this. It would not be super hard to add, I think, though.

One of my main worries about this RFC though is simply having a custom registry to run. Right now you can't exactly check out the crates.io code base, hit a button, and have your own registry. There's a lot of internal legwork to do I think to support something like that. Is there a plan for how to get the crates.io repository into the shape that it's ready to run as a custom registry in a different location?

---

If crate X depends on crate Y from the default registry, could this dependency be satisfied by the crate Y from a private registry?

Note that internally Cargo identifies crates with three pieces of information: name, version, source. All distinct triples are distinct packages. So in the same way foo v0.1.0 is different when it comes from git vs crates.io, foo v0.1.0 will be different regardless of what registry it comes from (the source is changing).

In that sense I think it's perfectly fine (acceptable, and encouraged!) for non-crates.io registry to basically be their own namespace. A "registry" is a flat list of names but it's still scoped by the url of the registry itself.

[alexcrichton]

I'm interested in hearing from @alexcrichton @wycats @matklad about the technical implications of this proposal

AFAIK Cargo is nearly 100% ready for this in terms of internal plumbing. Since day 1 Cargo has "supported multiple registries" in the sense that it's not hardcoded to crates.io everywhere. The main technical hurdle in my mind is cross-registry dependencies. This is mentioned in the RFC as it should be supported (which I complete agree with!) but the index format for a registry currently has no support for this. It would not be super hard to add, I think, though.

One of my main worries about this RFC though is simply having a custom registry to run. Right now you can't exactly check out the crates.io code base, hit a button, and have your own registry. There's a lot of internal legwork to do I think to support something like that. Is there a plan for how to get the crates.io repository into the shape that it's ready to run as a custom registry in a different location?

---

If crate X depends on crate Y from the default registry, could this dependency be satisfied by the crate Y from a private registry?

Note that internally Cargo identifies crates with three pieces of information: name, version, source. All distinct triples are distinct packages. So in the same way foo v0.1.0 is different when it comes from git vs crates.io, foo v0.1.0 will be different regardless of what registry it comes from (the source is changing).

In that sense I think it's perfectly fine (acceptable, and encouraged!) for non-crates.io registry to basically be their own namespace. A "registry" is a flat list of names but it's still scoped by the url of the registry itself.

[cswindle]

@alexcrichton, I agree that work needs to be done to make spinning up a crates.io simpler, maybe making it a docker container may make life easier for people for very little effort. However, the fact that it will take quite a long time to get the changes into cargo stable, so that gives a good chunk of time to make progress on the crates.io server work.

Regarding the multiple crates with the same name, can you actually import two crates of the same name into a Cargo.toml file when they have different sources, if so how do you specify which one you want to use in the rust files?

For the cross-registry dependencies, I was not suggesting actually encoding that it in the index files as part of this RFC, instead when crates are pushed up to the alternate registry they include the registry key for dependencies for that server and no key for dependencies on the public crates.io server. Doing this just then works with Cargo with very minimal changes (as the PR I previously sent shows). Is there a reason that you think that the external dependencies should be encoded into the index file?

[cswindle]

@alexcrichton, I agree that work needs to be done to make spinning up a crates.io simpler, maybe making it a docker container may make life easier for people for very little effort. However, the fact that it will take quite a long time to get the changes into cargo stable, so that gives a good chunk of time to make progress on the crates.io server work.

Regarding the multiple crates with the same name, can you actually import two crates of the same name into a Cargo.toml file when they have different sources, if so how do you specify which one you want to use in the rust files?

For the cross-registry dependencies, I was not suggesting actually encoding that it in the index files as part of this RFC, instead when crates are pushed up to the alternate registry they include the registry key for dependencies for that server and no key for dependencies on the public crates.io server. Doing this just then works with Cargo with very minimal changes (as the PR I previously sent shows). Is there a reason that you think that the external dependencies should be encoded into the index file?

[alexcrichton]

However, the fact that it will take quite a long time to get the changes into cargo stable, so that gives a good chunk of time to make progress on the crates.io server work.

A very good point! This sounds like a fine plan to me. I'm just thinking we may want to have some rough plan in place for how to make this usable (but just refactoring/working on existing crates.io sounds perfect)

Regarding the multiple crates with the same name, can you actually import two crates of the same name into a Cargo.toml file when they have different sources, if so how do you specify which one you want to use in the rust files?

An excellent point! Right now this mostly matters with transitive dependencies. For example if you use foo from a custom registry but one of your deps uses foo from crates.io that needs to work. But of course there's also one crate using foo from both registries! My thinking is that we should support something like:

[dependencies]
foo = "0.1"
bar = { version = "0.2", name = "foo" }

where that means that the dependency named foo is foo v0.1.0 from crates,io, and the dependency named bar is named foo v0.2.0 from crates.io. Basically I think we'd just extend the dependency format slightly to support this.

Note, though, that much of Cargo's internals I think aren't quite ready for this feature. They're mostly doing a lot of name/version matching right now, and so if we add aliasing like this it'll probably take a little bit to get the backend all working.

Is there a reason that you think that the external dependencies should be encoded into the index file?

When performing crate graph resolution Cargo only uses the index for registries, nothing else. In that sense it must be able to discover all the dependencies for a crate with just the index, and if we didn't encode this information in the index the crate graphs could be broken.

[alexcrichton]

However, the fact that it will take quite a long time to get the changes into cargo stable, so that gives a good chunk of time to make progress on the crates.io server work.

A very good point! This sounds like a fine plan to me. I'm just thinking we may want to have some rough plan in place for how to make this usable (but just refactoring/working on existing crates.io sounds perfect)

Regarding the multiple crates with the same name, can you actually import two crates of the same name into a Cargo.toml file when they have different sources, if so how do you specify which one you want to use in the rust files?

An excellent point! Right now this mostly matters with transitive dependencies. For example if you use foo from a custom registry but one of your deps uses foo from crates.io that needs to work. But of course there's also one crate using foo from both registries! My thinking is that we should support something like:

[dependencies]
foo = "0.1"
bar = { version = "0.2", name = "foo" }

where that means that the dependency named foo is foo v0.1.0 from crates,io, and the dependency named bar is named foo v0.2.0 from crates.io. Basically I think we'd just extend the dependency format slightly to support this.

Note, though, that much of Cargo's internals I think aren't quite ready for this feature. They're mostly doing a lot of name/version matching right now, and so if we add aliasing like this it'll probably take a little bit to get the backend all working.

Is there a reason that you think that the external dependencies should be encoded into the index file?

When performing crate graph resolution Cargo only uses the index for registries, nothing else. In that sense it must be able to discover all the dependencies for a crate with just the index, and if we didn't encode this information in the index the crate graphs could be broken.

[cswindle]

@alexcrichton, regarding the graph resolution, doesn't it do multiple rounds of resolution, certainly in my testing I was able to include crates from crates.io in a lib in a private repo, in the same way other sources can include crates.io dependencies and have it resolve what was needed.

[cswindle]

@alexcrichton, regarding the graph resolution, doesn't it do multiple rounds of resolution, certainly in my testing I was able to include crates from crates.io in a lib in a private repo, in the same way other sources can include crates.io dependencies and have it resolve what was needed.

[Ericson2314]

Integration with [augment] is the crucial missing piece for such a proposal as this.

[Ericson2314]

Integration with [augment] is the crucial missing piece for such a proposal as this.

[cswindle]

Hi Andy Not sure if the person is suggesting we should add support for new config [augment], or if there is already support (I couldn't find any mention from my phone). Either way I don't think it is the crucial part, any chance you could respond after having a little dig? Thanks Chris On 30 May 2017, at 03:19, John Ericson <notifications@github.com<mailto:notifications@github.com>> wrote: Integration with [augment] is the crucial missing piece for such a proposal as this. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#2006 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/APaLDgkqn08ZY_Pe9DuuBSJ1M8iATD-lks5r-3yZgaJpZM4Njlao>.

[cswindle]

Hi Andy Not sure if the person is suggesting we should add support for new config [augment], or if there is already support (I couldn't find any mention from my phone). Either way I don't think it is the crucial part, any chance you could respond after having a little dig? Thanks Chris On 30 May 2017, at 03:19, John Ericson <notifications@github.com<mailto:notifications@github.com>> wrote: Integration with [augment] is the crucial missing piece for such a proposal as this. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#2006 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/APaLDgkqn08ZY_Pe9DuuBSJ1M8iATD-lks5r-3yZgaJpZM4Njlao>.

[cswindle]

@carols10cents, @alexcrichton: I do not agree that allowing aliasing should be something that is included in this RFC on the following grounds:

• people can already get into this situation when using other sources (eg git)
• this seems like a useful thing that people who are not interested in an alternate registry may be interested in

I do think that it is a useful feature for cargo, however I would prefer that this gets dealt with as a separate RFC (which I am not currently planning on writing), that way it will have more visibility to people who have an opinion on how it should work.

@alexcrichton: Regarding encoding external dependencies in the index file, please can you comment on my response #2006 (comment)?

[cswindle]

@carols10cents, @alexcrichton: I do not agree that allowing aliasing should be something that is included in this RFC on the following grounds:

• people can already get into this situation when using other sources (eg git)
• this seems like a useful thing that people who are not interested in an alternate registry may be interested in

I do think that it is a useful feature for cargo, however I would prefer that this gets dealt with as a separate RFC (which I am not currently planning on writing), that way it will have more visibility to people who have an opinion on how it should work.

@alexcrichton: Regarding encoding external dependencies in the index file, please can you comment on my response #2006 (comment)?

[alexcrichton]

@cswindle your comment above I think unfortunately doesn't address the problem. Cargo needs to learn about cross-registry dependencies without downloading crates, and it has no mechanism to do so if it's not in the index. This information must be in the index somewhere.

[alexcrichton]

@cswindle your comment above I think unfortunately doesn't address the problem. Cargo needs to learn about cross-registry dependencies without downloading crates, and it has no mechanism to do so if it's not in the index. This information must be in the index somewhere.

[SamWhited]

Hi all, I recently wrote a similar proposal (Rendered) before being pointed here by someone on IRC. The main difference in my proposal is that most of the "discovery" legwork was handed off to DNS.

Right now, all crates on crates.io have a simple line telling users how to use the crate:

While a business that wants to import some crates in internal projects from an internal repo may not mind that you have to add a registry, I think it would be unfortunate for public crates where the primary distribution mechanism is a third party server to have to provide instructions for adding a named registry. That is to say, dependencies on third party registries should be just as easy to add to your imports as a dependency on the centralized crates.io. To do this, I allowed imports to be domain namespaced (eg. you'd import example.net/xmpp-addr; this way any domain that you control can be used, and the import line remains simple. Your custom crates.io instance might show:

This is separate from the actual package repository, however. If your package repo lives at example.net, nothing further is needed (cargo will find it based on that import line), however, if it doesn't (because your companies webserver is already running there), cargo would look up a DNS SRV record which determines where the package lives. This means that as long as you own the domain, if you eg. decide to move your packages back to crates.io you can do so and the import remains the same (cargo looks up the SRV record, sees that it delegates responsibility for packages at example.net to crates.io, and then fetches the package from crates.io). This does require that crates.io be able to handle packages with names such as example.net/package, which is a change that's not yet in the RFC I was writing.

Having this sort of decoupling makes it easy for there to be several different public crates.io repos, and I'd love to see some of these ideas borrowed for whatever the final solution ends up being (I'd like to run my own crates.io instance for my package, namespaced under the projects domain, similar to what Go does with "vanity imports", but with a bit more structure).

[SamWhited]

Hi all, I recently wrote a similar proposal (Rendered) before being pointed here by someone on IRC. The main difference in my proposal is that most of the "discovery" legwork was handed off to DNS.

Right now, all crates on crates.io have a simple line telling users how to use the crate:

While a business that wants to import some crates in internal projects from an internal repo may not mind that you have to add a registry, I think it would be unfortunate for public crates where the primary distribution mechanism is a third party server to have to provide instructions for adding a named registry. That is to say, dependencies on third party registries should be just as easy to add to your imports as a dependency on the centralized crates.io. To do this, I allowed imports to be domain namespaced (eg. you'd import example.net/xmpp-addr; this way any domain that you control can be used, and the import line remains simple. Your custom crates.io instance might show:

This is separate from the actual package repository, however. If your package repo lives at example.net, nothing further is needed (cargo will find it based on that import line), however, if it doesn't (because your companies webserver is already running there), cargo would look up a DNS SRV record which determines where the package lives. This means that as long as you own the domain, if you eg. decide to move your packages back to crates.io you can do so and the import remains the same (cargo looks up the SRV record, sees that it delegates responsibility for packages at example.net to crates.io, and then fetches the package from crates.io). This does require that crates.io be able to handle packages with names such as example.net/package, which is a change that's not yet in the RFC I was writing.

Having this sort of decoupling makes it easy for there to be several different public crates.io repos, and I'd love to see some of these ideas borrowed for whatever the final solution ends up being (I'd like to run my own crates.io instance for my package, namespaced under the projects domain, similar to what Go does with "vanity imports", but with a bit more structure).

[cswindle]

@alexcrichton, I have now updated the RFC to include information on updating the index file.

[cswindle]

@alexcrichton, I have now updated the RFC to include information on updating the index file.

[cswindle]

@SamWhited, my proposal is for providing an alternative crates.io for closed source projects to allow a company to setup an internal crates.io server to publish to. I have major reservations about pushing things to crates.io that include links to an external crates.io server, once you start down that road it becomes very hard for a company to ensure build continuity in the future, at least if it is a single server you can put in place plans to cope with that server being taken offline, that becomes a lot harder to do if the single server has references to hundreds of other servers.

[cswindle]

@SamWhited, my proposal is for providing an alternative crates.io for closed source projects to allow a company to setup an internal crates.io server to publish to. I have major reservations about pushing things to crates.io that include links to an external crates.io server, once you start down that road it becomes very hard for a company to ensure build continuity in the future, at least if it is a single server you can put in place plans to cope with that server being taken offline, that becomes a lot harder to do if the single server has references to hundreds of other servers.

[SamWhited]

my proposal is for providing an alternative crates.io for closed source projects to allow a company to setup an internal crates.io server to publish to.

Something I said was probably unclear, sorry for the confusion; I am aware this is what the proposal is for, it is what my proposal was about too. I just handled cargo discovering the location of the server as part of the package name (of course, this should not stop you from mirroring the packages on your internal repo); slightly broader scope, perhaps, but it's related enough that I wouldn't want the two proposals to stomp on eachother.

I have major reservations about pushing things to crates.io that include links to an external crates.io server, once you start down that road it becomes very hard for a company to ensure build continuity in the future, at least if it is a single server you can put in place plans to cope with that server being taken offline, that becomes a lot harder to do if the single server has references to hundreds of other servers.

That's fair; that's why I brought it up for discussion. I do think one requirement would be for companies to mirror other packages locally so that they could guarantee consistency.

[SamWhited]

my proposal is for providing an alternative crates.io for closed source projects to allow a company to setup an internal crates.io server to publish to.

Something I said was probably unclear, sorry for the confusion; I am aware this is what the proposal is for, it is what my proposal was about too. I just handled cargo discovering the location of the server as part of the package name (of course, this should not stop you from mirroring the packages on your internal repo); slightly broader scope, perhaps, but it's related enough that I wouldn't want the two proposals to stomp on eachother.

I have major reservations about pushing things to crates.io that include links to an external crates.io server, once you start down that road it becomes very hard for a company to ensure build continuity in the future, at least if it is a single server you can put in place plans to cope with that server being taken offline, that becomes a lot harder to do if the single server has references to hundreds of other servers.

That's fair; that's why I brought it up for discussion. I do think one requirement would be for companies to mirror other packages locally so that they could guarantee consistency.

[Blacktiger]

I just handled cargo discovering the location of the server as part of the package name (of course, this should not stop you from mirroring the packages on your internal repo); slightly broader scope, perhaps, but it's related enough that I wouldn't want the two proposals to stomp on eachother.

I don't think you want to have the package name include where to find the repository. What if the location of that repository has nothing to do with the name? Even if you do something like name which repository to lookup (with the location stored somewhere separate) I think it might get confusing to someone reading the dependencies.

[Blacktiger]

I just handled cargo discovering the location of the server as part of the package name (of course, this should not stop you from mirroring the packages on your internal repo); slightly broader scope, perhaps, but it's related enough that I wouldn't want the two proposals to stomp on eachother.

I don't think you want to have the package name include where to find the repository. What if the location of that repository has nothing to do with the name? Even if you do something like name which repository to lookup (with the location stored somewhere separate) I think it might get confusing to someone reading the dependencies.

[SamWhited]

I don't think you want to have the package name include where to find the repository. What if the location of that repository has nothing to do with the name?

The name and import path are decoupled; you don't have to import it this way, it's just a method of discovering a server that's not whatever you have your [source] (or default named repo, in this proposal) set too.

Even if you do something like name which repository to lookup (with the location stored somewhere separate) I think it might get confusing to someone reading the dependencies.

How so? This is broadly similar to what Go does (although Go does it in a somewhat less structured manner which I think sometimes leads to problems), and it is often considered one of the main benefits of the language. It certainly makes things easy.

Effectively my proposals scope is just a bit broader; it also includes the use case where I want to host all my own packages but also make some of them public (and not have name conflicts with crates.io), not just host all my own packages and keep them all private (a use case that should also be covered). In effect, it becomes a federated package system where packages can be cached locally on any node.

EDIT: I think I came up with a better way to say what I'm getting at; the current proposal's primary focus is on decoupling private internal, private package distribution from crates.io. What I think needs to be added (and what the proposal I had started was going to mostly focus on), is that the crates.io monopoly on public package distribution also needs to be broken. If I'm a company distributing a few packages that I've decided to make public (or even a private individual that just doesn't like crates.io's policies), I should be able to distribute my own packages under my own policies without the users having to jump through extra hoops to depend on my packages in their projects. Companies (and some individuals) may not be comfortable with granting crates.io an unlimited license to use their packages forever, which is effectively what crates.io requires you to do. As an example, if I discover a serious security flaw in a package and need to remove (not just yank) all versions of the package that contain the flaw, I can't currenlty do that (I can email the crates.io team, and maybe they do it and maybe they don't, or maybe they don't see the email for a week, who knows). If I maintain control of my own package distribution, I have that flexibility if I need it.

[SamWhited]

I don't think you want to have the package name include where to find the repository. What if the location of that repository has nothing to do with the name?

The name and import path are decoupled; you don't have to import it this way, it's just a method of discovering a server that's not whatever you have your [source] (or default named repo, in this proposal) set too.

Even if you do something like name which repository to lookup (with the location stored somewhere separate) I think it might get confusing to someone reading the dependencies.

How so? This is broadly similar to what Go does (although Go does it in a somewhat less structured manner which I think sometimes leads to problems), and it is often considered one of the main benefits of the language. It certainly makes things easy.

Effectively my proposals scope is just a bit broader; it also includes the use case where I want to host all my own packages but also make some of them public (and not have name conflicts with crates.io), not just host all my own packages and keep them all private (a use case that should also be covered). In effect, it becomes a federated package system where packages can be cached locally on any node.

EDIT: I think I came up with a better way to say what I'm getting at; the current proposal's primary focus is on decoupling private internal, private package distribution from crates.io. What I think needs to be added (and what the proposal I had started was going to mostly focus on), is that the crates.io monopoly on public package distribution also needs to be broken. If I'm a company distributing a few packages that I've decided to make public (or even a private individual that just doesn't like crates.io's policies), I should be able to distribute my own packages under my own policies without the users having to jump through extra hoops to depend on my packages in their projects. Companies (and some individuals) may not be comfortable with granting crates.io an unlimited license to use their packages forever, which is effectively what crates.io requires you to do. As an example, if I discover a serious security flaw in a package and need to remove (not just yank) all versions of the package that contain the flaw, I can't currenlty do that (I can email the crates.io team, and maybe they do it and maybe they don't, or maybe they don't see the email for a week, who knows). If I maintain control of my own package distribution, I have that flexibility if I need it.

[internetionals]

@SamWhited, I do see the charm of having easily accessible decentralised hosting, it would probably also take away the single most important guarantee that Cargo in combination with crates.io brings: Continued availability of dependencies.

When it becomes main swing to host your own package repository and people or companies effectively "move on" to other solutions, than those repositories will disappear in due time. Now you have to locally archive all remote (indirect) dependencies in your project to ensure continued access to them.

I think most companies would prefer full local hosting of all crates that are being used inside their company. So they would probably locally mirror (parts) of crates.io in combination with their internal libraries rather than only host their own packages locally.

I do think your proposal has merrits, but I think more as a way to automatically pull crates into a central repository like crates.io. This way you keep the central "blessed" repository, which also acts as a way to ensure package names are unique.

Incorporating support for this mechanism allows certain designated crates in a private "blessed" repository to be shared with the public or other private repositories.

[internetionals]

@SamWhited, I do see the charm of having easily accessible decentralised hosting, it would probably also take away the single most important guarantee that Cargo in combination with crates.io brings: Continued availability of dependencies.

When it becomes main swing to host your own package repository and people or companies effectively "move on" to other solutions, than those repositories will disappear in due time. Now you have to locally archive all remote (indirect) dependencies in your project to ensure continued access to them.

I think most companies would prefer full local hosting of all crates that are being used inside their company. So they would probably locally mirror (parts) of crates.io in combination with their internal libraries rather than only host their own packages locally.

I do think your proposal has merrits, but I think more as a way to automatically pull crates into a central repository like crates.io. This way you keep the central "blessed" repository, which also acts as a way to ensure package names are unique.

Incorporating support for this mechanism allows certain designated crates in a private "blessed" repository to be shared with the public or other private repositories.

[cswindle]

I would not be keen on a situation where DNS is used to locate packages as I think that is going to have the following major impacts:

• external servers can go down, then it would not be possible to re-compile older code - this is major for a company that needs to be able to build patches for programs in 10yrs
• discoverability of crates goes down, you will effectively end up in a situation like C, where you need to try and find each and every library you need, plus work out how/where to get them from

On the basis of that, I do not want to go down that route with this PR as I do not think it would be a good direction for rust to go in. You are welcome to raise your proposal and see if enough people wish that to be the direction, however I would now like this thread to get back to the RFC as proposed in this PR.

[cswindle]

I would not be keen on a situation where DNS is used to locate packages as I think that is going to have the following major impacts:

• external servers can go down, then it would not be possible to re-compile older code - this is major for a company that needs to be able to build patches for programs in 10yrs
• discoverability of crates goes down, you will effectively end up in a situation like C, where you need to try and find each and every library you need, plus work out how/where to get them from

On the basis of that, I do not want to go down that route with this PR as I do not think it would be a good direction for rust to go in. You are welcome to raise your proposal and see if enough people wish that to be the direction, however I would now like this thread to get back to the RFC as proposed in this PR.

[SamWhited]

also take away the single most important guarantee that Cargo in combination with crates.io brings: Continued availability of dependencies.

It would not take this guarantee away; it would move it to the crates.io service. If you want that guarantee, only use the original service or other services that you trust that provide similar guarantees.

I think most companies would prefer full local hosting of all crates that are being used inside their company.

…

external servers can go down, then it would not be possible to re-compile older code - this is major for a company that needs to be able to build patches for programs in 10yrs

I agree, I am not suggesting that we get rid of this requirement. Companies would still probably set a global blessed package repo as the default and pull everything from there and would not use the DNS discovery features at all.

Thanks for your feedback.

[SamWhited]

also take away the single most important guarantee that Cargo in combination with crates.io brings: Continued availability of dependencies.

It would not take this guarantee away; it would move it to the crates.io service. If you want that guarantee, only use the original service or other services that you trust that provide similar guarantees.

I think most companies would prefer full local hosting of all crates that are being used inside their company.

…

external servers can go down, then it would not be possible to re-compile older code - this is major for a company that needs to be able to build patches for programs in 10yrs

I agree, I am not suggesting that we get rid of this requirement. Companies would still probably set a global blessed package repo as the default and pull everything from there and would not use the DNS discovery features at all.

Thanks for your feedback.

[alexcrichton]

@cswindle yes for the index changes that looks sufficient

[alexcrichton]

@cswindle yes for the index changes that looks sufficient

[cswindle]

@carols10cents, I believe that with the latest updatesthere is nothing else that is blocking this RFC, do you agree?

[cswindle]

@carols10cents, I believe that with the latest updatesthere is nothing else that is blocking this RFC, do you agree?

[carols10cents]

One thing that I mentioned to @cswindle in IRC is that this week is the Mozilla All Hands, so a bunch of us are going to all be in one place for the week and will be discussing lots of things. I plan to bring this up in order to move this forward.

Rereading the text of the RFC, I think my main issues are (listing them here rather than on comments in the RFC so that I can better track when all of these have been resolved, because I think some of my comments are getting lost):

• I think we want to move towards specifying host rather than index location everywhere. I'd rather have to put https://my-awesome-fork.com in my Cargo.tomls instead of https://github.com/my_awesome_fork/crates.io-index, for example. So I'd like to see that work get done before implementing this.

• Regarding the "Crate naming on alternate servers" section, I think I'd also like to see support in Cargo.toml for renaming a dependency, much like we can alias things with use. For example:

  [dependencies]
  libc = { version = "*" } // crates.io libc
  libc = { as="awesomefork-libc", version = "*", registry = "https://github.com/my_awesome_fork/crates.io-index" }
  

  Then everywhere within my crate, I could refer to extern crate awesomefork-libc. I know @alexcrichton brought this up recently as something that would be useful for something else, but I can't remember where or why. Feature and crate namespaces colliding maybe?

  I think we need to have a plan for supporting the case of people who aren't caching their dependencies on their local server, and who do want to allow publishing of crates without a prefix as you've proposed, but also want to be able to depend on crates on crates.io that might have conflicting names.

• Regarding the "Index files changes" section, this part:

  As Cargo requires the index file to include all the dependencies, the crates.io index file format is updated to include the registry in the dependency. The registry is an optional field, where by default it is None, and will only be set when using an alternate crate server.

  sounds like you're saying people will be able to publish crates to crates.io that depend on crates that come from other registries. I don't think this is something we want to allow; we want to guarantee that all crates on crates.io will build from only the information that crates.io hosts. Is that what you're saying or am I misunderstanding?

• Regarding the Blocking requests to push to a registry section, this part:

  the first is a change to Cargo which checks if the registry provided in the registry matches the host for the publish, if it does not it gets rejected

  In your proposed change to the Cargo.toml, you have a registry specified for the current crate. Couldn't cargo publish then just use that registry value and only publish to that registry? I think it would be convenient if I didn't have to specify an --index when running cargo publish if I've already specified the index in the Cargo.toml. Are you saying that this rejection would only happen if I explicitly specified an index when publishing?

  Also in this comment thread, you suggested allowing private registries to have a setting to allow publishing a crate without a registry set? This seems like unnecessary complexity on the server side.... I think this needs to be worked out in a different way. Why not allow multiple registries to be specified in Cargo.toml that this crate is allowed to be published to?

• Regarding the Allow publishing when referencing external dependencies section, it says:

  We still want to support private crates having dependencies on the public crates.io server, so we propose relaxing a check which ensures that the source for a dependency matches the registry.

  I understand that you're trying to separate this proposal from companies having cached versions of crates.io crates, but I think they interact here. I could see a company wanting to disallow publishes to their internal registry if the dependencies haven't been cached from crates.io to their internal registry yet.

• Regarding the "Making it easier for users using an alternate crates.io registry" section, it took me a few readings to figure out that you were talking about the fragment that gets shown on a crate's page in the web UI of the registry. Could you clarify this section please, perhaps including a screenshot with your proposed changes, to pre-empt confusion?

[carols10cents]

One thing that I mentioned to @cswindle in IRC is that this week is the Mozilla All Hands, so a bunch of us are going to all be in one place for the week and will be discussing lots of things. I plan to bring this up in order to move this forward.

Rereading the text of the RFC, I think my main issues are (listing them here rather than on comments in the RFC so that I can better track when all of these have been resolved, because I think some of my comments are getting lost):

• I think we want to move towards specifying host rather than index location everywhere. I'd rather have to put https://my-awesome-fork.com in my Cargo.tomls instead of https://github.com/my_awesome_fork/crates.io-index, for example. So I'd like to see that work get done before implementing this.

• Regarding the "Crate naming on alternate servers" section, I think I'd also like to see support in Cargo.toml for renaming a dependency, much like we can alias things with use. For example:

  [dependencies]
  libc = { version = "*" } // crates.io libc
  libc = { as="awesomefork-libc", version = "*", registry = "https://github.com/my_awesome_fork/crates.io-index" }
  

  Then everywhere within my crate, I could refer to extern crate awesomefork-libc. I know @alexcrichton brought this up recently as something that would be useful for something else, but I can't remember where or why. Feature and crate namespaces colliding maybe?

  I think we need to have a plan for supporting the case of people who aren't caching their dependencies on their local server, and who do want to allow publishing of crates without a prefix as you've proposed, but also want to be able to depend on crates on crates.io that might have conflicting names.

• Regarding the "Index files changes" section, this part:

  As Cargo requires the index file to include all the dependencies, the crates.io index file format is updated to include the registry in the dependency. The registry is an optional field, where by default it is None, and will only be set when using an alternate crate server.

  sounds like you're saying people will be able to publish crates to crates.io that depend on crates that come from other registries. I don't think this is something we want to allow; we want to guarantee that all crates on crates.io will build from only the information that crates.io hosts. Is that what you're saying or am I misunderstanding?

• Regarding the Blocking requests to push to a registry section, this part:

  the first is a change to Cargo which checks if the registry provided in the registry matches the host for the publish, if it does not it gets rejected

  In your proposed change to the Cargo.toml, you have a registry specified for the current crate. Couldn't cargo publish then just use that registry value and only publish to that registry? I think it would be convenient if I didn't have to specify an --index when running cargo publish if I've already specified the index in the Cargo.toml. Are you saying that this rejection would only happen if I explicitly specified an index when publishing?

  Also in this comment thread, you suggested allowing private registries to have a setting to allow publishing a crate without a registry set? This seems like unnecessary complexity on the server side.... I think this needs to be worked out in a different way. Why not allow multiple registries to be specified in Cargo.toml that this crate is allowed to be published to?

• Regarding the Allow publishing when referencing external dependencies section, it says:

  We still want to support private crates having dependencies on the public crates.io server, so we propose relaxing a check which ensures that the source for a dependency matches the registry.

  I understand that you're trying to separate this proposal from companies having cached versions of crates.io crates, but I think they interact here. I could see a company wanting to disallow publishes to their internal registry if the dependencies haven't been cached from crates.io to their internal registry yet.

• Regarding the "Making it easier for users using an alternate crates.io registry" section, it took me a few readings to figure out that you were talking about the fragment that gets shown on a crate's page in the web UI of the registry. Could you clarify this section please, perhaps including a screenshot with your proposed changes, to pre-empt confusion?

[vsimonian]

@carols10cents

Regarding the "Crate naming on alternate servers" section, I think I'd also like to see support in Cargo.toml for renaming a dependency, much like we can alias things with use. For example:

[dependencies]
libc = { version = "*" } // crates.io libc
libc = { as="awesomefork-libc", version = "*", registry = "https://github.com/my_awesome_fork/crates.io-index" }

Then everywhere within my crate, I could refer to extern crate awesomefork-libc.

I ❤️ this idea! I think it can be made even better by abstracting the registry URL away from the individual dependency and to its own registries section. This way you don't get a bunch of URLs floating around in the same Cargo.toml when you could instead have just one (especially useful if you use multiple packages from the same private registry).

For example:

[registries]
bizcorp = "https://code.example.com/DefaultCollection/_git/private-crates-index"

[dependencies]
libc = { version = "*" } # crates.io libc
libc = { as = "bizcorp-libc", version = "*", registry = "bizcorp" }
coffeepot-integrations = { version = "*", registry = "bizcorp" }
boss-key = { version = "*", registry = "bizcorp" }

Or, alternately:

[registries]
bizcorp = "https://code.example.com/DefaultCollection/_git/private-crates-index"

[dependencies]
libc = { version = "*" } # crates.io libc
"bizcorp/libc" = { as = "bizcorp-libc", version = "*" }
"bizcorp/coffeepot-integrations" = { version = "*" }
"bizcorp/boss-key" = { version = "*" }

And an example with multiple private registries:

[registries]
# Internal libraries
bizcorp = "https://code.example.com/DefaultCollection/_git/private-crates-index"
# Private libraries licensed to us by 3rd party vendor
software-vendor = "https://libraries.vendor.com/rust-libraries/index"

[dependencies]
libc = { version = "*" } # crates.io libc
libc = { as = "bizcorp-libc", version = "*", registry = "bizcorp" }
coffeepot-integrations = { version = "*", registry = "bizcorp" }
boss-key = { version = "*", registry = "bizcorp" }
proprietary-db-reader = { version = "*", registry = "software-vendor" }
proprietary-ui-toolkit = { version = "*", registry = "software-vendor" }

This keeps the TOML super clean and cute, and also makes it a whole lot easier to update the URL should the path to the repository change (hopefully a rare event, but still!).

[vsimonian]

@carols10cents

Regarding the "Crate naming on alternate servers" section, I think I'd also like to see support in Cargo.toml for renaming a dependency, much like we can alias things with use. For example:

[dependencies]
libc = { version = "*" } // crates.io libc
libc = { as="awesomefork-libc", version = "*", registry = "https://github.com/my_awesome_fork/crates.io-index" }

Then everywhere within my crate, I could refer to extern crate awesomefork-libc.

I ❤️ this idea! I think it can be made even better by abstracting the registry URL away from the individual dependency and to its own registries section. This way you don't get a bunch of URLs floating around in the same Cargo.toml when you could instead have just one (especially useful if you use multiple packages from the same private registry).

For example:

[registries]
bizcorp = "https://code.example.com/DefaultCollection/_git/private-crates-index"

[dependencies]
libc = { version = "*" } # crates.io libc
libc = { as = "bizcorp-libc", version = "*", registry = "bizcorp" }
coffeepot-integrations = { version = "*", registry = "bizcorp" }
boss-key = { version = "*", registry = "bizcorp" }

Or, alternately:

[registries]
bizcorp = "https://code.example.com/DefaultCollection/_git/private-crates-index"

[dependencies]
libc = { version = "*" } # crates.io libc
"bizcorp/libc" = { as = "bizcorp-libc", version = "*" }
"bizcorp/coffeepot-integrations" = { version = "*" }
"bizcorp/boss-key" = { version = "*" }

And an example with multiple private registries:

[registries]
# Internal libraries
bizcorp = "https://code.example.com/DefaultCollection/_git/private-crates-index"
# Private libraries licensed to us by 3rd party vendor
software-vendor = "https://libraries.vendor.com/rust-libraries/index"

[dependencies]
libc = { version = "*" } # crates.io libc
libc = { as = "bizcorp-libc", version = "*", registry = "bizcorp" }
coffeepot-integrations = { version = "*", registry = "bizcorp" }
boss-key = { version = "*", registry = "bizcorp" }
proprietary-db-reader = { version = "*", registry = "software-vendor" }
proprietary-ui-toolkit = { version = "*", registry = "software-vendor" }

This keeps the TOML super clean and cute, and also makes it a whole lot easier to update the URL should the path to the repository change (hopefully a rare event, but still!).

[cswindle]

@carols10cents, thanks for the detailed response. Below I have gone through each of them one by one, if you could let me know if you are happy with the responses and I will update the document to reflect.

Rereading the text of the RFC, I think my main issues are (listing them here rather than on comments in the RFC so that I can better track when all of these have been resolved, because I think some of my comments are getting lost:
• I think we want to move towards specifying host rather than index location everywhere. I'd rather have to put https://my-awesome-fork.com in my Cargo.tomls instead of https://github.com/my_awesome_fork/crates.io-index, for example. So I'd like to see that work get done before implementing this.

When is this likely to be implemented? If it is going to be a while (which I can imagine it may be), then an alternative could be to update to use the host in the registry fields, then do a query to see what the crates.io index is when required. That way the interface will be correct, even if the backend won’t be. I do not envisage that this will create much (if any) additional work for whoever fixes the issue you linked to. Would you like me to make the change to the RFC?

I have now updated the document to switch to use the server URL.

• Regarding the "Crate naming on alternate servers" section, I think I'd also like to see support in Cargo.toml for renaming a dependency, much like we can alias things with use. For example:
[dependencies]
libc = { version = "" } // crates.io libc
libc = { as="awesomefork-libc", version = "", registry = "https://github.com/my_awesome_fork/crates.io-index" }
Then everywhere within my crate, I could refer to extern crate awesomefork-libc. I know @alexcrichton brought this up recently as something that would be useful for something else, but I can't remember where or why. Feature and crate namespaces colliding maybe?
I think we need to have a plan for supporting the case of people who aren't caching their dependencies on their local server, and who do want to allow publishing of crates without a prefix as you've proposed, but also want to be able to depend on crates on crates.io that might have conflicting names.

This was brought up by @alexcrichton in this thread (#2006 (comment)), I responded that this is something that I think is actually worthy of its own RFC as I think there will be people who would find this useful, for git repos for example, that would not notice this RFC. I am willing to write a new RFC if nobody else proposes it, however I will not be able to get around to doing this until around October due to work commitments. Does that seem reasonable?

• Regarding the "Index files changes" section, this part:
As Cargo requires the index file to include all the dependencies, the crates.io index file format is updated to include the registry in the dependency. The registry is an optional field, where by default it is None, and will only be set when using an alternate crate server.
sounds like you're saying people will be able to publish crates to crates.io that depend on crates that come from other registries. I don't think this is something we want to allow; we want to guarantee that all crates on crates.io will build from only the information that crates.io hosts. Is that what you're saying or am I misunderstanding?

No, I was meaning for a private crates.io server it would allow external dependencies, but not for sending to crates.io, which I agree should never reference an external dependency and this should be blocked on the server. I will update the RFC to clarify.

I have now updated the document to include this change.

• Regarding the Blocking requests to push to a registry section, this part:
the first is a change to Cargo which checks if the registry provided in the registry matches the host for the publish, if it does not it gets rejected
In your proposed change to the Cargo.toml, you have a registry specified for the current crate. Couldn't cargo publish then just use that registry value and only publish to that registry? I think it would be convenient if I didn't have to specify an --index when running cargo publish if I've already specified the index in the Cargo.toml. Are you saying that this rejection would only happen if I explicitly specified an index when publishing?

Good point, I will update to get the host automatically from Cargo.toml (although the token would still need to be provided until we have support for multiple tokens stored).

I have now updated the document to include this change.

Also in this comment thread, you suggested allowing private registries to have a setting to allow publishing a crate without a registry set? This seems like unnecessary complexity on the server side.... I think this needs to be worked out in a different way. Why not allow multiple registries to be specified in Cargo.toml that this crate is allowed to be published to?

This would just be allowing the private registry to use the same code path which is used for crates.io (which does allow a crate to be published with no registry). Also in this case I am pretty sure that this can be done today by just overriding the registry. I am not keen on specifying multiple registries as I think that having a single registry is more sensible as I personally feel it would lead to more confusion allowing multiple registries.

• Regarding the Allow publishing when referencing external dependencies section, it says:
We still want to support private crates having dependencies on the public crates.io server, so we propose relaxing a check which ensures that the source for a dependency matches the registry.
I understand that you're trying to separate this proposal from companies having cached versions of crates.io crates, but I think they interact here. I could see a company wanting to disallow publishes to their internal registry if the dependencies haven't been cached from crates.io to their internal registry yet.

Maybe this could be a config flag which is set for crates.io by default and private registries can override this to allow external dependencies, that way it is common code for crates.io and the private registries which want to just look like an internal crates.io, would you be happy with that change?

• Regarding the "Making it easier for users using an alternate crates.io registry" section, it took me a few readings to figure out that you were talking about the fragment that gets shown on a crate's page in the web UI of the registry. Could you clarify this section please, perhaps including a screenshot with your proposed changes, to pre-empt confusion?

I will update the RFC to include a diagram to clarify.

I have now updated the document to include this change.

Hopefully that deals with all of your queries.

[cswindle]

@carols10cents, thanks for the detailed response. Below I have gone through each of them one by one, if you could let me know if you are happy with the responses and I will update the document to reflect.

Rereading the text of the RFC, I think my main issues are (listing them here rather than on comments in the RFC so that I can better track when all of these have been resolved, because I think some of my comments are getting lost:
• I think we want to move towards specifying host rather than index location everywhere. I'd rather have to put https://my-awesome-fork.com in my Cargo.tomls instead of https://github.com/my_awesome_fork/crates.io-index, for example. So I'd like to see that work get done before implementing this.

When is this likely to be implemented? If it is going to be a while (which I can imagine it may be), then an alternative could be to update to use the host in the registry fields, then do a query to see what the crates.io index is when required. That way the interface will be correct, even if the backend won’t be. I do not envisage that this will create much (if any) additional work for whoever fixes the issue you linked to. Would you like me to make the change to the RFC?

I have now updated the document to switch to use the server URL.

• Regarding the "Crate naming on alternate servers" section, I think I'd also like to see support in Cargo.toml for renaming a dependency, much like we can alias things with use. For example:
[dependencies]
libc = { version = "" } // crates.io libc
libc = { as="awesomefork-libc", version = "", registry = "https://github.com/my_awesome_fork/crates.io-index" }
Then everywhere within my crate, I could refer to extern crate awesomefork-libc. I know @alexcrichton brought this up recently as something that would be useful for something else, but I can't remember where or why. Feature and crate namespaces colliding maybe?
I think we need to have a plan for supporting the case of people who aren't caching their dependencies on their local server, and who do want to allow publishing of crates without a prefix as you've proposed, but also want to be able to depend on crates on crates.io that might have conflicting names.

This was brought up by @alexcrichton in this thread (#2006 (comment)), I responded that this is something that I think is actually worthy of its own RFC as I think there will be people who would find this useful, for git repos for example, that would not notice this RFC. I am willing to write a new RFC if nobody else proposes it, however I will not be able to get around to doing this until around October due to work commitments. Does that seem reasonable?

• Regarding the "Index files changes" section, this part:
As Cargo requires the index file to include all the dependencies, the crates.io index file format is updated to include the registry in the dependency. The registry is an optional field, where by default it is None, and will only be set when using an alternate crate server.
sounds like you're saying people will be able to publish crates to crates.io that depend on crates that come from other registries. I don't think this is something we want to allow; we want to guarantee that all crates on crates.io will build from only the information that crates.io hosts. Is that what you're saying or am I misunderstanding?

No, I was meaning for a private crates.io server it would allow external dependencies, but not for sending to crates.io, which I agree should never reference an external dependency and this should be blocked on the server. I will update the RFC to clarify.

I have now updated the document to include this change.

• Regarding the Blocking requests to push to a registry section, this part:
the first is a change to Cargo which checks if the registry provided in the registry matches the host for the publish, if it does not it gets rejected
In your proposed change to the Cargo.toml, you have a registry specified for the current crate. Couldn't cargo publish then just use that registry value and only publish to that registry? I think it would be convenient if I didn't have to specify an --index when running cargo publish if I've already specified the index in the Cargo.toml. Are you saying that this rejection would only happen if I explicitly specified an index when publishing?

Good point, I will update to get the host automatically from Cargo.toml (although the token would still need to be provided until we have support for multiple tokens stored).

I have now updated the document to include this change.

Also in this comment thread, you suggested allowing private registries to have a setting to allow publishing a crate without a registry set? This seems like unnecessary complexity on the server side.... I think this needs to be worked out in a different way. Why not allow multiple registries to be specified in Cargo.toml that this crate is allowed to be published to?

This would just be allowing the private registry to use the same code path which is used for crates.io (which does allow a crate to be published with no registry). Also in this case I am pretty sure that this can be done today by just overriding the registry. I am not keen on specifying multiple registries as I think that having a single registry is more sensible as I personally feel it would lead to more confusion allowing multiple registries.

• Regarding the Allow publishing when referencing external dependencies section, it says:
We still want to support private crates having dependencies on the public crates.io server, so we propose relaxing a check which ensures that the source for a dependency matches the registry.
I understand that you're trying to separate this proposal from companies having cached versions of crates.io crates, but I think they interact here. I could see a company wanting to disallow publishes to their internal registry if the dependencies haven't been cached from crates.io to their internal registry yet.

Maybe this could be a config flag which is set for crates.io by default and private registries can override this to allow external dependencies, that way it is common code for crates.io and the private registries which want to just look like an internal crates.io, would you be happy with that change?

• Regarding the "Making it easier for users using an alternate crates.io registry" section, it took me a few readings to figure out that you were talking about the fragment that gets shown on a crate's page in the web UI of the registry. Could you clarify this section please, perhaps including a screenshot with your proposed changes, to pre-empt confusion?

I will update the RFC to include a diagram to clarify.

I have now updated the document to include this change.

Hopefully that deals with all of your queries.

[cswindle]

@vsimonian, this is meant to be the minimal work to allow progress to be made on private registries. Having a registry section was already discussed and I proposed that is something that should not be part of this RFC and should instead be proposed as an extension in a subsequent RFC as I think that the notation that you propose can happily work alongside the way I propose.

[cswindle]

@vsimonian, this is meant to be the minimal work to allow progress to be made on private registries. Having a registry section was already discussed and I proposed that is something that should not be part of this RFC and should instead be proposed as an extension in a subsequent RFC as I think that the notation that you propose can happily work alongside the way I propose.

[vsimonian]

@cswindle Thanks for the heads up! I spent last night reading all I could find related to this discussion, but I guess I missed that.

[vsimonian]

@cswindle Thanks for the heads up! I spent last night reading all I could find related to this discussion, but I guess I missed that.

[alexcrichton]

I personally feel like there's two crucial points to fix before landing this RFC:

• Today there's no ability for unstable features to work in Cargo. This is such a big feature I think we need to finally bite the bullet and implement the ability to have unstable features in Cargo. This itself may be its own RFC or its own discussion thread (doesn't need to happen here), but I wouldn't want to land an implementation of this today unless it were unstable.

• Next, I find the usage of URLs here to be "the wrong URL". The index of a registry is an internal implementation detail that I don't think should ever be surfaced. The fact that it's "exposed" today in a few commands is one that'll likely get fixed/removed in later versions of Cargo. In my mind the "correct" URL for crates.io is 'https://crates.io' or something like that. Basically I don't think we've got a great way to identify registries right now. Using an index is an ergonomic nightmare and otherwise we don't have an option on the table. I think this'll need some thinking to figure out the best option here.

[alexcrichton]

I personally feel like there's two crucial points to fix before landing this RFC:

• Today there's no ability for unstable features to work in Cargo. This is such a big feature I think we need to finally bite the bullet and implement the ability to have unstable features in Cargo. This itself may be its own RFC or its own discussion thread (doesn't need to happen here), but I wouldn't want to land an implementation of this today unless it were unstable.

• Next, I find the usage of URLs here to be "the wrong URL". The index of a registry is an internal implementation detail that I don't think should ever be surfaced. The fact that it's "exposed" today in a few commands is one that'll likely get fixed/removed in later versions of Cargo. In my mind the "correct" URL for crates.io is 'https://crates.io' or something like that. Basically I don't think we've got a great way to identify registries right now. Using an index is an ergonomic nightmare and otherwise we don't have an option on the table. I think this'll need some thinking to figure out the best option here.

[cswindle]

@alexcrichton, I have updated the document to include switching to URL. Regarding the unstable feature, is that due to the fact that when people start experimenting with it you might want to tweak the interface, thus wanting to discourage widespread use of the feature? Is the discussion/RFC something that you/someone on the Rust team would be willing to propose as I will not have the time required to commit to get agreement on how it should be implemented.

@carols10cents, there are a few of your comments addressed in the changes I have just pushed. I will update the comment above to indicate which are updated to make it easier for you to see what feedback is still requiring some discussions.

[cswindle]

@alexcrichton, I have updated the document to include switching to URL. Regarding the unstable feature, is that due to the fact that when people start experimenting with it you might want to tweak the interface, thus wanting to discourage widespread use of the feature? Is the discussion/RFC something that you/someone on the Rust team would be willing to propose as I will not have the time required to commit to get agreement on how it should be implemented.

@carols10cents, there are a few of your comments addressed in the changes I have just pushed. I will update the comment above to indicate which are updated to make it easier for you to see what feedback is still requiring some discussions.

[alexcrichton]

Thanks for the updates @cswindle. The method of switching URLs to the "right one" specified locally here makes sense here and is a plausible way forward. It's a pretty big change to how Cargo works internally and can add latency when updating the registry, so we'll want to discuss the exact implementation over time (doesn't need to be 100% hammered out here though).

And yes we may be able to work on unstable features. I mostly wanted to point out that I would be uncomfortable landing this in Cargo before we have unstable features. Having this be unstable allows us flexibility to tweak the design as necessary without worrying about breaking changes. This is such a large addition that we're inevitable going to learn something during an implementation and deployment that we didn't think about before.

[alexcrichton]

Thanks for the updates @cswindle. The method of switching URLs to the "right one" specified locally here makes sense here and is a plausible way forward. It's a pretty big change to how Cargo works internally and can add latency when updating the registry, so we'll want to discuss the exact implementation over time (doesn't need to be 100% hammered out here though).

And yes we may be able to work on unstable features. I mostly wanted to point out that I would be uncomfortable landing this in Cargo before we have unstable features. Having this be unstable allows us flexibility to tweak the design as necessary without worrying about breaking changes. This is such a large addition that we're inevitable going to learn something during an implementation and deployment that we didn't think about before.

[cswindle]

@alexcrichton, does that mean that you think that the proposal is fine to go to FCP, albeit the implementation after that is blocked based on your desire to have unstable features?

[cswindle]

@alexcrichton, does that mean that you think that the proposal is fine to go to FCP, albeit the implementation after that is blocked based on your desire to have unstable features?

[alexcrichton]

I believe that @carols10cents was going to shepherd this RFC, so I'll leave that up to her.

[alexcrichton]

I believe that @carols10cents was going to shepherd this RFC, so I'll leave that up to her.

[wycats]

I'd like to raise a concern that hasn't been raised yet (as far as I can tell).

I personally really want multiple registries (and mirrors) as a public API. I've been motivated about this from Day 1 (which is why the internals are so amenable to this kind of change).

However, in order to make alternate registries a public API, we really do need to make the registry API and format, as used by Cargo, a public API. This means versioning the git repository format, and formalizing its contents in a compatible way.

In particular, while the registry has been scaling well so far, it's unclear that the current sharding strategy will work in perpetuity as registries get much larger. I personally feel very strongly that incremental updates and purely local dependency resolution are important features, which is why I went with a git repository in the first place. That said, I (in the early days) and we (more recently) never worked on stabilizing the current format.

I'm not saying this is extremely difficult, but rather that the work of stabilizing the registry format (rather than just assuming the current version is defacto public, as this RFC does), is the bulk of what I expect to be difficult about this RFC.

@cswindle are you interested in doing the work to expand this RFC to formalize the details of how Cargo interacts with registries?

[wycats]

I'd like to raise a concern that hasn't been raised yet (as far as I can tell).

I personally really want multiple registries (and mirrors) as a public API. I've been motivated about this from Day 1 (which is why the internals are so amenable to this kind of change).

However, in order to make alternate registries a public API, we really do need to make the registry API and format, as used by Cargo, a public API. This means versioning the git repository format, and formalizing its contents in a compatible way.

In particular, while the registry has been scaling well so far, it's unclear that the current sharding strategy will work in perpetuity as registries get much larger. I personally feel very strongly that incremental updates and purely local dependency resolution are important features, which is why I went with a git repository in the first place. That said, I (in the early days) and we (more recently) never worked on stabilizing the current format.

I'm not saying this is extremely difficult, but rather that the work of stabilizing the registry format (rather than just assuming the current version is defacto public, as this RFC does), is the bulk of what I expect to be difficult about this RFC.

@cswindle are you interested in doing the work to expand this RFC to formalize the details of how Cargo interacts with registries?

[cswindle]

@wycats, I can see how having the interfaces being public would make sense, however I am unable to commit to that as part of this RFC. The whole point of this RFC is to get something in place which can be built upon, the fact that @alexcrichton is wanting this to be unstable for a period of time, then maybe a subsequent RFC could deal with making the API public and you could have that as one of the criteria for the feature to become stable. Does that sound ok to you?

[cswindle]

@wycats, I can see how having the interfaces being public would make sense, however I am unable to commit to that as part of this RFC. The whole point of this RFC is to get something in place which can be built upon, the fact that @alexcrichton is wanting this to be unstable for a period of time, then maybe a subsequent RFC could deal with making the API public and you could have that as one of the criteria for the feature to become stable. Does that sound ok to you?

[bbatha]

I just wanted to voice my concern about this part of the proposed change:

I think we want to move towards specifying host rather than index location everywhere. I'd rather have to put https://my-awesome-fork.com in my Cargo.tomls instead of https://github.com/my_awesome_fork/crates.io-index, for example. So I'd like to see that work get done before implementing this.

For multiple repository hosting solutions like nexus and artifactory it needs to be possible to specify a path as well. For instance, artifactory hosts npm repos at https://host.company.com/api/npm/private-repo so you can host multiple repo types and multiple repos for the same language. Specifying just the host should have a good default but it should be overridable.

[bbatha]

I just wanted to voice my concern about this part of the proposed change:

I think we want to move towards specifying host rather than index location everywhere. I'd rather have to put https://my-awesome-fork.com in my Cargo.tomls instead of https://github.com/my_awesome_fork/crates.io-index, for example. So I'd like to see that work get done before implementing this.

For multiple repository hosting solutions like nexus and artifactory it needs to be possible to specify a path as well. For instance, artifactory hosts npm repos at https://host.company.com/api/npm/private-repo so you can host multiple repo types and multiple repos for the same language. Specifying just the host should have a good default but it should be overridable.

[carols10cents]

@bbatha could you add that use case to rust-lang/cargo#4208 please?

[carols10cents]

@bbatha could you add that use case to rust-lang/cargo#4208 please?

[bbatha]

@carols10cents added there as well.

[bbatha]

@carols10cents added there as well.

[joshtriplett]

@wycats While I do see value in stabilizing the repository format itself, would it also help to define a library for both reading and writing it?

[joshtriplett]

@wycats While I do see value in stabilizing the repository format itself, would it also help to define a library for both reading and writing it?

[cswindle]

@wycats, I have updated the RFC to include that a public API is required in order to be used in a stable build, as discussed on IRC. Does that cover what you were wanting?

[cswindle]

@wycats, I have updated the RFC to include that a public API is required in order to be used in a stable build, as discussed on IRC. Does that cover what you were wanting?

[aturon]

Nominating for discussion at the next Cargo team meeting.

[aturon]

Nominating for discussion at the next Cargo team meeting.

[carols10cents]

Just realized we never posted the decision made in the Cargo team meeting here-- we decided the specification of the index format was important, so we created a new RFC that contains that spec.

[carols10cents]

Just realized we never posted the decision made in the Cargo team meeting here-- we decided the specification of the index format was important, so we created a new RFC that contains that spec.