Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC: Packages as (optional) namespaces #3243

Merged
merged 36 commits into from
Mar 11, 2024
Merged
Changes from 3 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
e590c72
Copy over RFC text from https://github.com/Manishearth/namespacing-rfc/
Manishearth Mar 9, 2022
5e6ad7c
Fix template
Manishearth Mar 9, 2022
d9d1556
Add basic FAQ
Manishearth Mar 9, 2022
e29aa55
Add more separator options
Manishearth Mar 9, 2022
3a0b099
leaf crate names
Manishearth Mar 10, 2022
4422adf
sep choice
Manishearth Mar 10, 2022
24db227
Clarify motivation on projects vs organizations
Manishearth Mar 10, 2022
1986c3d
Switch to colons
Manishearth Mar 25, 2022
42046f8
fixes
Manishearth Mar 26, 2022
2fff101
Document Python prior art
epage Mar 26, 2022
8e667ce
Expand on the guide-level explanation
epage Mar 26, 2022
01d37b2
Document feature flags drawbacks
epage Mar 26, 2022
4501047
Merge pull request #4 from epage/guide
Manishearth Mar 27, 2022
1f62468
Merge pull request #3 from epage/prior
Manishearth Mar 27, 2022
1ca3e62
Merge pull request #5 from epage/flags
Manishearth Mar 27, 2022
f7be349
Fix a couple of typos
epage Mar 28, 2022
9a144fb
Updaet motivation for `::` semantics
epage Mar 28, 2022
a578a67
Merge pull request #7 from epage/motivation
Manishearth Mar 28, 2022
3396a2e
Merge pull request #6 from epage/typos
Manishearth Mar 28, 2022
11c6354
Update summary to focus on `::` semantics
epage Mar 29, 2022
cc5872d
Merge pull request #8 from epage/summary
Manishearth Mar 29, 2022
65f7083
add renames as unresolved q
Manishearth Mar 29, 2022
5c497c5
fix syntax
Manishearth May 24, 2023
67ef913
fixes
Manishearth May 24, 2023
56fae2e
trie-unresolved
Manishearth May 24, 2023
482fae8
Update 0000-packages-as-optional-namespaces.md
Manishearth Jul 23, 2023
4c14f9c
Update text/0000-packages-as-optional-namespaces.md
Manishearth Nov 4, 2023
d9a3c90
Update text/0000-packages-as-optional-namespaces.md
Manishearth Nov 4, 2023
8bd7fbc
Add .crate file as unresolved
epage Nov 13, 2023
abde93f
Be explicit that open questions are deferred out
epage Nov 13, 2023
7d77485
Call out distributions
epage Nov 13, 2023
75dd867
Merge pull request #9 from epage/unresolved
Manishearth Nov 13, 2023
0db096f
Update text/0000-packages-as-optional-namespaces.md
Manishearth Mar 11, 2024
6cc8886
Rename 0000-packages-as-optional-namespaces.md to 3243-packages-as-op…
oli-obk Mar 11, 2024
a93b1e4
Add RFC PR link
oli-obk Mar 11, 2024
656203f
Link to rust tracking issue
oli-obk Mar 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
175 changes: 175 additions & 0 deletions text/0000-packages-as-optional-namespaces.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,175 @@
- Feature Name: `packages_as_namespaces`
- Start Date: (fill me in with today's date, 2022-03-09)
- RFC PR: [rust-lang/rfcs#0000](https://github.com/rust-lang/rfcs/pull/0000)
- Rust Issue: [rust-lang/rust#0000](https://github.com/rust-lang/rust/issues/0000)

# Summary

Grant exclusive access to publishing crates `parent/foo` for owners of crate `parent`.
Manishearth marked this conversation as resolved.
Show resolved Hide resolved

Namespaced crates can be named in Rust code using underscores (e.g. `parent_foo`).

# Motivation

While Rust crates are practically unlimited in size, it is a common pattern for organizations to split their projects into many crates, especially if they expect users to only need a fraction of their crates.

For example, [unic](https://crates.io/search?page=1&per_page=10&q=unic-), [tokio](https://crates.io/search?page=1&per_page=10&q=tokio-), [async-std](https://crates.io/search?page=1&per_page=10&q=async-), [rusoto](https://crates.io/search?q=rusoto) all do something like this, with lots of `projectname-foo` crates. At the moment, it is not necessarily true that a crate named `projectname-foo` is maintained by `projectname`, and in some cases that is even desired! E.g. `serde` has many third party "plugin" crates like [serde-xml-rs](https://github.com/RReverser/serde-xml-rs). Similarly, [async-tls](https://crates.io/crates/async-tls) is a general crate not specific to the async-std ecosystem.

Regardless, it is nice to have a way to signify "these are all crates belonging to a single organization, and you may trust them the same". When starting up [ICU4X](https://github.com/unicode-org/icu4x/), we came up against this problem: We wanted to be able to publish ICU4X as an extremely modular system of `icu-foo` or `icu4x-foo` crates, but it would be confusing to users if third-party crates could also exist there (or take names we wanted to use).

This is distinct from the general problem of squatting -- with general squatting, someone else might come up with a cool crate name before you do. However, with `projectname-foo` crates, it's more of a case of third parties "muscling in" on a name you have already chosen and are using.

# Guide-level explanation

If you own a crate `foo`, you may create a crate namespaced under it as `foo/bar`. Only people who are owners of `foo` may _create_ a crate `foo/bar` (and all owners of `foo` are implicitly owners of `foo/bar`). After such a crate is created, additional per-crate publishers may be added who will be able to publish subsequent versions as usual.

The crate can be imported in Cargo.toml using its name as normal:

```toml
[dependencies]
"foo/bar" = "1.0"
```


In Rust code, the slash gets converted to an underscore, the same way we do this for dashes.

```rs
use foo_bar::Baz;
```

# Reference-level explanation

`/` is now considered a valid identifier inside a crate name Crates.io. For now, we will restrict crate names to having a single `/` in them, not at the beginning or end of the name, but this can be changed in the future.

When publishing a crate `foo/bar`, if the crate does not exist, the following must be true:

- `foo` must exist
- The user publishing the crate must be an owner of `foo`

For the crate `foo/bar`, all owners of `foo` are always considered owners of `foo/bar`, however additional owners may be added. People removed from ownership of `foo` will also lose access to `foo/bar` unless they were explicitly added as owners to `foo/bar`.

Crates.io displays `foo/bar` crates with the name `foo/bar`, though it may stylistically make the `foo` part link to the `foo` crate.

The [registry index trie](https://doc.rust-lang.org/nightly/cargo/reference/registries.html#index-format) may represent subpackages by placing `foo/bar` in `foo@/bar`, placed next to where `foo` is in the trie (i.e. the full path will be `fo/foo@/bar`).

No changes are made to `rustc`. When compiling a crate `foo/bar`, Cargo will automatically pass in `--crate-name foo_bar`, and when referring to it as a dependency Cargo will use `--extern foo_bar=....`. This is the same thing we currently do for `foo-bar`.
Manishearth marked this conversation as resolved.
Show resolved Hide resolved

If you end up in a situation where you have both `foo/bar` and `foo-bar` as active dependencies of your crate, your code will not compile and you must [rename](https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#renaming-dependencies-in-cargotoml) one of them.

The `features = ` key in Cargo.toml continues parsing `foo/bar` as "the feature `bar` on dependency `foo`", however it now will unambiguously parse strings ending with a slash (`foo/` and `foo/bar/`) as referring to a dependency, as opposed to feature on a dependency. Cargo may potentially automatically handle the ambiguity or error about it.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another option for disambiguation here could be namespaced features. If namespaced crates did not automatically create an unnamespaced feature like current optional dependencies do, then you would have to use dep:foo/bar to refer to it and foo/bar would unambiguously refer to the feature bar on crate foo.


# Drawbacks

## Slashes
So far slashes as a "separator" have not existed in Rust. There may be dissonance with having another non-identifier character allowed on crates.io but not in Rust code. Dashes are already confusing for new users. Some of this can be remediated with appropriate diagnostics on when `/` is encountered at the head of a path.


Furthermore, slashes are ambiguous in feature specifiers (though a solution has been proposed above for this):
epage marked this conversation as resolved.
Show resolved Hide resolved

```toml
[dependencies]
"foo" = "1"
"foo/std" = { version = "1", optional = true }

[features]
# Does this enable crate "foo/std", or feature "std" of crate "foo"?
default = ["foo/std"]
```


## Namespace root taken
Not all existing projects can transition to using namespaces here. For example, the `unicode` crate is reserved, so `unicode-rs` cannot use it as a namespace despite owning most of the `unicode-foo` crates. In other cases, the "namespace root" `foo` may be owned by a different set of people than the `foo-bar` crates, and folks may need to negotiate (`async-std` has this problem, it manages `async-foo` crates but the root `async` crate is taken by someone else). Nobody is forced to switch to using namespaces, of course, so the damage here is limited, but it would be _nice_ for everyone to be able to transition.


## Dash typosquatting

This proposal does not prevent anyone from taking `foo-bar` after you publish `foo/bar`. Given that the Rust crate import syntax for `foo/bar` is `foo_bar`, same as `foo-bar`, it's totally possible for a user to accidentally type `foo-bar` in `Cargo.toml` instead of `foo/bar`, and pull in the wrong, squatted, crate.

We currently prevent `foo-bar` and `foo_bar` from existing at the same time. We _could_ do this here as well, but it would only go in one direction: if `foo/bar` exists, neither `foo-bar` nor `foo_bar` will be allowed to be published. However, if `foo-bar` or `foo_bar` exist, we would choose to allow `foo/bar` to be published, because we don't want to limit the use of names within a crate namespace due to crates outside the namespace existing. This limits the "damage" to cases where someone pre-squats `foo-bar` before you publish `foo/bar`, and the damage can be mitigated by checking to see if such a clashing crate exists when publishing, if you actually care about this attack vector. There are some tradeoffs there that we would have to explore.

One thing that could mitigate `foo/bar` mapping to the potentially ambiguous `foo_bar` is using something like `foo::crate::bar` or `~foo::bar` or `foo::/bar` in the import syntax.
Manishearth marked this conversation as resolved.
Show resolved Hide resolved


## Slow migration

Existing projects wishing to use this may need to manually migrate. For example, `unic-langid` may become `unic/langid`, with the `unic` project maintaining `unic-langid` as a reexport crate with the same version number. Getting people to migrate might be a bit of work, and furthermore maintaining a reexport crate during the (potentially long) transition period will also be some work. Of course, there is no obligation to maintain a transition crate, but users will stop getting updates if you don't.

A possible path forward is to enable people to register aliases, i.e. `unic-langid` is an alias for `unic/langid`.
Manishearth marked this conversation as resolved.
Show resolved Hide resolved

# Rationale and alternatives

This change solves the ownership problem in a way that can be slowly transitioned to for most projects.

## `foo::bar` on crates.io and in Rust

While I cover a bunch of different separator choices below, I want to call out `foo::bar` in particular. If we went with `foo::bar`, we could have the same crate name in the Rust source and Cargo manifest. This would be _amazing_.

Except, of course, crate `foo::bar` is ambiguous with module `bar` in crate `foo` (which might actually be a reexport of `foo::bar` in some cases).

This can still be made to work, e.g. we could use `foo::crate::bar` to disambiguate, and encourage namespace-using crates to ensure that `mod bar` in crate `foo` either doesn't exist or is a reexport of crate `foo::bar`. I definitely want to see this discussed a bit more.
Manishearth marked this conversation as resolved.
Show resolved Hide resolved

## Separator choice

A different separator might make more sense.

We could perhaps have `foo-*` get autoreserved if you publish `foo`, as outlined in https://internals.rust-lang.org/t/pre-rfc-hyper-minimalist-namespaces-on-crates-io/13041 . I find that this can lead to unfortunate situations where a namespace traditionally used by one project (e.g. `async-*`) is suddenly given over to a different project (the `async` crate). Furthermore, users cannot trust `foo-bar` to be owned by `foo` because the vast number of grandfathered crates we will have.

Another separator idea would be to use `::`, e.g. `foo::bar`. This looks _great_ in Rust code, provided that the parent crate is empty and does not also have a `bar` module. See the section above for more info.
Manishearth marked this conversation as resolved.
Show resolved Hide resolved

Triple colons could work. People might find it confusing, but `foo:::bar` evokes Rust paths without being ambiguous.

We could use `~` which enables Rust code to directly name namespaced packages (as `~` is no longer used in any valid Rust syntax). It looks extremely weird, however.
Manishearth marked this conversation as resolved.
Show resolved Hide resolved

We could use dots (`foo.bar`). This does evoke some similarity with Rust syntax, however there are ambiguities: `foo.bar` in Rust code could either mean "the field `bar` of local/static `foo`" or it may mean "the crate `foo.bar`".

Note that unquoted dots have semantic meaning in TOML, and allowing for unquoted dots would freeze the list of dependency subfields allowed (to `version`, `git`, `branch`, `features`, etc).

## Separator mapping

The proposal suggests mapping `foo/bar` to `foo_bar`, but as mentioned in the typosquatting section, this has problems. There may be other mappings that work out better:

- `foo::bar` (see section above)
- `foo::crate::bar`
- `foo::/bar`
- `~foo::bar`

and the like.


## User / org namespaces

Another way to handle namespacing is to rely on usernames and GitHub orgs as namespace roots. This ties `crates.io` strongly to Github -- currently while GitHub is the only login method, there is nothing preventing others from being added.

Furthermore, usernames are not immutable, and that can lead to a whole host of issues.

## Registry trie format

Instead of placing `foo/bar` in `foo@/bar`, it can be placed in `foo@bar` or something else.

# Prior art
Copy link
Member

@scottmcm scottmcm Sep 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional prior art: Microsoft had to add https://learn.microsoft.com/en-us/nuget/nuget-org/id-prefix-reservation to NuGet, since otherwise a ton of people were publishing things under Microsoft. and System..

One interesting thing it allows is delegation of subnamespaces. If applied here, that could mean that if serde- were reserved then they could delegate serde-json to a particular team, if they wanted to curate-but-not-own some of the sub-packages.

EDIT: Not intending to say that's the model this should use—I haven't formed a position yet—just think it's good prior art for the RFC.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels like a bad argument since C# works pretty differently for packages, with the difference between top-level packages and modules not being as clear. Systems which use domain-style namespaces are a different story, since for example it's not clear whether Category.Thing is a single dependency by itself, or if Thing is part of the Category dependency.

In Rust, the top-level name is the dependency, period. So, you can be confident that serde::json is a json module within the serde crate, and serde_json is a crate by itself. Reserving the serde_ prefix for all crates feels like it's solving a problem that the solution created, since the only notion of ownership is for the root serde crate itself.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the problem is that in rust a crate can't be splitted while still indicating verifiable common ownership of the resulting cluster of crates, and that it will also stay that way.


This proposal is basically the same as https://internals.rust-lang.org/t/pre-rfc-packages-as-namespaces/8628 and https://internals.rust-lang.org/t/pre-rfc-idea-cratespaces-crates-as-namespace-take-2-or-3/11320 .

Namespacing has been discussed in https://internals.rust-lang.org/t/namespacing-on-crates-io/8571 , https://internals.rust-lang.org/t/pre-rfc-domains-as-namespaces/8688, https://internals.rust-lang.org/t/pre-rfc-user-namespaces-on-crates-io/12851 , https://internals.rust-lang.org/t/pre-rfc-hyper-minimalist-namespaces-on-crates-io/13041 , https://internals.rust-lang.org/t/blog-post-no-namespaces-in-rust-is-a-feature/13040/4 , https://internals.rust-lang.org/t/crates-io-package-policies/1041/37, https://internals.rust-lang.org/t/crates-io-squatting/8031, and many others.

# Unresolved questions

- Is `/` really the separator we wish to use?
- How do we avoid ambiguity in feature syntax
- Is there a way to avoid `foo/bar` turning in to the potentially ambiguous `foo_bar`?
- Can we mitigate some of typosquatting?
- How can we represent namespaced crates in the registry trie?
Manishearth marked this conversation as resolved.
Show resolved Hide resolved

# Future possibilities

We can allow multiple layers of nesting if people want it.

# FAQ

## What if I don't want to publish my crate under a namespace?

You don't have to, namespaces are completely optional when publishing.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't everything be grandfathered into a default namespace like crates::serde?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, because the current set of crates is the top level namespace


## Does this stop people from squatting on `coolcratename`?

No, this proposal does not intend to address the general problem of squatting (See [crates.io's policy](https://crates.io/policies#squatting), a lot of this has been discussed many times before). Instead, it allows people who own an existing crate to publish sub-crates under the same namespace. In other words, if you own `coolcratename`, it stops people from squatting `coolcratename/derive`.
Manishearth marked this conversation as resolved.
Show resolved Hide resolved