Decision: Separator choice

[Manishearth]

Related: #2

The current RFC lists slashes as the separator choice. They have some drawbacks

So far slashes as a "separator" have not existed in Rust. There may be dissonance with having another identifier character allowed on crates.io but not in Rust code. Dashes are already confusing for new users. Some of this can be remediated with appropriate diagnostics on when / is encountered at the head of a path.

Furthermore, slashes are ambiguous in feature specifiers:

[dependencies]
"foo" = "1"
"foo/std" = { version = "1", optional = true }

[features]
# Does this enable crate "foo/std", or feature "std" of crate "foo"?
default = ["foo/std"]

There are many other alternatives we could pick:

We could perhaps have foo-* get autoreserved if you publish foo, as outlined in https://internals.rust-lang.org/t/pre-rfc-hyper-minimalist-namespaces-on-crates-io/13041 . I find that this can lead to unfortunate situations where a namespace traditionally used by one project (e.g. async-*) is suddenly given over to a different project (the async crate). Furthermore, users cannot trust foo-bar to be owned by foo because the vast number of grandfathered crates we will have.

Another separator idea would be to use ::, e.g. foo::bar. This looks great in Rust code, provided that the parent crate is empty and does not also have a bar module. See the section above for more info.

Triple colons could work. People might find it confusing, but foo:::bar evokes Rust paths without being ambiguous.

We could use ~ which enables Rust code to directly name namespaced packages (as ~ is no longer used in any valid Rust syntax). It looks extremely weird, however.

We could use dots (foo.bar). This does evoke some similarity with Rust syntax, however there are ambiguities: foo.bar in Rust code could either mean "the field bar of local/static foo" or it may mean "the crate foo.bar".

Note that unquoted dots have semantic meaning in TOML, and allowing for unquoted dots would freeze the list of dependency subfields allowed (to version, git, branch, features, etc).

[carols10cents]

Ugh. No great choices here. I'm currently leaning toward something new and weird to avoid ambiguities with existing code; we'll get used to the way it looks. But I'm open to being convinced otherwise, especially by people on the Cargo, compiler, and lang teams.

[Manishearth]

@carols10cents an important point brought up by @withoutboats was that something that "converts to an underscore" would not require rustc changes, and they prefer a solution that is self contained in cargo/crates.

[pksunkara]

From your example:

[package]
name = "using-foo"

[dependencies]
"foo" = "1"
"foo/std" = { version = "1", optional = true }

[features]
# Does this enable crate "foo/std", or feature "std" of crate "foo"?
default = ["foo/std"]

What features does using-foo have? It would have foo/std, which means if I am using this crate as a dep, I might do something like below:

[dependencies]
"using-foo" = { version = "1", no-default-features = true }

[features]
# Does this enable feature "foo/std" of crate "using-foo", or feature "std" of crate "foo" of "using-foo"?
default = ["using-foo/foo/std"]

As you can see, this will quickly blow up and will be a bit inefficient for cargo to resolve.

---

Recently namespace features in cargo added syntax as crate:serde in feature strings. We could try to adopt it.

[package]
name = "using-foo"

[dependencies]
"foo" = "1"
"foo:std" = { version = "1", optional = true }
"foo:bar" = { version = "1", optional = true }

[features]
# This enables crate "foo:std" & feature "std" of crate "foo" & "foo:bar"
default = ["foo:std", "foo/std", "crate:foo:bar"]
baz = []

Now the using-foo will have features foo:std and baz.

[dependencies]
"using-foo" = { version = "1", no-default-features = true }

[features]
# enable feature "foo:std" of crate "using-foo"
default = ["using-foo/foo:std"]
# enable feature "std" of crate "foo" of "using-foo"
zzz = ["using-foo/foo/std"]

[Manishearth]

@pksunkara I don't see how this will "quickly blow up" since at worst there are never more than two options.

Discussions about feature syntax should go in #4 , I've proposed some fixes there.

[pksunkara]

Since we were discussing separator choice, I was arguing against / by showing the problem of #4.

To clarify, what I meant by my previous comment is proposing the separator to be changed to : and how it fixes #4.

[Manishearth]

Ah, I see.

Would we map : to - in rust code, or would we do use foo:bar::baz? (that seems weird, and there are also potential ambiguities)

[pksunkara]

We would map and that is being discussed in #2, right? I wanted to keep the context here to just the separator.

[Manishearth]

Correct, but it's important to note if you want it to have a mapping in the first place or be in-source.

[JWorthe]

One issue that might come up with slashes which I haven't seen mentioned elsewhere is that it's commonly used as a path separator. This could specifically make it more difficult for CLI tools to know what people are trying to do.

An interesting example is cargo new, since it also handles paths that might contain slashes and infers the crate name you want.
cargo new foo/bar will create ./foo/bar/Cargo.toml with the crate name bar.
In this case, you can resolve the ambiguity by being explicit about the name you want, for example cargo new foo/bar --name foo/bar

[ritobanrc]

Edit: I just saw that Manish pointed out here that colons are already being using by Cargo for features. Feel free to disregard my comment in that case.

Personally, I'm fond of using :. It feels consistent with the existing :: (one colon for the first "level" (getting to the right crate), and then two colons for navigating inside that create), and is close enough to current Rust syntax to not seem too foreign. I imagine there may be some confusion with new users, wondering "when do I use : vs ::", but I think its simple enough to explain : is for crates in namespaces, while :: is for modules in crates. I don't feel particularly strongly about mapping : to _ or not -- I'd be fine either way, though I'm tempted to buy the argument that its an added bump that new users may stumble over, and I'd learn towards using : in the Rust source if it wouldn't be too inconvenient to fit into current Rust syntax.

I'd also like to second the concern of / being annoying for CLI tools to deal with -- it would almost universally get treated as a path, not a crate in a namespace, and I think it would require some substantial changes to the command-line API of tools like cargo and cargo add to be convenient to use.

[joshtriplett]

@Manishearth I believe we could unambiguously allow :: to work in crate names.

The crate named abc::xyz on crates.io can use abc::xyz in Cargo.toml, and can use abc::xyz in Rust code, because your crate has already declared a dependency on abc::xyz, which constitutes an opt-in.

If abc contains a top-level item abc::xyz, and wants to use that name, it wouldn't publish a crate abc::xyz.

And conversely, if you want to import abc::xyz but not refer to it as abc::xyz, you could use Cargo's existing support for renaming to rename it to arbitrary_name.

There are plenty of disambiguators we could use as well, such as abc::crate::xyz, but I don't think we need them here. Similarly, we could make this wait for an edition boundary, and use another syntax in the current and previous editions, but I don't see any reason we need to.

We already pervasively use :: in Rust with a connotation of namespacing; this seems like a natural extension of that to me, and it will read quite well in Rust code.

[Manishearth]

@joshtriplett so yes, I think this can work, but I kinda disagree with this point:

If abc contains a top-level item abc::xyz, and wants to use that name, it wouldn't publish a crate abc::xyz.

There is a very valid use case for this: If the crate abc reexports the crate xyz (icu does this). And yes, this is a case of an ambiguity not being "real" (since it's a clash between two logically similar things), but it does feel like it can be tricky.

I don't think this is a deal breaker, but it's worth flagging.

With your lang team hat, if you think this would not be a huge deal to make happen, I'm quite amenable to this solution. It's what I originally preferred, but there seemed to be a lot of push at the time (granted, this was ages ago) to avoid changing rustc.

[Manishearth]

To be clear: it's worth flagging because the exact way we handle such an ambiguity still needs to be selected, with tooling choices that make it not confusing. Perhaps it's okay for us to not offer any in-source disambiguation path (and rely on Cargo renaming if people need it)

[Manishearth]

Importantly, it would be very bad if rustc considered this an error. This kind of thing is already annoying in, e.g., the "conflicting From impls" coherence case where 99% of the time the conflicting impls are the exact same code. One understands why the coherence error works that way, but there's no need for us to replicate that experience here 😄

[joshtriplett]

I don't think we should consider it an error; I think we should consider it unambiguously a crate name if declared in Cargo.toml.

With my lang hat on, I don't see a reason we couldn't do this, and it seems reasonable to me. There may be an implementation reason unknown to me why this would be an issue, but I don't see a language issue with it.

[CAD97]

There is a very valid use case for this: If the crate abc reexports the crate xyz (icu does this).

So to be clear, the issue here is that we want

• If I use just the crate icu::properties, we would like to refer to it as icu::properties.
• If I use just the crate icu, it reëxports icu::properties as icu::properties.
• If I use both icu and icu::properties, then we have a name clash at icu::properties.

(If the feature had existed when unic was being worked on, the unic crate would be empty and just exist as a namespace for the package-as-module tree. I even considered killing the package forest and replacing it with a feature forest instead, since the cost of maintaining the package forest was part of the reason work slowed to a halt.

As such, I'm strongly positive for making :: work.)

The most name clashing possible would then be

// extern crate foo { mod bar; };
// extern crate foo::bar;
mod foo { mod bar; }

// ambiguity error [E0659]
use foo::bar;

// use local mod
use self::foo::bar;

// this needs to deambiguate between (crate foo::bar) and (crate foo)::(mod bar)
use ::foo::bar;

E0659

It's important to note that if the crate literally does pub extern crate foo::bar as bar; or pub use ::foo::bar;, the item referred to is the same in both cases, and E0659 already doesn't fire and "just works" if both names refer to the same item. In general, this'd be adding another ambiguity that E0659 has to detect and handle.

To re(ab)use existing path syntax, we could use <> from qualified paths to disambiguate when required?

// (crate foo)::(mod bar)
use ::<foo>::bar;

// (crate foo::bar)
use ::<foo::bar>;

If we require the path to start with a leading ::, then this isn't ambiguous with simple paths (only identifiers), generic args (must have a path segment before), nor qualified types (must be first item). I can't find the use tree grammar offhand, but it currently doesn't allow <> at all IIRC.¹

And to repeat, this only would cause a hard error when

• Both foo and foo::bar are used by the crate, AND
• foo has a public bar item which is not just a reëxport of the foo::bar crate item
• (or if the crate would run into E0659 today by defining its own foo item)

it's worth flagging because the exact way we handle such an ambiguity still needs to be selected

TL;DR: we already do with E0659 and the module/crate ambiguity. Do the same thing for crate-module/crate ambiguity, but it should happen even less frequently.

Footnotes

1. Extending use trees to handle namespaced crates, generic mods, and qualified type paths all isn't impractical, either: generic mods would not clash due to the generic args requirement, and to handle qualified type paths you "just" select between a QualifiedPathTree and a UnambiguousCrateRootTree grammar based on whether a leading :: is provided before the {} tree. If no root level {} is used, the existing rules cover it. ↩

[CAD97]

Two separate but related things that has to be brought up:

1. Package names are not lib names. If I have

[package]
name = "foo::bar"
version = "1.0.0"

[lib]
name = "spam"

And the consumer has

[dependencies]
"foo::bar" = "1.0.0"

then what do they get? This applies equally to / as a separator. If the package were foo-bar, they'd get extern crate spam; today. I assume / is just handled the same way as - is; it's mapped to _ for the default lib.name.

It's reasonable to model setting lib.name as making dependencies."package.name" = "1.0.0" desugar as dependencies."lib.name" = { package = "package.name", version = "1.0.0" }

2. What happens if I use :: in a package rename?

[dependencies.foo::bar]
package = "foo-bar"
version = "1.0.0"

(This IIUC doesn't apply to / or even - since IIRC lib renames are required to be valid identifier names.) Does this mount the package in the multi-level cratespace? Is this just disallowed, and you can only rename into the top level cratespace?

3. Combine both of them! What the heck happens if I

[package]
name = "evil-icu-hijack"
version = "1.0.0"

[lib]
name = "icu::ucd"

This feels like it should definitely be disallowed, since the point of this RFC is that icu::ucd should have some amount of confidence that it belongs to icu. This would break that. Making code look like it comes from someone else's crate (which is actually being used, to boot!) seems far too abuse prone. At least if it's in the toml manifest of the user renaming it there, it was (supposedly) intentional by the user.

[mitsuhiko]

I think / is a problematic separator because it already has hierarchical meaning. We use it already for features as has been identified in the RFC, but it also causes issues with routing typically. Systems implementing namespace packages now need to deal with the complexity of handling this accurately.

As an example: mapping foo/bar into a folder foo containing a file bar causes issues when foo also wants to be a file. For web frameworks often the slash is not included in routing as routing is done hierarchical. The workaround is often to require the slash to be escaped %2f which often does not work due to limitations in web servers. NPM requires / to be escaped for the mirror protocol which causes constant struggles for people trying to host their own mirrors (npm/npm#8141, npm/npm#11738). A related hazard that comes up with git often. People sometimes name their tags package@version or similar. When someone has foo/bar@1.0 then git can no longer have a branch named foo. This has been a common cause of frustration for users (example).

My proposal would be to use almost anything but a slash based on my bad experiences with this in the past. . and :: are both choices that cause little harm in comparison.

[pksunkara]

It's important to note that if the crate literally does pub extern crate foo::bar as bar; or pub use ::foo::bar;, the item referred to is the same in both cases, and E0659 already doesn't fire and "just works" if both names refer to the same item. In general, this'd be adding another ambiguity that E0659 has to detect and handle.

TL;DR: we already do with E0659 and the module/crate ambiguity. Do the same thing for crate-module/crate ambiguity, but it should happen even less frequently.

I was going to propose the same too. The ambiguity of the name clash when doing use foo::bar between (crate foo::bar) and (crate foo)::(mod bar) can be handled by triggering E0659 or maybe a new specific error.

If we require the path to start with a leading ::, then this isn't ambiguous with simple paths (only identifiers), generic args (must have a path segment before), nor qualified types (must be first item).

As I mentioned in my previous comment, requiring a leading :: is a no go because of a certain scenario pertaining to macros. For example, clap intentionally does not have leading :: in the generated derive code when doing use clap::App etc.. because we want to allow the end user to have clap as a dependency of another library. Same with diesel and it's derives too. There might be other crates that do this but I am only familiar with the code of these two crates.

[CAD97]

requiring a leading :: is a no go because of a certain scenario pertaining to macros

Shouldn't crates that want to do this just be able to avoid a situation where they need the disambiguation? Keep in mind that because the owner of foo controls the foo namespace, they control whether there's a module/crate name conflict that would need resolution in the first place. (It's in fact almost reasonable to not even provide a disambiguation other than renaming one of the crates.)

One could be constructed via malicious renames, but... you can already break such macros via renaming and even trigger E0659 on such macros' output.

Plus, the case where this'd be problematic is rather specific: you have to have the root crate in scope via use but not the extern prelude and have root::sub in the extern prelude. This seems at least 2nd if not 3rd degree edge case of an edge case. But in this situation it'd actually still be possible to refer to (use foo)::(mod bar): you'd write ::<foo::bar> for (crate foo::bar) and self::foo::bar for (use foo)::(mod bar).¹

The proper solution for those proc macros is for them to have some way to properly get a $crate token for the target crate. If the crate is clap::derive, perhaps it could be spelled as a def-site² $crate::super to get to ::clap without requiring it to be in the extern prologue.

If it's not a proc macro, it should almost certainly be using $crate::__macro_export::* already.

Footnotes

1. The one case that wouldn't be workable is all of:

   • You have a (crate foo)::(mod bar) and a (crate foo::bar) which are ambiguous
   • You want to provide a macro emitting call-site tokens which uses (crate foo)::(mod bar) and for it to work when
     • (crate foo) is not in the extern prelude but is in use scope, and (crate foo::bar) is in the extern prelude; AND
     • (crate foo) is in the extern prelude but is not in use scope, and (crate foo::bar) is in the extern prelude.

   I don't think there is a way to properly disambiguate that, since in one case you want the item from the extern prelude, and in another you want the local name. ↩

2. so far from stable, unless mixed-site would work; I don't know how $crate currently behaves when emitted mixed-site from a proc macro ↩

[Manishearth]

• Both foo and foo::bar are used by the crate, AND
• foo has a public bar item which is not just a reëxport of the foo::bar crate item

Ah, that's reasonable. I do like this approach.

[joshtriplett]

@Manishearth I do as well, conditional on that being easy to detect. I don't think the proposal should depend on it; if it turns out to be hard to detect, it seems perfectly fine to just always disambiguate in favor of a subcrate if one exists.

[Manishearth]

@joshtriplett I believe what @CAD97 is saying is that this detection exists for globs already.

But yes, I agree. If i get a chance I might draft language for this and update my RFC (at least as a separate "alternative" though I'd like to promote it to the main solution once we're sure we have some consensus -- let's wait some more days). I'm also happy to accept PRs doing the same!

[CAD97]

that this detection exists for globs already.

While it may exist for globs, that's not what I'm referencing. Specifically:

//extern crate regex;
mod regex { pub use ::regex::*; }
use regex::Regex;

produces

error[E0659]: `regex` is ambiguous
 --> src/lib.rs:3:5
  |
3 | use regex::Regex;
  |     ^^^^^ ambiguous name
  |
  = note: ambiguous because of multiple potential import sources
  = note: `regex` could refer to a crate passed with `--extern`
  = help: use `::regex` to refer to this crate unambiguously
note: `regex` could also refer to the module defined here
 --> src/lib.rs:2:1
  |
2 | mod regex { pub use ::regex::*; }
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  = help: use `crate::regex` to refer to this module 

but

extern crate regex;
use regex::Regex;

works fine, even though the import is nominally ambiguous between self::regex::Regex and crate::regex::Regex, which are both valid import paths.

The same exact handling should in theory be able to apply unchanged to ::{crate}::{item} as currently applies to {item} when disambiguating between the extern crate and locally defined item.

[Manishearth]

Ah, makes sense!

[nagisa]

Any mechanism to refer to crates within namespaced packages on the Rust code side must not include any symbols that aren't allowed in a filename. This, due to existence of fat32 filesystems, prohibits at least \, /, :, *, ?, ", <, >, |.

Today there are two and a half of a way to provide a crate to rustc: by having it in the sysroot; and by specifying it via the --extern {ident=}path flag. For crates in sysroot, the crate is found by generating the path from the crate name. For --extern=path.rlib the reverse happens. That means the conversion between crate names and file basenames must be a bijective function. Unless we introduce some form of escaping (lets not?), such a function can only remain bijective if crate names are limited to symbols valid in filenames.

I wonder if it would make sense to figure out the two problems separately. For now require that crate name specified in package's manifest (lib.name key) is a valid crate identifier according to today's rules and maybe in the future we could look for ways to extend support to refer to these crates in new ways?

As a side note: I believe invoking rustc directly is as valid a use-case as is invoking it through cargo. Using rustc directly needs to remain at least minimally straightforward. In my eyes any solution that involves having to be smart about whether --extern is specified with or without crate name is a fairly large hit to usability of rustc itself.

[Manishearth]

@nagisa hmm, so my general idea for this was going to be:

• rustc supports --extern foo::bar=path, which Cargo uses here.
• normally, Cargo generates something like bar-somehash.rlib
• The lib.name key is by default just the leaf name (or perhaps foo_bar?)
• -L folder/ and --extern path will pull in namespaced crates under the leaf name, or perhaps not at all
• one MUST use --extern ident=path to get the crate

Overall I consider it okay if the rustc case doesn't automatically namespace crates with -L and --extern path, i consider namespacing to primarily be a Cargo feature and it's okay to require rustc to be invoked with --extern foo::bar=path. I'm not actually clear on why it must be a bijection: From what I can tell, it's okay for --extern path to end up with a different in-source crate name than when Cargo is doing it via --extern ident=path, if you're invoking stuff from rustc then you name your crates differently in source (or pass in --extern).

But I would like to see what ideas other people have here

[pksunkara]

The proper solution for those proc macros is for them to have some way to properly get a $crate token for the target crate.

IMO that's not really ergonomic because it needs the end users to always provide $crate in the proc macro calls.

Shouldn't crates that want to do this just be able to avoid a situation where they need the disambiguation?

You are right. I just realised that you are mentioning the leading :: as an optional and not required. But I guess the point is moot anyway if we decide to show E0659 instead of trying to change the import syntax.

[mitsuhiko]

The crate named abc::xyz on crates.io can use abc::xyz in Cargo.toml, and can use abc::xyz in Rust code, because your crate has already declared a dependency on abc::xyz, which constitutes an opt-in.

How does that work for cases like serde_derive? serde::derive does not depend on serde, so does that imply that an empty dummy serde toplevel module is crated for derive to be injected in?

[ShadowJonathan]

2c: This talk about ::, which intentionally blurs the line between sub-packages and modules on import, reminds me of Java in a way, which uses an ambiguous . to work with importing whole packages and their sub-packages.

Granted, Java coalesces this at runtime in a single tree, but maybe prior art could be taken from the language and approach that Java takes to explain this, to help with writing guides.

---

Personally, while I absolutely love the use of ::, I think it’s biggest complexity hazard will be properly differentiating between importing a module, and importing a sub-crate, at import time.

This is not only relevant in the language used to understand and work with it, but also for explicit/implicit notation of importing such a crate. Rust so far has always erred to the side of explicitness when it comes to showing/requiring a thing, it does not do ambiguous things implicitly - at least, this is my feeling with the language so far, this feels like a breakage of that notion.

I quite liked the suggestion of <> that @CAD97 had; it is explicit in how it addresses crates, and it re-uses the idea of “sub-something” that :: has already imbued.

If not for the primary syntax, I heavily recommend it for the “explicit” syntax, to disambiguate and work with cases where both bar exists as a module, and as a sub-crate. (This would be particularly useful in macros, to prevent sub-module vs sub-crate clashes when the latter is introduced in the context of the macro expansion at a later time than that macro was defined)

[nagisa]

-L folder/ and --extern path will pull in namespaced crates under the leaf name, or perhaps not at all

How would the former work with multiple crates named e.g. derive (coming from distinct packages)?

But I would like to see what ideas other people have here

I'm not other people, but I think punting the implementation of ::-scoping to the package developers can work quite well initially, and would also give an opportunity for evolution of the idea without blocking on this particular question. Lets take for example packages serde and serde::derive. There are a few ways a user might depend on both at the same time. Today serde users typically do something like this:

[dependencies]
serde = { version = "1", features = ["derive"] }

For this approach serde's Cargo.toml would have:

[dependencies]
derive = { package = "serde::derive", version = "1", optional = true }

and then within the crate implementation

#[cfg(feature="derive")] // or whatever the RFC decides on...
pub use derive;

Presume we wanted to enable an equivalent experience with users of serde specifying the two dependencies as such:

[dependencies]
serde = "1"
"serde::derive" = "1" # rustc gets a valid crate identifier as specified in `lib.name`

As long as serde::derive dependency enables the derive feature in serde itself, the serde::derive then would become available via two names: serde::derive (via re-export from serde) & whatever is specified in serde::derive's required lib.name key. This is largely enabled by the fact that ownership of the entire package::* namespace belongs to whomever published the package first.

This would be sufficient for the ecosystem to experiment with this idea and also explore alternatives, while giving the end users of these namespaced package families largely a good enough experience. The two downsides: it'd mean more complexity for package authors for the time being; and that in the scenario serde isn't depended on, serde::derive is only accessible via the name specified in lib.name.

---

I should probably have written this comment on the RFC itself, but then the context would be lost...

[Manishearth]

So I think the problem there is that for many users the "root" crate won't actually be in use: e.g. icu::provider would not want people to have to pull in icu to use it. So we need some name that works well as a default.

Perhaps the answer to that in your scheme is that namespaced crates must be renamed, which works i guess.

[CAD97]

While requiring an identifier Cargo.toml:package.lib.name is a valid option (so the crate itself picks the default lib name), cargo currently doesn't work the way you suggest where adding cargo_derive as a dependency activates the feature in cargo; this'd be another angle of new semantics to add to package resolution.

[Manishearth]

Fixed in Manishearth/rfcs@1986c3d, we have switched to ::