Tracking issue for RFC 2126: Clarify and streamline paths and visibility

[aturon]

This is a tracking issue for the RFC "Clarify and streamline paths and visibility" (rust-lang/rfcs#2126).

Steps:

• Implement the RFC. The various implementation steps are broken out into subissues (or will be, in some cases).
  • permit foo.rs or foo/mod.rs to support submodules like foo/bar.rs #45385
    • (mentoring instructions here)
  • introduce crate as a visibility modifier #45388
    • (mentoring instructions here)
  • allow crate to begin an absolute path #45477
  • add a lint for pub items that are not visible from crate root #45521
• Some implementation issues are blocked on permitting crate to begin an absolute path. Neither of these items would take effect unless we are using that extension:
  • add a lint for absolute paths that do not begin with crate (#48722)
  • allow eliding extern crate when feature gate is given
    • Rationale for delaying: if we require all local paths to begin with crate, we don't need to worry about fallback
    • This feature also depends on Allowing renaming of crates in the dependencies section
    • Add epoch lint for this #48719
• Adjust documentation (see instructions on forge)
• Stabilization PR (see instructions on forge)

Unresolved questions:

• How should we approach migration? Via a fallback, as proposed, or via epochs? It is probably best to make this determination with more experience, e.g. after we have a rustfix tool in hand.

• The final syntax for absolute paths; there's more bikeshedding to be done here in a context where we can actually try out the various options. In particular, there are some real advantages to having both crate:: and extern:: paths, but ideally we could do it in a more succinct way.

  • Currently, we accept ::crate::foo only. However, we also parse struct Foo(crate ::foo) as an illegal path, rather than treating crate as a visibility modifier. @petrochenkov describes some of the reasoning in this comment.

[retep998]

That RFC has 4 distinct features and yet we still only have one tracking issue for all of them. Can we please not shove together disparate features like this?

[aturon]

@retep998 As explained throughout the RFC discussions, these features are connected through global design considerations. E.g., providing an external mechanism for renaming crates is motivated in part by the eventual deprecation of extern crate. We can and will gate various aspects separately (and will likely have some overlapping gates for trying out different syntaxes), but for discussion on the overall design and stabilization, it's important to keep global coherence in mind.

[retep998]

Having an external mechanism to rename crates is something which has already been desired and needed for years (rust-lang/cargo#1311) and could have stood alone just fine, but instead it's being used as nothing more than a pawn to support killing off extern crate.

We've had no problems with having separate RFCs for closely related features in the past (the RFCs for repr(align(N)) and repr(packed(N)) come to mind), yet now we're claiming that changing foo/mod.rs to foo.rs is so closely related to extern:: and crate:: that they have to be in the same RFC and use the same tracking issue?

[rpjohnst]

Just to make sure this point sticks around, since it's a relatively subtle aspect of the syntax unresolved question: Using crate as a visibility modifier as well as a path prefix introduces a parsing ambiguity between crate ::absolute::path and crate::relative::path. Using a contextual keyword introduces the same ambiguity, and there are no other reserved keywords that really make sense as visibility modifiers.

I would thus like to (at least) experiment with leaving out the crate visibility modifier and sticking with the existing pub(crate).

[rpjohnst]

I wouldn't mind splitting the foo.rs/foo/mod.rs point into a separate tracking issue, since it truly does seem independent of the path and visibility modifier changes.

As far as external crate renaming... there already is a separate issue for it? It is pretty important for the path changes so I think it's fine being part of this one as well.

[neon64]

Despite lots of discussion a few weeks ago regarding the choice of crate as a visibility modifier (not a path prefix), I'm disappointed to see that although this choice of keyword was listed as an 'unresolved question' in the RFC, it has now been apparently forgotten. I myself, and several others I've noted, find this choice confusing since it isn't an adjective/modifier, and can also be ambiguous: does crate mean part of the crate's public API? No! It means local or internal or pub(lished) to the crate (hence why I prefer the latter keywords). So I'm not demanding immediate change, but at least acknowledge it as an unresolved question in this tracking issue, so that it is not forgotten upon stabilisation.

It's great that this module redesign has progressed so far, but at the same time it's important that we don't blunder ahead just to make the 'impl period', and end up making decisions without consulting the majority of Rust users. And, unfortunately, I think the people involved in Github RFC discussions aren't representative of the whole user base, because of the sheer flood of information/comments/opinions there can be discouraging. So somehow that needs to be dealt with too.

[lilianmoraru]

It isn't clear how the implementation ended-up...

[vitiral]

@rpjohnst @retep998 I have opened a new RFC to discuss foo.rs + foo/ as well as propose a few improvements to this RFC.

Edit: I suggest we open another RFC to discuss crate as a visibility modifier. Personally I would like to see the opposite: pub(extern) be added and required for all externally published symbols. This would make bare pub be the equivalent of pub(crate) in the next epoch.

[est31]

@rpjohnst

introduces a parsing ambiguity between crate ::absolute::path and crate::relative::path

Isn't crate ::relative::path invalid code anyway? Can't it be rejected during parsing?

[rpjohnst]

@est31 No, that requires doing name resolution during parsing which is definitely not something we want to do. It's one of the more annoying parts of parsing C and C++, which have similar ambiguities around a * b being a multiplication or a declaration, and around a b(c, d); being a variable declaration or a function prototype.

If we ever start allowing dependencies and top-level modules with the same name, even tangling up name resolution with parsing won't save us- crate :: some :: item could be a crate-visible item from dependency some, or a private item from top-level module some.

To keep this in perspective, we could just arbitrarily resolve the ambiguity one way or the other and require parentheses to write the other case (probably the crate-visibility absolute path case, which seems rarest), but that's still a foot-gun that we don't need if we stick with pub(crate), which already solved the syntactic ambiguity.

[petrochenkov]

Using crate as a visibility modifier as well as a path prefix introduces a parsing ambiguity between crate ::absolute::path and crate::relative::path.

This is a tiny issue, IMO, given that both visibilities on tuple struct fields and "inline" absolute paths are rare.
Paths are always parsed greedily currently, so it makes sense for crate :: x :: y to mean crate::x::y.
If the opposite meaning is wanted, then pub(crate) ::x::y or crate (::x::y) can be used.

[est31]

@rpjohnst @petrochenkov could you please share a code example where pub ::relative::path is valid code? I don't get the entire ambiguity thing because I think one of the two cases is invalid.

Edit: the only place where this could be relevant is inside macros when you match for a visibility qualifier + a path. But that's a very small breakage so acceptable IMO.

[rpjohnst]

@est31 You seem to have the wrong idea- it's not about relative paths, they're never an issue. It's about building the AST before you know what any of the names refer to. Here's a full sample:

struct S(crate :: x :: y);

Given that you're not allowed to look up any of those names yet, how do you convert that string of characters into an AST? There are two possible answers. One has a private field of a type y defined in module x of the current crate. The other has a crate-visible field of a different type y defined at the top level of dependency x. No macros needed.

[est31]

@rpjohnst I see thanks for clarifying. Its indeed an ambiguity.

[CasualX]

@rpjohnst

A simple solution is to define it unambiguously parse to the crate visibility modifier. If you want to parse it as a tuple struct with private member of the given type, do it like this:

    struct S(:: crate :: x :: y);

(note the :: prefix to indicate the root 'namespace')

This is consistent with how other root namespaces need to be referred to in submodules (eg ::std::x::y).

[rpjohnst]

Just disambiguating to crate-as-a-visibility would be somewhat surprising, I think. But it might be a good idea to "canonicalize" that workaround a bit and enforce that crate:: is always used with a leading :: (outside of uses of course). As you point out, this increases symmetry with ::std::x::y-like paths, and leaves the un-prefixed form for truly relative paths (self::/super::/something_in_scope::).

Should we do this? Allowing crate::x::y in non-use paths makes crate kind of magical, while enforcing ::crate::x::y scopes it to the same level as dependencies, which is what we're trying to imply anyway.

[petrochenkov]

@rpjohnst

enforce that crate:: is always used with a leading :: (outside of uses of course)

This may be reasonable, at least for a start (nothing prevents relaxing this later).

[Zoxc]

I'm not a fan of the use extern::bar::foo or use crate::bar::foo syntax. It seems very noisy. I'd prefer some sugar for this. I suggest extern bar::foo for this.

[zengsai]

How about add one more implicit rule? Automatically import extern crate into root namespace, make a more coincident path expression.( I'm not good at English, pls learn my view from the following example)

e.g.

1. Our project structure like this:

src
|--lib.rs
|--foo.rs
|--foo
|----|--bar.rs

2. you config a dependence in Cargo.toml, like
   [dependencies] serde = "3.0.0"

3. we add mod bar into mod foo, and add mod foo into lib.rs,

4. we define a function finlib in lib.rs, define a function finfoo in foo.rs, define a function finbar in bar.rs

Make the extern crate's scope as the top level module in crate, then we can write code like this anywhere.

for fully/qualified path

::serde::Deserialize
::serde::x::y
::finlib                            // not ::crate::finlib
::foo::finfoo                   // not ::crate::foo::finfoo
::foo::bar::finbar           // not ::crate::foo::bar::finbar

relative path write like this

serde::Deserialize      // no need to write `use serde`
serde::x::y                   // no need to write `use serde`
finlib                          
foo::finfoo                  
bar::finbar       

we first look up serde\finlib\foo in current mod scope, if not found lookup in supper mod, until the root namespace. if there have name conflict, we write fully path instead.

also we can use self::bar::finbar in foo.rs to avoid name look up.

[rpjohnst]

I'm not able to find the earlier thread that discussed this, but the compiler used to work that way and it caused major problems for name resolution. IIRC @pcwalton or @arielb1 probably know more.

[neon64]

Wouldn't the easiest solution to this ambiguity be to use a different keyword for the crate-local visibility modifier (ie: thepub(crate) replacement). For example, local, as suggested many times before in previous RFC discussions. Then struct S(local :: x :: y) is distinct from struct S(crate :: x :: y).

[rpjohnst]

That just introduces another ambiguity between <visibility> <absolute path> and <relative path starting with the new keyword>, since the new keyword would have to be contextual.

[neon64]

@rpjohnst ah damn.. But isn't that a problem the epochs were designed to solve? Eg: in the current epoch we just use pub(crate), and then, in the next epoch, introduce a new non-contextual keyword that is more 'ergonomic'.

[stjepang]

@UtherII

The reason why we need a short name for pub(crate) is because a lot of (maybe even most of) the current uses of pub will be replaced with it.

In Rust 2018, using pub for effectively private items (because they appear in in private modules) will be a warning. One should use pub(crate) for such items instead. This convention will improve readability of code: an item is visible to other crates if and only if it is marked pub, while currently whether it is visible to other crates may not be obvious at a glance.

[boojies]

Sorry, I accidentally created a new issue for this rather than responding to this thread. Oops, I am rather new to github and simply clicked a button that caused me to sign in to what I thought was responding here but which was actually creating a new issue. I will paste what I wrote in the new issue here, and my new issue can be deleted, sorry about that.

Seeing as the edition manual suggested people leave feedback here, I've decided to do so.

I don't really mind either crate:: or :: for accessing elements from the crate root. :: is what is already the syntax however, so if it is possible to have both crate:: and :: I think that both should be used. The way I see it is that the new suggested crate system regarding paths is essentially equivalent to the old syntax, with the only difference being that you no longer need to use the extern keyword, and things are more accessible rather than needing to use them explicitly in submodules they are essentially implicitly imported everywhere.

The only other addition seems to be that you start the crate root with crate:: rather than ::. I prefer starting it with just ::, as this is consistent with the way I originally learned it, however I can see people who haven't learned the module system yet finding starting it with crate:: more intuitive. Why not allow for both syntax forms? Is there some reason that makes it infeasible to have both of them? If both can be supported I thoroughly endorse supporting both of them. If only one can be supported, I am inclined more so toward :: just as it is what I've become accustomed to, though it may be easier for newbies to learn it as crate:: and I can trivially update my understanding of it to be such, so perhaps my motivation is selfish.

So essentially, I ultimately don't really care, but I personally prefer ::, but I think supporting both would be ideal, though if one must be selected and people think that starting it with crate:: is superior for newbies then I don't object to this.

[gnzlbg]

How is #[path] supposed to work with nested modules in Rust 2018 ? (#35016 (comment))

IIUC, right now, given two files src/lib.rs and src/bar.rs, there is no X that can be substituted in:

mod foo {
    #[path = "X/bar.rs"]
    mod bar;
}

such that the module bar.rs will be found, because the path to bar.rs will always be src/foo/X/bar.rs which is invalid because the directory foo does not exist.

[parasyte]

Since we're bikeshedding on a good name for a pub(crate) replacement and the core team is still seeking feedback, I would like to share my experience and offer a suggestion.

In one crate I wrote, I found typing pub(crate) everywhere infuriating. So I just shortened it to pub and inadvertently exported a lot of symbols that didn't need to be visible outside of the crate. Oops. 😛 (In my specific case it's not really a problem, because the crate is internal and unpublished, but still! Convenience over correctness.) So yeah, I strongly believe we need something better than a mashup of two keywords to convey crate-level visibility.

But I don't believe crate is the right keyword for it (see above for feedback from others on this topic), and I feel like some other suggested keywords kind of miss the point by approaching the problem from the perspective of the crate root; the issue lies at the perspective of the module, i.e. the specific line of code containing the keyword. In other words, the code is not trying to provide "crate locality", it's trying to export the reference from the module where it is defined.

In that sense, I like the keyword export (even if it's possible to confuse with extern, but maybe that's moot since extern crate is dead?) export struct Foo; looks very readable, and aligns with some other languages (ish). I wasn't able to find any mention of export as a keyword, in this RFC or elsewhere. So there's also that going for it.

For completeness, it covers the use cases brought up by @seanmonstar, @johnthagen, et al.:

export use crate::generic::{Combine, Func};

// Or even better using relative paths
export use ::generic::{Combine, Func};

export struct Bar {
    export foo: crate::Foo,
}

export struct Baz(export crate::Foo);

use crate::menu::{Sound, Volume};

export mod color;

// ...

/// A type for storing text and an associated color it should
/// be drawn as.
export struct ColoredText {
    export color: types::Color,
    export text: &'static str,
}

All that said, I'm in favor of killing pub(crate) with fire. I even prefer crate, FWIW.

[johnthagen]

@parasyte The problem I see with export is that it could be confused with "this crate exports ____" (which is what pub is for), but I see how you're advocating for a different perspective.

It seems like most people agree pub(crate) isn't ideal, and many have raised concerns that the noun keyword crate, which now used in other contexts, can be jarring as a replacement. Making sure we've fully considered other (potentially new) keywords seems like it would be a very good use of time before Rust 2018 sets this in stone.

I haven't heard any "official" feedback if this is still open for discussion though?

[jonhoo]

In the interest of introducing some more words along the lines of @parasyte's suggestion (I agree with @johnthagen that export seems more like pub than pub(crate)):

shared use ::generic::{Combine, Func};
shared struct ColoredText {
    export color: types::Color,
    export text: &'static str,
}

global use ::generic::{Combine, Func};
global struct ColoredText {
    export color: types::Color,
    export text: &'static str,
}

We could also follow in Java's footsteps and use something like protected, but I'm not sure that that's particularly easy to intuit the understanding of. There's also local (in the sense of crate-local), but I think global is actually less likely to confuse (e.g., local could be file-local).

[mark-i-m]

How would people feel about pub(cr) or even just cr?

[CasualX]

Using crate seems too good to pass up on. It is already a keyword, its previous use in extern crate is going away. Its other use is already related to visibility pub(crate).

If you squint a little using crate as an adjective doesn't sound that disconcerting. Words such as 'house' can be used as adjectives just fine. IANAEnglishProfessor.

I think reexporting a crate visible item isn't super problematic. Personally I've only ever reexported an item in a direct parent of the module which defines it, which means you (probably) want to use self instead of crate (seanmonstar's case of code organizing notwithstanding). Eg.

    mod detail {
        crate struct Foo;
    }

    crate use self::detail::Foo;

Using crate as a visibility combined with absolute path to a type (again using seanmonstar's example) does look more problematic: crate struct Foo(crate crate::Bar); as opposed to pub(crate) struct Foo(pub(crate) ::Foo);. My only hope is that this construct isn't popular and thus transitioning won't create this crate soup. Users can avoid this by explicit importing:

use crate::Bar;

crate struct Foo(crate Bar);

[parasyte]

I do like the suggestion of share or something like it. give, provide, deliver, offer, serve, post, forward ... If it must be an adjective, these can all be transformed with a suffix: shared, givable, providable, etc.

To pick on JavaScript for a moment, ES6 doesn't even have a concept of package versus module built into the language. There are only modules. The export keyword in ES6 always exports the reference from the module; it's then up to another module to import that reference by path if it wants to use it.

This isn't greatly different from pub and use, respectively. and I guess that's where some of the confusion would come from with using an export keyword. pub(crate) is actually kind of niche. Which is why I opted to just use pub in the past. ;(

[Centril]

The issue of the crate visibility modifier has been extracted out to #53120. Further debate and decisions should continue over there.

[Centril]

The issue of the mod.rs changes has been extracted out to #53125. Further debate and decisions should continue over there.

[Centril]

The issue of the extern crate deprecation as well as having use crate_name::foo and crate_name::foo Just Work™ has been extracted out to #53128. Further debate and decisions should continue over there.

[Centril]

The issue of picking a Module Path System has been extracted out to #53130.

[Centril]

Having extracted out all bits from this issue into separate issues; I am hereby closing this one.

[sanmai-NL]

@Centril: https://doc.rust-lang.org/unstable-book/print.html#extern_prelude links here. Where is this being discussed? If this info is indeed missing, could you add it to the OP?

[Centril]

@sanmai-NL can you refresh my memory on what the "extern prelude" was?

If it is just the ability to do use some_crate::foo::bar; then that should be #53128.

[sanmai-NL]

Some of it is being discussed in #54230.

[Centril]

@sanmai-NL I believe that issue is mainly for diagnostics.

[eddyb]

extern_prelude is crate_name::foo::bar outside of use (import) paths.

[petrochenkov]

@Centril

can you refresh my memory on what the "extern prelude" was?

In general, "prelude" is currently used for all names that are in scope for the whole crate and not attached to a specific module. (There are a lot of those actually.)

[alexreg]

@petrochenkov So what's "extern" prelude in relation to that?

[petrochenkov]

@alexreg
Crates passed with --extern are in scope for the whole crate while not being attached to any specific module.

[sanmai-NL]

It would be great if these nuggets of info, no matter how volatile, are recorded in some official source of documentation. Esp. if there are outside mentions of the concept, e.g. in the Unstable book. I’m not saying the maintainers who implement these concepts should do that, though.

[alexreg]

@petrochenkov Thanks, makes sense.

[shepmaster]

Having extracted out all bits from this issue into separate issues; I am hereby closing this one.

@Centril Tracking issues in the current beta version link here. Would you update the original comment with the most recent information so that people don't have to spelunk the comments?