Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stabilize --extern flag without a path. #64882

Merged
merged 8 commits into from Nov 8, 2019

Conversation

@ehuss
Copy link
Contributor

ehuss commented Sep 28, 2019

This stabilizes the --extern flag without a path, implemented in #54116.

This flag is used to add a crate that may be found in the search path to the extern prelude. The intent of stabilizing this now is to change Cargo to emit this flag for proc_macro when building a proc-macro crate. This will allow the ability to elide extern crate proc_macro; for proc-macros, one of the few places where it is still necessary.

It is intended that Cargo may also use this flag for other cases in the future as part of the std-aware work. There will likely be some kind of syntax where users may declare dependencies on other crates (such as alloc), and Cargo will use this flag so that they may be used like any other crate. At this time there are no short-term plans to use it for anything other than proc-macro.

This will not help for non-proc-macro crates that use proc_macro, which I believe is not too common?

An alternate approach for proc-macro is to use the meta crate, but from my inquiries there doesn't appear to be anyone interested in pushing that forward. The meta crate also doesn't help with things like alloc or test.

cc #57288

@@ -4,7 +4,6 @@ all:
$(RUSTC) bar.rs --crate-type=rlib

This comment has been minimized.

Copy link
@Centril

Centril Sep 28, 2019

Member

Is this the only test?

This comment has been minimized.

Copy link
@ehuss

ehuss Sep 30, 2019

Author Contributor

extern-flag-fun is not actually testing this feature. It is a 5-year old test from when --extern was first added. The removed --extern hello line was originally there to verify that it would error due to the missing path. When pathless --extern was added, the error changed to "missing -Z unstable-options". With this PR, the hello line no longer fails (and thus breaks the test), due to -L flags injected by the makefile. I figured it no longer fit the spirit of the test.

I have added some tests.


* `CRATENAME=PATH` — Indicates the given crate is found at the given path.
* `CRATENAME` — Indicates the given crate may be found in the search path,
such as within the sysroot or via the `-L` flag.

This comment has been minimized.

Copy link
@Centril

Centril Sep 28, 2019

Member

Do we have tests ensuring that sysroot crates, e.g. private rustc details emit an error when used on a stable compiler? (Also when an explicit path is used to them, not just a search.)

This comment has been minimized.

Copy link
@petrochenkov

petrochenkov Sep 29, 2019

Contributor

What happens if both --extern CRATENAME=PATH and --extern CRATENAME are specified for the same CRATENAME? It would be nice to document that.
AFAIR, multiple --extern CRATENAME=PATHs with the same CRATENAME are possible if they lead to different kinds of libraries, like rlib and dylib.

This comment has been minimized.

Copy link
@ehuss

ehuss Sep 30, 2019

Author Contributor

There are about 30 tests doing various things with pathless --extern (search for // compile-flags:.*--extern).

I have added some more specific tests:

  • run-make-fulldeps/extern-flag-pathless: Shows what happens when you mix pathless with path --extern flags. Also demonstrates the preference of rlib over dylib.
  • ui-fulldeps/pathless-extern-unstable.rs: Shows --extern rustc is not allowed.
  • ui/pathless-extern-ok.rs: Basic test for a sysroot crate.

I don't have a good intuition of how sophisticated rustc tests should be. They are also a bit disorganized, so I didn't know where to stick them, or which style would be preferred. I think the dylib tests should be safe from a cross platform standpoint (I believe they only run on Linux?).

I updated the documentation with some more detail. I never really know how detailed rustc docs should be. I intentionally left the prefer-dynamic algorithm vague. It is described somewhat in src/librustc_metadata/dependency_format.rs. I figure since it is complex, and possibly subject to change, it may not be worthwhile. I can add more or remove some if desired.

@withoutboats

This comment has been minimized.

Copy link
Contributor

withoutboats commented Oct 3, 2019

We discussed this in the lang team meeting. In general, we are comfortable with stabilizing this feature. We would like some more commentary from other people with more knowledge of build systems, the sysroot, etc (lang team members with more relevant experience here weren't available today), mainly about what possible hazards there could be with this feature.

In terms of the larger goal this is trying to solve (eliminating the remaining use cases for extern crate), we had some other thoughts. This is really something that has been dropped on the floor, and no one seems available to shepherd it. It seems good to make proc_macro available in crates with proc-macro marked in lib, and this plus the associated change to cargo seems like a great solution to that problem. In the long term, we would like to reorganize the proc_macro APIs under meta and deprecate proc_macro, but - again - no one is shepherding that (as you commented).

However, it's not clear if this will be a step to solving the no_std and test problems, which have some important differences (most importantly, currently there's no clear way the need for these crates is marked in the toml). We think a full RFC process would be necessary to figure out the interface to access these crates without using extern crate.

@ehuss

This comment has been minimized.

Copy link
Contributor Author

ehuss commented Oct 3, 2019

We think a full RFC process would be necessary to figure out the interface to access these crates without using extern crate.

Absolutely. That is the intent of rust-lang/wg-cargo-std-aware#5 to produce an RFC with the necessary Cargo.toml syntax to make these dependencies explicit, and is the next major milestone I plan to tackle.

@@ -0,0 +1,9 @@
// edition:2018
// compile-flags:--extern alloc

This comment has been minimized.

Copy link
@Centril

Centril Oct 6, 2019

Member

Is this achieved by whitelist?

This comment has been minimized.

Copy link
@ehuss

ehuss Oct 7, 2019

Author Contributor

In a sense, yes. The crate must be marked with #![stable], otherwise the -Zforce-unstable-if-unmarked makes it unstable.

This comment has been minimized.

Copy link
@Centril

Centril Oct 8, 2019

Member

Ah... so just for my education, what happens if -Z force-unstable-if-unmarked isn't there? (And it cannot be there on stable when you are not using cargo since it's a -Z flag?)

This comment has been minimized.

Copy link
@ehuss

ehuss Oct 8, 2019

Author Contributor

It will successfully link whatever library is in the given path. That library will be added to the extern prelude with the identifier "alloc".

This comment has been minimized.

Copy link
@Centril

Centril Oct 8, 2019

Member

Isn't that a problem then if you are not using cargo since it would allow you to use unstable stuff on stable? I'm rather surprised that -Z force-unstable-if-unmarked isn't the default and that you need a flag to opt-out for sysroot crates.

This comment has been minimized.

Copy link
@ehuss

ehuss Oct 8, 2019

Author Contributor

Sorry, I'm not understanding the question.

On stable rustc, you cannot access any crates that are marked "unstable". Either from --extern foo or --extern foo=path/to/sysroot/libfoo.rlib or extern crate foo; or -L path/to/unstable/crates.

On nightly, you can include the attribute #![feature(rustc_private)] to override this check.

All of the sysroot crates are built with the -Zforce-unstable-if-unmarked flag.

The opt-in nature is just saying #![stable] crates are OK to access, even with the -Zforce-unstable-if-unmarked flag (that is the "if unmarked" part). I believe the only crates marked stable are core, std, alloc, and proc_macro.

In theory, anyone can compile any crate using unstable features using nightly, and then load those crates from stable. In practice, you can't (due to rustc version checking) or clearly something you shouldn't do (like set RUSTC_BOOTSTRAP env var).

Can you say more how you think that can be circumvented?

This comment has been minimized.

Copy link
@Centril

Centril Oct 8, 2019

Member

All of the sysroot crates are built with the -Zforce-unstable-if-unmarked flag.

Oh! -- I missed this part. I thought you meant that when the sysroot crate is used, unless there is a -Zforce..., then it won't be gated... but I see now that this is about when the crate is built, not used. Now it clicks. Maybe this conversation should be distilled into the rustc guide?

This comment has been minimized.

Copy link
@ehuss

ehuss Oct 9, 2019

Author Contributor
src/test/ui-fulldeps/pathless-extern-unstable.rs Outdated Show resolved Hide resolved
@withoutboats

This comment has been minimized.

Copy link
Contributor

withoutboats commented Oct 7, 2019

Absolutely. That is the intent of rust-lang/wg-cargo-std-aware#5 to produce an RFC with the necessary Cargo.toml syntax to make these dependencies explicit, and is the next major milestone I plan to tackle.

As a note, one reason I think we have not been enthusiastic on this is for the primary use case - core, alloc - there's a lot of good reason to deprecate these crates and move to a system that works more like feature flags on a single crate (so we can add impls without orphan issues, for example). However, this is a big and complicated refactor of the standard library and no one is driving it.

@ehuss ehuss force-pushed the ehuss:stabilize-bare-extern branch from c6ad892 to 6ba1de4 Oct 8, 2019
@nikomatsakis

This comment has been minimized.

Copy link
Contributor

nikomatsakis commented Oct 10, 2019

@rfcbot fcp merge

Dear compiler team,

I propose that we merge this PR. It adds a --extern foo flag which adds a crate foo from the syspath (in contrast to --extern foo=path which adds the crate foo from a specific path). The lang team has signed off on the general concept.

@rfcbot

This comment has been minimized.

Copy link

rfcbot commented Oct 10, 2019

Team member @nikomatsakis has proposed to merge this. The next step is review by the rest of the tagged team members:

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

@ehuss

This comment has been minimized.

Copy link
Contributor Author

ehuss commented Oct 15, 2019

I just realized that this does not support crate renaming (the equivalent of extern crate test as std_test). Some options:

  1. Delay this stabilization until it is supported.
  2. Add support later through some extension of the existing CLI syntax.
  3. Don't support renaming.

It looks like it would be relatively easy to add. I'm curious what anyone else thinks. I'd also like to hear if anyone would have any ideas on the cli syntax.

I think I'd like to lean towards 1 or 2, as I think it would be an odd wart not to support it. And if others agree, maybe waiting would be prudent?

@petrochenkov

This comment has been minimized.

Copy link
Contributor

petrochenkov commented Oct 15, 2019

2 is ok, IMO.

The semantics are clear

--extern foo=PATH // search by exact path `PATH` and add to extern prelude as `foo`
--extern foo???bar // search by name `bar` in the search directories and add to extern prelude as `foo`
--extern foo // sugar for `--extern foo???foo`

we only need to review the syntactic space so that --extern foo???bar is never ambiguous with --extern foo=PATH and perhaps with --extern:modifier foo[=PATH|???bar].

such as within the sysroot or via the `-L` flag.

The same crate name may be specified multiple times for different crate types.
For loading metadata, `rlib` takes precedence over `rmeta`, which takes

This comment has been minimized.

Copy link
@petrochenkov

petrochenkov Oct 16, 2019

Contributor

Do we want to guarantee that? Does any tools rely on that? (The "rlib ->rmeta -> dylib" order for metadata.)

Hypothetically, what if we want to change it to "rmeta -> rlib -> dylib"?
rmeta is smaller, easier to obtain, perhaps faster to read, etc - same reasons why rlib is preferred to dylib, basically. Perhaps we can exploit that somehow.
Then maybe it better stay unspecified in that case?

This comment has been minimized.

Copy link
@ehuss

ehuss Oct 16, 2019

Author Contributor

Sounds good, I have removed it. I am not aware of any tools relying on that behavior, and I think it would be a little strange to do so.

@ehuss ehuss force-pushed the ehuss:stabilize-bare-extern branch from 6ba1de4 to 098ffd9 Oct 16, 2019
@JohnCSimon

This comment has been minimized.

Copy link
Member

JohnCSimon commented Oct 26, 2019

Ping from triage:
This PR seems to be waiting on review @frewsxcv @petrochenkov
Cc: @ehuss
Thanks

@frewsxcv frewsxcv removed their assignment Oct 26, 2019
@frewsxcv

This comment has been minimized.

Copy link
Member

frewsxcv commented Oct 26, 2019

I'm not familiar with any of this code; can someone else review it?

@Centril

This comment has been minimized.

Copy link
Member

Centril commented Oct 26, 2019

Centril added a commit to Centril/rust that referenced this pull request Nov 7, 2019
Stabilize --extern flag without a path.

This stabilizes the `--extern` flag without a path, implemented in rust-lang#54116.

This flag is used to add a crate that may be found in the search path to the extern prelude. The intent of stabilizing this now is to change Cargo to emit this flag for `proc_macro` when building a proc-macro crate. This will allow the ability to elide `extern crate proc_macro;` for proc-macros, one of the few places where it is still necessary.

It is intended that Cargo may also use this flag for other cases in the future as part of the [std-aware work](https://github.com/rust-lang/wg-cargo-std-aware/). There will likely be some kind of syntax where users may declare dependencies on other crates (such as `alloc`), and Cargo will use this flag so that they may be used like any other crate. At this time there are no short-term plans to use it for anything other than proc-macro.

This will not help for non-proc-macro crates that use `proc_macro`, which I believe is not too common?

An alternate approach for proc-macro is to use the `meta` crate, but from my inquiries there doesn't appear to be anyone interested in pushing that forward. The `meta` crate also doesn't help with things like `alloc` or `test`.

cc rust-lang#57288
@Centril Centril referenced this pull request Nov 7, 2019
bors added a commit that referenced this pull request Nov 7, 2019
Rollup of 5 pull requests

Successful merges:

 - #63793 (Have tidy ensure that we document all `unsafe` blocks in libcore)
 - #64696 ([rustdoc] add sub settings)
 - #64882 (Stabilize --extern flag without a path.)
 - #66049 (consistent handling of missing sysroot spans)
 - #66182 (invalid_value lint: fix help text)

Failed merges:

r? @ghost
@@ -16,9 +16,9 @@ For examples, `--cfg 'verbose'` or `--cfg 'feature="serde"'`. These correspond
to `#[cfg(verbose)]` and `#[cfg(feature = "serde")]` respectively.

## `-L`: add a directory to the library search path
<a id="option-l-search-path"></a>

This comment has been minimized.

Copy link
@GuillaumeGomez

GuillaumeGomez Nov 7, 2019

Member

This line is making the html check fails because the id is not unique.

@Centril

This comment has been minimized.

Copy link
Member

Centril commented Nov 7, 2019

^-- Failed in #66185 (comment), @bors r-

@GuillaumeGomez

This comment has been minimized.

Copy link
Member

GuillaumeGomez commented Nov 7, 2019

Please fix the id duplicate then it can get r+ again.

@bors: r-

@Centril Centril modified the milestones: 1.40, 1.41 Nov 7, 2019
@ehuss ehuss force-pushed the ehuss:stabilize-bare-extern branch from 098ffd9 to ee459c6 Nov 7, 2019
@ehuss

This comment has been minimized.

Copy link
Contributor Author

ehuss commented Nov 7, 2019

@bors r=eddyb

Strange that github/bors did not have a merge conflict, since I got one locally.

@bors

This comment has been minimized.

Copy link
Contributor

bors commented Nov 7, 2019

📌 Commit ee459c6 has been approved by eddyb

@bors

This comment has been minimized.

Copy link
Contributor

bors commented Nov 8, 2019

⌛️ Testing commit ee459c6 with merge d257440...

bors added a commit that referenced this pull request Nov 8, 2019
Stabilize --extern flag without a path.

This stabilizes the `--extern` flag without a path, implemented in #54116.

This flag is used to add a crate that may be found in the search path to the extern prelude. The intent of stabilizing this now is to change Cargo to emit this flag for `proc_macro` when building a proc-macro crate. This will allow the ability to elide `extern crate proc_macro;` for proc-macros, one of the few places where it is still necessary.

It is intended that Cargo may also use this flag for other cases in the future as part of the [std-aware work](https://github.com/rust-lang/wg-cargo-std-aware/). There will likely be some kind of syntax where users may declare dependencies on other crates (such as `alloc`), and Cargo will use this flag so that they may be used like any other crate. At this time there are no short-term plans to use it for anything other than proc-macro.

This will not help for non-proc-macro crates that use `proc_macro`, which I believe is not too common?

An alternate approach for proc-macro is to use the `meta` crate, but from my inquiries there doesn't appear to be anyone interested in pushing that forward. The `meta` crate also doesn't help with things like `alloc` or `test`.

cc #57288
@bors

This comment has been minimized.

Copy link
Contributor

bors commented Nov 8, 2019

☀️ Test successful - checks-azure
Approved by: eddyb
Pushing d257440 to master...

@bors bors added the merged-by-bors label Nov 8, 2019
@bors bors merged commit ee459c6 into rust-lang:master Nov 8, 2019
5 checks passed
5 checks passed
homu Test successful
Details
pr Build #20191107.36 succeeded
Details
pr (Linux mingw-check) Linux mingw-check succeeded
Details
pr (Linux x86_64-gnu-llvm-6.0) Linux x86_64-gnu-llvm-6.0 succeeded
Details
pr (Linux x86_64-gnu-tools) Linux x86_64-gnu-tools succeeded
Details
@rust-highfive

This comment has been minimized.

Copy link
Collaborator

rust-highfive commented Nov 8, 2019

📣 Toolstate changed by #64882!

Tested on commit d257440.
Direct link to PR: #64882

💔 rls on linux: test-pass → test-fail (cc @Xanewok, @rust-lang/infra).

rust-highfive added a commit to rust-lang-nursery/rust-toolstate that referenced this pull request Nov 8, 2019
Tested on commit rust-lang/rust@d257440.
Direct link to PR: <rust-lang/rust#64882>

💔 rls on linux: test-pass → test-fail (cc @Xanewok, @rust-lang/infra).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.