- Feature Name: cargo_the_std_awakens
- Start Date: 2018-02-09
- RFC PR: (leave this empty)
- Rust Issue: (leave this empty)
As of Rust 1.33.0, the core and std components of Rust are handled in a different way than Cargo handles other crate dependencies. This causes issues for non-mainstream targets, such as WASM, Embedded, and new not-yet-tier-1 targets. The following RFC proposes a roadmap to address these concerns in a consistent and incremental process.
In today's Rust environment, core and std are shipped as precompiled objects. This was done for a number of reasons, including faster compile times, and a more consistent experience for users of these dependencies. This design has served the bulk of users fairly well. However there are a number of less common uses of Rust, that are not well served by this approach. Examples include:
- Supporting new/arbitrary targets, such as those defined by a custom target (".json") file
- Modifying
coreorstdthrough use of feature flags - Users who would like to make different optimizations to
coreorstd, such asopt-level = 's', withpanic = "abort"
Previously, these needs were somewhat addressed by the external tool xargo, which managed the recompilation of these dependencies when necessary. However, this tool has become deprecated, and even when supported, required a nightly version of the compiler for all operation.
This approach has gathered support from various rust team members. This RFC aims to take inspiration from tools and workflows used by tools like xargo, integrating them into Cargo itself.
This proposal aims to make core and std feel a little bit less like a special case compared to other dependencies to the end users of Cargo. This proposal aims to minimize the number of new concepts introduced to achieve this, by interacting, configuring, modifying, and patching core and std in a similar manner to other dependent crates.
This RFC proposes the following concrete changes, which may or may not be implemented in this order, and may be done incrementally. The details and caveats around these stages are discussed in the Reference Level Explanation.
In this document, we use the term "root crate" to refer to the Rust project being built directly by Cargo. This crate contains the Cargo.toml used to guide the modifications described below. This would typically be a crate containing a binary application, or a standalone item, such as an rlib.
- Allow developers of root crates to recompile
core(andcompiler-builtins) when their desired target does not match one available as arustup target addtarget, without the usage of a nightly compiler. This version ofcorewould be built from the same source files used to build the current version ofrustc/cargo. - Allow the usage of Cargo features with the
corelibrary, additionally introducing the concept of "stable features" forcore, which allow the end user to influence the behavior of their custom version ofcorewithout the use of a nightly compiler. - Extend the new behaviors described in step 1 and 2 for
std(andalloc). - Allow the user to provide their own custom source versions of
coreandstd, allowing for deep customizations when necessary. This will require a nightly version of the compiler.
As a new concept, the items above propose the existence of "stable features" for core and std. These features would be considered stable with the same degree of guarantees made for stability in the rest of the language. These features would allow configuration of certain functionalities of core or std, in a way decided at compile time.
For example, we could propose a feature called force-tiny-fmt, which would use different algorithms to implement fmt for use on resource constrained systems. The developer of the root crate would be able to choose the default behavior, or the force-tiny-fmt behavior while still retaining the ability of using a stable compiler.
A reference-level explanation is made for each of the items enumerated above.
For developers working with new targets not yet supported by the Rust project, this feature would allow the compilation of core for any target that can be specified as a valid custom target specification.
This functionality would be possible even with the use of a stable compiler.
Users of a nightly compiler would be able to set compile time feature flags for core through settings made in their Cargo.toml.
For users of a stable compiler, it would not be possible to modify the source code contents of core, or change any compile time features of core from the defaults used when publishing pre-compiled versions of core.
The source code used to build core would be the same as the compiler used for building the current project.
When compiling for a non-standard target, users may specify their target using a target specification file, rather than a pre-defined target.
NOTE: The current target specification is described in JSON, and contains some implementation details regarding the use of LLVM as the compiler backend. This RFC does not prescribe any changes to the Target Specification format, and is intended to work with whatever the current/stable method of specifying a custom target is.
For example, currently a user may cross-compile by specifying a target known by Rust:
cargo build --target thumbv7em-none-eabihfUsers would also be able to specify a target specification file, by providing a path to the file to be used.
cargo build --target thumbv7em-freertos-eabihf.jsonIn general, any of the following would prompt Cargo to recompile core, rather than use a pre-compiled version:
- A custom target specification is used
- The root crate has modified the feature flags of
core - The root crate has set certain profile settings, such as opt-level, etc.
- The root crate has specified a
patch.sysroot(this is defined in a later section)
Users of a stable compiler would not be able to customize core outside of these profile settings.
For users of a nightly compiler, compile time features of core may be specified using the same syntax used for other crate dependencies. These specified features may include unstable features.
[dependencies.core]
default-features = false
features = [...]It is not necessary to explicitly mention the dependency of core, unless changes to features are necessary.
Cargo would use the source of core located in the user's SYSROOT directory. This source code would be obtained in the same was as necessary today, through the use of rustup component add rust-src. If this component is missing, Cargo would exit with an error code, and would prompt the user to execute the command specified above.
As the custom target specifications (currently JSON) would become part of the stable interface of Cargo. The format used by this file must become stabilized, and further changes must be made in a backwards compatible way to guarantee stability.
Currently, compiler-builtins contains components implemented in the C programming language. While these dependencies have been highly optimized, the use of them would require the builder of the root crate to also have a working compilation environment for compilation in C.
This RFC proposes instead to use the pure rust implementation when compiling for a custom target, removing the need for a C compiler.
While this may have code size or performance implications, this would allow for maximum portability.
It is necessary to use unstable features to build core. To allow users of a stable compiler to build core, we would set the RUSTC_BOOTSTRAP environment variable ONLY for the compilation of core.
This should be considered sound, as stable users may not change the source used to build core, or the features used to build core.
In some cases, it may be desirable to modify core in set of predefined manners. For example, on some targets it may be necessary to have lighter weight machinery for fmt.
This step would provide a path for stabilization of compile time core features, which would be a subset of all total compile time features of core.
Initially, the list of stable compile time features for core would be empty, as none of the current features have had an explicit decision to be stable or not.
Compile time features for core may be specified using the same Cargo.toml syntax used for other crates.
The syntax is the same when using unstable and stable features, however the former may only be used with a nightly compiler, and use of an unstable feature with a stable compiler would result in a compile time error.
The syntax for these features would look as follows:
[dependencies.core]
default-features = false
features = [...]It is not necessary to explicitly mentione the dependency of core, unless changes to features are necessary.
The stabilization of a core feature flag would require a process similar to the stabilization of a feature in the language:
- Any new feature begins as unstable, requiring a nightly compiler
- When the feature is sufficiently explored, an RFC/PR can be made to
libcoreto promote this feature to stable - When this has been accepted, the feature of
coremay be used with the stable compiler.
There would be some mechanism of differentiating between flags used to build core, sorting them into the groups unstable and stable. This RFC does not prescribe a certain way of implementation.
Once the design and implications of the changes have been made for core, it will be necessary to extend these abilities for std, including components like liballoc.
In general, the same restrictions for building core will apply to building std. These include:
- Users of the stable compiler must use the source used to build the current rust compiler
- Only compile time features considered
stablemay be used outside of nightly. Initially the list ofstablefeatures would be empty, and stabilizing these features would require a PR/RFC tolibstd.
The building of std would respect the current build profile, including optimization settings.
The syntax for these features would look as follows:
[dependencies.std]
default-features = false
features = [
"force_alloc_system",
]It is not necessary to explicitly mention the dependency of std, unless changes to features are necessary.
None beyond the technical implications listed for core.
This will allow users of a nightly compiler to provide a custom version of core and std, without requiring the recompilation of the compiler itself.
As stability guarantees cannot be made around modified versions of core or std, a nightly compiler must always be used.
For this interaction, the existing patch syntax of Cargo.toml will be used. For example:
[patch.sysroot]
core = { path = 'my/local/core' }
std = { git = 'https://github.com/example/std' }NOTE: The use of
sysrootas a category may be changed to a less loaded category name. This is likely an area for bikeshedding.sysrootwill be used for the remainder of the document for consistency.
The patch.sysroot term will be introduced for patch when referring to components such as std and core.
This RFC introduces new concepts to the use of Rust and Cargo, and could be confusing for current users of Rust who have not had to consider changes to core or std previously. However, in the normal case, most users are unlikely to need these settings, while they allow users that DO need to make changes to control important steps of the build process.
Why is this design the best in the space of possible designs?
This approach borrows from existing behaviors used by Cargo to allow configuration of core and std, as if they were a regular crate dependency.
This approach also offers an approach that can be developed and applied incrementally, allowing for time to find coner cases not considered by this RFC
What other designs have been considered and what is the rationale for not choosing them?
To the author of this RFC's knowledge, there are no other open designs, other than the use tools that wrap Cargo entirely, such as xargo.
What is the impact of not doing this?
By not doing this, Rust will continue to be difficult to use for users and platforms "on the edge", such as new platform developers or embedded and WASM users.
- RFC1133 - This RFC from 2015 proposed making cargo aware of std. I still need to review in more detail to find the parts and syntax that may solve some open questions.
- xargo - This external tool was used to achieve a similar workflow as described above, limited to use with a nightly compiler
- Cargo Issue 5002 - This issue proposed a syntax for explicit dependency on std
- Cargo Issue 5003 - This issue discussed how to be backwards compatible with crates that don't explicitly mention std
For example in a no_core or no_std crate, how would we tell Cargo not to build the core and/or std dependencies?
Is it necessary to rebuild std using the customized core, even if no changes to std are necessary?
Right now we depend on rustup to obtain the correct source code for these libraries, and we rely on the user not to tamper with the contents. Are these reasonable decisions?
e.g., should the build artifacts be placed in target/, only usable by this project, or in .cargo/, to be possibly reused by multiple projects, if they happen to have the same settings?
Right now these are built using the global lock file in rust-lang/rust. Should this always be true? How should Cargo handle this gracefully?
For example, if a user sets their debug build to use opt-level = 'z', should this rebuild core/std to use that opt level? Or should an additional flag, such as apply-to-sysroot be required to opt-in to this behavior, unless otherwise needed?
This could increase compile times for users that have set profile overrides, but have not previously needed a custom core or std.
Another option in this area is to force the use of profile overrides, as specified by RFC2822.
It is currently unknown whether it is possible to provide a custom version of core or std without unstable features, as there are some compiler intrinsics and "magic" that are necessary (the format macros and box keyword come to mind).
I initially wrote the RFC in this manner, however I was later convinced this was not possible to do.
I am of the opinion that if you could, then it should be allowed to use a stable compiler, but that might be too theoretical for this RFC.
We could also move forward with the current restriction to nightly, and allow that to be lifted later by a follow-on RFC if this is possible and necessary.
If we are to uphold stability guarantees for all configurations of core and std, this could require testing 2^(n+m) versions of Rust, where n is the number of core features, and m is the number of std features. This would have a negative impact on CI times.
With the mechanisms specified above, it could be possible to remove the concept of core and std from the user, leaving only std.
By using stable feature flags for std, we could say that std as a crate with default-features = false would essentially be no_core, or with features = ["core"], we would be the same as no_std.
This abstraction may not map to the actual implementation of libcore or libstd, but instead be an abstraction layer for the end developer.
With the ability to build these crates on demand, we may want to decide not to ship target bundles for any users.
This would come at a cost of increased compile times, at least for the first build, if the artifacts are cached globally. However it would remove a mental snag of having to sometimes run rustup target add, and confusion from some users why parts of std and core have different optimization settings (particularly for debug builds) when debugging.