New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking issue for location of facade crates #27783

Open
alexcrichton opened this Issue Aug 13, 2015 · 31 comments

Comments

Projects
None yet
@alexcrichton
Member

alexcrichton commented Aug 13, 2015

We probably don't want to indefinitely have a large number of facade crates which are all unstable, so we should eventually stabilize these crates, fold them back into the standard library, or find a good middle ground in which they can all reside.

The current set of crates considered under this issue are:

  • rustc_unicode
  • libc
  • collections
  • alloc
  • rand

Update 2018-04-05:

  • libc and rand have moved to crates.io
  • collections was merged into alloc
  • rustc_unicode was renamed to std_unicode and later merged into core

This leaves only the alloc crate still unstable, which with the stable core and std crates (plus arguably proc_macro) form the standard library.


Update 2018-06-19:

RFC 2480 proposes stabilizing the alloc crate, which would close this issue.

@Gankro

This comment has been minimized.

Contributor

Gankro commented Aug 13, 2015

It has been noted that the separation between alloc and collections is seemingly arbitrary. However with a robust allocator API, it could be possible to specify collections without any allocator. So e.g. no_std users could provide their own allocator and still use Vec.

Basically I don't think these should be stabilized at all in anything approaching the short term pending the ultimate allocator designs.

@zitsen

This comment has been minimized.

zitsen commented Dec 16, 2015

Found this through alloc documents.

Is there any news about this issue?
btw, I just want a RawVec for custom data-structures.

@SimonSapin

This comment has been minimized.

Contributor

SimonSapin commented Dec 16, 2015

RawVec would be great. Any way to allocate memory other than Vec::with_capacity + mem::forget would be nice.

@chao787

This comment has been minimized.

chao787 commented Jan 5, 2016

Is there any news about this issue?
btw, I just want a RawVec for custom data-structures.

RawVec would be great. Any way to allocate memory other than Vec::with_capacity + mem::forget would be nice.

It would be nice if rust officially support some low-level mem layer (in rust way).

@steveklabnik

This comment has been minimized.

Member

steveklabnik commented Jan 5, 2016

@chao787 that's the alloc crate.

@Shnatsel

This comment has been minimized.

Shnatsel commented Oct 4, 2016

I've found this issue through compiler error message:

error: use of unstable library feature 'alloc_system': this library is unlikely to be stabilized in its current form or name (see issue #27783)

I'm just trying to opt out of jemalloc and get Rust to compile my binary with system default malloc implementation. Is there a way to do it that would not involve unstable language features, like a compiler switch to get jemalloc out of my binaries and off my lawn?

Rationale: jemalloc has abysmal fork performance (80x slowdown) on default kernel configuration of recent Ubuntu. See issue #36705 for more info.

Opting out of jemalloc is also important for fuzzing, e.g. with afl-fuzz; even on jemalloc-friendly configurations using default malloc gets you 20% more fork performance which is significant for fuzzing workloads, and lets you substitute abusive memory allocators at runtime that catch more errors than default allocator or jemalloc but don't incur the performance penalty of DUMA or AddressSanitizer. See AFL's libdislocator as an example.

@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Oct 4, 2016

@Shnatsel yeah I'd love to explore the possibilities of stabilizing the "please give me the system allocator" intent. Right now it's unfortunately not possible to do that in stable Rust.

The tracking issue for global allocators in general is #27389, but it may be worth spawning off a separate thread of discussion for just declaring the intent to use the system allocator.

@brson

This comment has been minimized.

Contributor

brson commented Oct 4, 2016

FWIW I'm not in favor of merging the facade crates. Having std be decomposed into reusable components should be useful for those targeting more exotic systems, particularly if I complete my further ambitions to extract all system-specific components of std into their own crates. Ultimately people should be able to create custom stds for whatever weird systems they want be reusing our small self-contained building blocks.

@Shnatsel

This comment has been minimized.

Shnatsel commented Oct 4, 2016

Decomposed stdlib would be really useful for asm.js target that I will need in the near future and really want to use Rust for.

@whitequark

This comment has been minimized.

Contributor

whitequark commented Oct 16, 2016

@brson As a person targeting a mildly exotic system, I agree. For example, I would expect that a significant proportion of bare-metal use cases (like mine) would want alloc and collections but nothing beyond.

@SimonSapin SimonSapin referenced this issue Feb 19, 2017

Open

Tracking issue for 1.0.0 tracking issues #39954

9 of 28 tasks complete
@SimonSapin

This comment has been minimized.

Contributor

SimonSapin commented Mar 1, 2017

The current set of crates considered under this issue are:

  • rustc_unicode
  • libc
  • collections
  • alloc
  • rand

I surveyed what’s tracked by this issue when writing #39954. Unless I missed something:

  • libc and rand are now available on crates.io.
  • All functionality in collections is re-exported as stable in std, is deprecated, or has a more specific tracking issue.
  • Some non-deprecated functionality in std_unicode (formerly rustc_unicode) and alloc is re-exported as stable in std.
  • Functionality not available on stable Rust is:
    • alloc::heap
      • When a type T can be written such as mem::align_of::<T>() is the desired alignment, Vec + mem::forget can be used to re-implement allocate, deallocate, and reallocate.
      • As far as I know, no such work-around is possible for reallocate_inplace or usable_size.
      • EMPTY is “known” to be 1 as *mut (), but keeping a separate definition and assuming it keeps matching alloc’s seems fragile
    • alloc::oom
    • alloc::raw_vec
    • std_unicode::derived_property
    • std_unicode::property
    • std_unicode::str::is_utf16 #40190
    • std_unicode::str::utf8_char_width #40189
    • std_unicode::str::Utf16Encoder
  • Additional functionality not available on stable Rust with #![no_std]:
    • alloc::arc
    • alloc::boxed
    • alloc::rc
    • collections
    • std_unicode::char
    • std_unicode::str::UnicodeStr

Some environments might require #![no_std] because they lack threads for example, but still have a memory allocator available.

My subjective opinions:

  • derived_property, property, utf8_char_width, and raw_vec are public in order to be used in other crates. Arguably they’re not something we ever want to stabilize.
  • Utf16Encoder is almost in this category, but it is usefully more general than the stable str::encode_utf16 method since it works on arbitrary char iterators rather than just &str.
  • is_utf16 has never been used in the compiler or standard library since 47e7a05 added it in 2012 “for OS API interop”. It can be replaced with a one-liner: std::char::decode_utf16(s).all(|r| r.is_ok()). I propose removing it: #40190
@SimonSapin

This comment has been minimized.

Contributor

SimonSapin commented Mar 3, 2017

Looking at Utf16Encoder again, it’s based on char::encode_utf16 and so fairly easy to reproduce outside of std, so never mind. (Mostly, it’s only verbose because defining an iterator is verbose.)

I think this leaves two items in need of attention:

  • Stabilize raw memory allocation (alloc::heap, maybe also alloc::oom).
  • Figure out how to provide non-core functionality to #![no_std] crates.

I vaguely recall GC-related concerns about the former, but I think those would also apply to the Vec + mem::forget trick that is used on stable today. There’s even a crate for it: https://crates.io/crates/memalloc

@SimonSapin

This comment has been minimized.

Contributor

SimonSapin commented Mar 29, 2017

Stabilize raw memory allocation

This is #27700, not sure how I missed that.

@nagisa

This comment has been minimized.

Contributor

nagisa commented May 3, 2017

For unicode stuff, now that we can use crates.io crates in rustc build system, could the crates.io crates be used?

@SimonSapin

This comment has been minimized.

Contributor

SimonSapin commented May 11, 2017

The compiler depends on crates from crates.io. Does std? Should it?

@clarcharr

This comment has been minimized.

Contributor

clarcharr commented Jun 8, 2017

Question: is there a reason why Box is re-exported in libcollections but Rc and Arc are not?

@tarcieri

This comment has been minimized.

tarcieri commented Jun 17, 2017

@alexcrichton might want to remove collections from the list in the description since it's gone as of #42648

@Ericson2314

This comment has been minimized.

Contributor

Ericson2314 commented Jun 17, 2017

@SimonSapin I think it would be a huge boost to the no-std ecosystem if std was allowed to depend on crates.io (at least official out-of-tree crates).

@su8

This comment has been minimized.

su8 commented Jun 21, 2017

How do I install libc, can't use the following

extern crate libc;
use libc::{c_char, uint8_t};
@clarcharr

This comment has been minimized.

Contributor

clarcharr commented Jun 22, 2017

@su8 libc is on crates.io now

@clarcharr

This comment has been minimized.

Contributor

clarcharr commented Jun 22, 2017

@alexcrichton the OP should probably be updated

@SimonSapin

This comment has been minimized.

Contributor

SimonSapin commented Mar 16, 2018

https://internals.rust-lang.org/t/a-vision-for-portability-in-rust/6719 proposes undoing the facade and unifying the standard library under one std crate.

The relevant crates are now:

  • std
  • core
  • alloc
  • std_unicode
  • proc_macro (maybe? It’s sort of separate, but it’s distributed with std and intended for "public consumption")
@Ericson2314

This comment has been minimized.

Contributor

Ericson2314 commented Mar 16, 2018

rust-lang-nursery/portability-wg#1 talks about what to do with std. I'd argue the goal for out-of-tree ports (among other things) actually pushes in the direction of making std more of a facade.

@Ericson2314

This comment has been minimized.

Contributor

Ericson2314 commented Mar 16, 2018

[As an aside, some of our language here is rather confusing. The "facade" is the exterior skin of the building, so std is the facade, not the creates it depends on. I'd rename this issue to start to clean things up.]

@SimonSapin

This comment has been minimized.

Contributor

SimonSapin commented Mar 17, 2018

I’ve been using "the facade" as short for "the fact that std is (also) a facade for other crates, rather than a stand-alone crate."

@SimonSapin

This comment has been minimized.

Contributor

SimonSapin commented Apr 5, 2018

#49698 proposes merging std_unicode into core.

bors added a commit that referenced this issue Apr 7, 2018

Auto merge of #49698 - SimonSapin:unicode-for-everyone, r=alexcrichton
Merge the std_unicode crate into the core crate

[The standard library facade](#27783) has historically contained a number of crates with different roles, but that number has decreased over time. `rand` and `libc` have moved to crates.io, and [`collections` was merged into `alloc`](#42648). Today we have `core` that applies everywhere, `std` that expects a full operating system, and `alloc` in-between that only requires a memory allocator (which can be provided by users)… and `std_unicode`, which doesn’t really have a reason to be separate anymore. It contains functionality based on Unicode data tables that can be large, but as long as relevant functions are not called the tables should be removed from binaries by linkers.

This deprecates the unstable `std_unicode` crate and moves all of its contents into `core`, replacing them with `pub use` reexports. The crate can be removed later. This also removes the `CharExt` trait (replaced with inherent methods in libcore) and `UnicodeStr` trait (merged into `StrExt`). There traits were both unstable and not intended to be used or named directly.

A number of new items are newly-available in libcore and instantly stable there, but only if they were already stable in libstd.

Fixes #49319.

bors added a commit that referenced this issue Apr 8, 2018

Auto merge of #49698 - SimonSapin:unicode-for-everyone, r=alexcrichton
Merge the std_unicode crate into the core crate

[The standard library facade](#27783) has historically contained a number of crates with different roles, but that number has decreased over time. `rand` and `libc` have moved to crates.io, and [`collections` was merged into `alloc`](#42648). Today we have `core` that applies everywhere, `std` that expects a full operating system, and `alloc` in-between that only requires a memory allocator (which can be provided by users)… and `std_unicode`, which doesn’t really have a reason to be separate anymore. It contains functionality based on Unicode data tables that can be large, but as long as relevant functions are not called the tables should be removed from binaries by linkers.

This deprecates the unstable `std_unicode` crate and moves all of its contents into `core`, replacing them with `pub use` reexports. The crate can be removed later. This also removes the `CharExt` trait (replaced with inherent methods in libcore) and `UnicodeStr` trait (merged into `StrExt`). There traits were both unstable and not intended to be used or named directly.

A number of new items are newly-available in libcore and instantly stable there, but only if they were already stable in libstd.

Fixes #49319.

bors added a commit that referenced this issue Apr 8, 2018

Auto merge of #49698 - SimonSapin:unicode-for-everyone, r=alexcrichton
Merge the std_unicode crate into the core crate

[The standard library facade](#27783) has historically contained a number of crates with different roles, but that number has decreased over time. `rand` and `libc` have moved to crates.io, and [`collections` was merged into `alloc`](#42648). Today we have `core` that applies everywhere, `std` that expects a full operating system, and `alloc` in-between that only requires a memory allocator (which can be provided by users)… and `std_unicode`, which doesn’t really have a reason to be separate anymore. It contains functionality based on Unicode data tables that can be large, but as long as relevant functions are not called the tables should be removed from binaries by linkers.

This deprecates the unstable `std_unicode` crate and moves all of its contents into `core`, replacing them with `pub use` reexports. The crate can be removed later. This also removes the `CharExt` trait (replaced with inherent methods in libcore) and `UnicodeStr` trait (merged into `StrExt`). There traits were both unstable and not intended to be used or named directly.

A number of new items are newly-available in libcore and instantly stable there, but only if they were already stable in libstd.

Fixes #49319.

bors added a commit that referenced this issue Apr 11, 2018

Auto merge of #49698 - SimonSapin:unicode-for-everyone, r=alexcrichton
Merge the std_unicode crate into the core crate

[The standard library facade](#27783) has historically contained a number of crates with different roles, but that number has decreased over time. `rand` and `libc` have moved to crates.io, and [`collections` was merged into `alloc`](#42648). Today we have `core` that applies everywhere, `std` that expects a full operating system, and `alloc` in-between that only requires a memory allocator (which can be provided by users)… and `std_unicode`, which doesn’t really have a reason to be separate anymore. It contains functionality based on Unicode data tables that can be large, but as long as relevant functions are not called the tables should be removed from binaries by linkers.

This deprecates the unstable `std_unicode` crate and moves all of its contents into `core`, replacing them with `pub use` reexports. The crate can be removed later. This also removes the `CharExt` trait (replaced with inherent methods in libcore) and `UnicodeStr` trait (merged into `StrExt`). There traits were both unstable and not intended to be used or named directly.

A number of new items are newly-available in libcore and instantly stable there, but only if they were already stable in libstd.

Fixes #49319.

bors added a commit that referenced this issue Apr 11, 2018

Auto merge of #49698 - SimonSapin:unicode-for-everyone, r=alexcrichton
Merge the std_unicode crate into the core crate

[The standard library facade](#27783) has historically contained a number of crates with different roles, but that number has decreased over time. `rand` and `libc` have moved to crates.io, and [`collections` was merged into `alloc`](#42648). Today we have `core` that applies everywhere, `std` that expects a full operating system, and `alloc` in-between that only requires a memory allocator (which can be provided by users)… and `std_unicode`, which doesn’t really have a reason to be separate anymore. It contains functionality based on Unicode data tables that can be large, but as long as relevant functions are not called the tables should be removed from binaries by linkers.

This deprecates the unstable `std_unicode` crate and moves all of its contents into `core`, replacing them with `pub use` reexports. The crate can be removed later. This also removes the `CharExt` trait (replaced with inherent methods in libcore) and `UnicodeStr` trait (merged into `StrExt`). There traits were both unstable and not intended to be used or named directly.

A number of new items are newly-available in libcore and instantly stable there, but only if they were already stable in libstd.

Fixes #49319.

kennytm added a commit to kennytm/rust that referenced this issue Apr 11, 2018

Rollup merge of rust-lang#49698 - SimonSapin:unicode-for-everyone, r=…
…alexcrichton

Merge the std_unicode crate into the core crate

[The standard library facade](rust-lang#27783) has historically contained a number of crates with different roles, but that number has decreased over time. `rand` and `libc` have moved to crates.io, and [`collections` was merged into `alloc`](rust-lang#42648). Today we have `core` that applies everywhere, `std` that expects a full operating system, and `alloc` in-between that only requires a memory allocator (which can be provided by users)… and `std_unicode`, which doesn’t really have a reason to be separate anymore. It contains functionality based on Unicode data tables that can be large, but as long as relevant functions are not called the tables should be removed from binaries by linkers.

This deprecates the unstable `std_unicode` crate and moves all of its contents into `core`, replacing them with `pub use` reexports. The crate can be removed later. This also removes the `CharExt` trait (replaced with inherent methods in libcore) and `UnicodeStr` trait (merged into `StrExt`). There traits were both unstable and not intended to be used or named directly.

A number of new items are newly-available in libcore and instantly stable there, but only if they were already stable in libstd.

Fixes rust-lang#49319.

kennytm added a commit to kennytm/rust that referenced this issue Apr 11, 2018

Rollup merge of rust-lang#49698 - SimonSapin:unicode-for-everyone, r=…
…alexcrichton

Merge the std_unicode crate into the core crate

[The standard library facade](rust-lang#27783) has historically contained a number of crates with different roles, but that number has decreased over time. `rand` and `libc` have moved to crates.io, and [`collections` was merged into `alloc`](rust-lang#42648). Today we have `core` that applies everywhere, `std` that expects a full operating system, and `alloc` in-between that only requires a memory allocator (which can be provided by users)… and `std_unicode`, which doesn’t really have a reason to be separate anymore. It contains functionality based on Unicode data tables that can be large, but as long as relevant functions are not called the tables should be removed from binaries by linkers.

This deprecates the unstable `std_unicode` crate and moves all of its contents into `core`, replacing them with `pub use` reexports. The crate can be removed later. This also removes the `CharExt` trait (replaced with inherent methods in libcore) and `UnicodeStr` trait (merged into `StrExt`). There traits were both unstable and not intended to be used or named directly.

A number of new items are newly-available in libcore and instantly stable there, but only if they were already stable in libstd.

Fixes rust-lang#49319.

bors added a commit that referenced this issue Apr 12, 2018

Auto merge of #49698 - SimonSapin:unicode-for-everyone, r=alexcrichton
Merge the std_unicode crate into the core crate

[The standard library facade](#27783) has historically contained a number of crates with different roles, but that number has decreased over time. `rand` and `libc` have moved to crates.io, and [`collections` was merged into `alloc`](#42648). Today we have `core` that applies everywhere, `std` that expects a full operating system, and `alloc` in-between that only requires a memory allocator (which can be provided by users)… and `std_unicode`, which doesn’t really have a reason to be separate anymore. It contains functionality based on Unicode data tables that can be large, but as long as relevant functions are not called the tables should be removed from binaries by linkers.

This deprecates the unstable `std_unicode` crate and moves all of its contents into `core`, replacing them with `pub use` reexports. The crate can be removed later. This also removes the `CharExt` trait (replaced with inherent methods in libcore) and `UnicodeStr` trait (merged into `StrExt`). There traits were both unstable and not intended to be used or named directly.

A number of new items are newly-available in libcore and instantly stable there, but only if they were already stable in libstd.

Fixes #49319.
@Ericson2314

This comment has been minimized.

Contributor

Ericson2314 commented Apr 12, 2018

OK now that #49698 is merged, I'd like to reiterate my concerns on this general direction of fewer creates behind the facade.

  • There is some tentative consensus that multiple facade creates is good, even necessary, for portability. See rust-lang-nursery/portability-wg#1. This does not apply to the std_unicode merge.

  • I (and a handful of other) concerned about too big/monolithic crates, and keeping library code unnecessarily in https://github.com/rust-lang/rust/.

  • Code that doesn't use unstable language features can always be moved out in principle.

  • Moving code out is friendlier to alternative Rust implementations, and even for the official one better formalizes the which code the lang team doesn't need to worry extra about.

  • Smaller libraries are easier to read.

  • Independent repos foster parallel development

  • core is getting quite big, especially. IMO It shouldn't contain all pure abstractions, but just the minimal amount to support lang items.

The second concern absolutely does apply to std_unicode merge.std_unicode doesn't appear to, and nobody said anything to the contrary. It's much easier to merge crates than split them apart as things naturally grow entangled when nothing prevents that. So to the extent that the jury is still out on the facade, we should not be "speculatively" merging crates without fully-formed justification.

@Ericson2314

This comment has been minimized.

Contributor

Ericson2314 commented Apr 12, 2018

Also, on a process level, I'm concerned that rather than getting a simple "we disagree with your points" which would have been fine, I was getting told "your concerns don't apply to this PR", which just isn't true. This leaves me wondering who actually read the discussion before checking off the box, and whether the rules "final comment period" was followed.

Admittedly, I did open with the unrelevant portability concerns, unfortunately sowing confusion I did not intend, but I thought that was cleared up in the end, at least between @SimonSapin and I. @SimonSapin indeed said in his last comment to me #49698 (comment) "I don’t care as much about how source code is organized internally" which at least felt like a "this PR isn't trying to disagree with you" leaving open other resolutions to the issue. But then @withoutboats rejoined with another "[your concern] seems off topic from this PR" #49698 (comment), and @alexcrichton, who had not participated in that discussion since a LGTM before I raised my points, did the r+.

Again, I'm not demanding that everyone agree to with me, but just to that explicit reject my points rather than leaving open that they were just missed. If there is no process violation going on, either FCP is different for PRs than RFCs in ways I wasn't aware of, or I misunderstand FCP overall.

@SimonSapin

This comment has been minimized.

Contributor

SimonSapin commented Jun 19, 2018

RFC 2480 proposes stabilizing the alloc crate, which would close this issue.

Related:

  • PR #51569: Make the public API of the alloc crate a subset of std
  • PR #51607: Move OOM handling to liballoc and remove the oom lang item
  • PR #51639: Update the error message for a missing global allocator
@glandium

This comment has been minimized.

Contributor

glandium commented Jun 20, 2018

There is something questionable with using alloc in a no_std context currently: you automatically get a global allocator without adding a #[global_allocator], and that allocator is jemalloc. I would have expected something similar to panic: a compiler error about the missing panic_implementation (which currently still talks about the lang item, but you get the point).

@SimonSapin

This comment has been minimized.

Contributor

SimonSapin commented Jun 20, 2018

@glandium #36963 should make this disappear, and it only happens for executables. A staticlib for example does get an error message: #51639, since the "default lib allocator" is not jemalloc and is defined in std.

But yeah it’s a good question whether we should change this in the meantime. Maybe it’s not much of a problem in practice?

bors added a commit that referenced this issue Jun 29, 2018

Auto merge of #51569 - SimonSapin:liballoc, r=sfackler
Make the public API of the alloc crate a subset of std

This only affects **unstable** APIs.

I plan to submit an RFC proposing to stabilize the crate. The reason it isn’t stable yet (#27783) is in case we end up merging the standard library crates into one. However the `core` crate is already stable, so if that happens we’ll need to keep it working somehow (likely by making replacing its contents by `pub use` items). We can do the same for `alloc`. This PR will hopefully make this easier, but even if that doesn’t happen consistency with `std` seems good.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment