New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC: Stabilize the alloc crate #2480

Open
wants to merge 5 commits into
base: master
from

Conversation

Projects
None yet
@SimonSapin
Contributor

SimonSapin commented Jun 19, 2018

Existing tracking issue: rust-lang/rust#27783

HTML view

@SimonSapin SimonSapin force-pushed the SimonSapin:liballoc branch from be28193 to eca6b28 Jun 19, 2018

@KodrAus

This comment has been minimized.

Contributor

KodrAus commented Jun 20, 2018

I think having a stable alloc crate to target will be great for the ecosystem over the immediate future. A single standard library that selectively enables/disables features sounds nice over the long term as a more granular platform for describing the capabilities of a platform, but I imagine that comes with its own challenges too.

I'm definitely in favour of not blocking a reasonable concrete solution to making fewer assumptions about the runtime environment on a perfect hypothetical vision of the future 👍

@Ericson2314

This comment has been minimized.

Contributor

Ericson2314 commented Jun 20, 2018

rust-lang/rust#42774 is so close unless we do this, because of issues I don't at this moment fully recall (having mentally paged them out over the last few months) adding default type variables.

off-topic rant:

Honestly more stabilization-as-is proposals kind of annoy me; I had a fairly detailed steps on how to make progress on the more pie-in-sky goals without blocking on research-level stuff from the portability working group, but I was never able to sit down with a quorum of people that mattered and fully explain things, so nothing could be prioritized and coordinated. Instead we keep on drifting by default towards "screw facade, stick it all back in std" despite the portability group leaning in the opposite direction. And the big-std approach does require way more language-level work before portability goals have any hope of being achieved, so I consider it a non-starter. With a stalling of the portability group (and 2nd-half-of-year delay for std-aware Cargo) I just went and worked on other things, and now I poke my head back in to see stuff like this.

@SimonSapin

This comment has been minimized.

Contributor

SimonSapin commented Jun 20, 2018

@Ericson2314 Could you explain that “unless”? I don’t see how having an alloc crate separate from core or std hinders in any way having an Alloc trait for generic collections.

As to your self-described off-topic rant, maybe you’re reacting to the discussion of a single-crate standard library in the RFC? That’s a possibility that may or may not happen, it is only discussed to show that this RFC doesn’t prevent it.

It is not what this RFC is proposing. In fact this RFC is moving in the opposite direction, committing to having more standard library crates.

Instead we keep on drifting by default towards "screw facade, stick it all back in std"

Beyond not factually describing this proposal, you’re making an accusation of attitude that I feel is not fair at all. I’m trying to argue for a way to make incremental improvement and to discuss alternatives, while I have yet to extract any concrete proposal from your many words written on this topic.

@Ericson2314

This comment has been minimized.

Contributor

Ericson2314 commented Jun 20, 2018

[a single-crate standard library] is not what this RFC is proposing. In fact this RFC is moving in the opposite direction, committing to having more standard library crates.

Indeed we agree that alloc should be its own crate and I am glad that we do.

Could you explain that “unless”? I don’t see how having an alloc crate separate from core or std hinders in any way having an Alloc trait for generic collections.

A separate alloc is not the problem; The problem is stabilizing alloc now prevents evolution of the collections in alloc.

If I recall correctly, there are difficulties going from Collection<T> to Collection<T, A=Heap>. At the very least it requires stabilization of defaulted type parameters on nominal types, which is @eddyb's plan. Also, with that plan we only have default params on nomimal types and not aliases. Taken together, these mean:

  1. We are blocked on language work, even if the road map is there
  2. Unless road map changes, no defining collections prior to global allocator

C.f with alloc unstable, we can freely add the extra parameter now. std can just do type Foo<T> = alloc::collections::Foo<T, Heap>; or even struct Foo<T>(alloc::collections::Foo<T, Heap>);, avoiding any reliance on new language features.

Beyond not factually describing this proposal, you’re making an accusation of attitude that I feel is not fair at all.

You're right that I was not being fair. I had the hardest time explaining my preference for many little crates which is based on many separate individually small and future benefits, which is hard to string together into a single argument. We thankfully do agree that alloc should be separate, and hopefully those compatibility and blockage-avoiding concerns above are clearer / less opinionated.

while I have yet to extract any concrete proposal from your many words written on this topic.

The conversation is indeed woefully decentralized, inevitably leading to the difficulties in coordination. rust-lang/rust#42774 (comment) I mention doing the alias. rust-lang/rust#42774 (comment) I link to a (now-deleted?) post where I linked to the branch where I started generalized the collections. rust-lang-nursery/portability-wg#1 (comment), a still-existing post, I link that branch, and a preparatory PR to to rustc by @eddyb, as the 2nd of 3 easier portability initiatives to pursue in parallel.

@SimonSapin

This comment has been minimized.

Contributor

SimonSapin commented Jun 20, 2018

I think there is value in alloc::foo::Bar and std::foo::Bar being the same item, if they both exist. They’re just two different paths to access the same thing, and the reason for both alloc and std to exist is orthogonal to what foo::Bar is.

So if the plan for allocator-generic collections is to make new types, I would prefer these types to have different names. Or at the very least to be in a dedicated module whose name indicates what the difference is, but not overloading the alloc/std split with this additional meaning.

We thankfully do agree that alloc should be separate

It sounds like you have no objection to this RFC as proposed, so further discussion of the Alloc trait and how it’s used in collections is probably better kept in the relevant tracking issue to avoid dispersing that discussion even more, and keep this thread on topic.

@Ericson2314

This comment has been minimized.

Contributor

Ericson2314 commented Jun 21, 2018

It sounds like you have no objection to this RFC as proposed, so f

My objection is until we actually start making allocator-polymorphic and fallible collections, we shouldn't stabilize the alloc crate.

For example rust-lang/rust#51607 (comment) I think registering a hook is a needlessly awkward interface I rather if we have ergonomic fallible collections. I also think that ergonomic fallible collections are way easier to implement than everyone else does.

I think there is value in alloc::foo::Bar and std::foo::Bar being the same item, if they both exist.

OK so the thing we really want is for the allocator polymorphic types and std types to unify. I.e. there exists some allocator such that poly_a_collection::<T, GlobalAlloc> = std::a_collection<T>; Initially, this won't be possible without the language work, but after the language work it could well be.

The library work I'm proposing is really easy if we could only get some consensus it's worth doing. [I don't really want to just crank out PRs until there's some consensus on the path should persued, but I made the linked branch to convert a few collections and show how easy it is]. I think all the collections could be converted in a month. The language work could take 2. We could definitely stabilize alloc by the end of the year, but alloc might look quite different by then. If, after we try that there is some sort of more difficult hurdle unifying the types, then and only then let's move the polymorphic allocators to a different namespace and make the alloc::collections ones specialized ones std reexports as-is.

@Ericson2314

This comment has been minimized.

Contributor

Ericson2314 commented Jun 21, 2018

I guess I'm afraid this goes back to our little-libraries philosophy debate in that I want the crates beneath std to sort of look nice and stand on their own, and I deem alloc too ugly compared to where it could be with just a little effort to warrant stabilization at this time.

@SimonSapin

This comment has been minimized.

Contributor

SimonSapin commented Jun 21, 2018

For example rust-lang/rust#51607 (comment) I think registering a hook is a needlessly awkward interface I rather if we have ergonomic fallible collections.

Again I don’t see how this is related to this RFC (stabilizing the alloc crate), and this really feels like derailing the conversation. That said:

  • Those APIs are still unstable (tracked at rust-lang/rust#51245), all that PR really commits to is that alloc has some "default" OOM handling when used without std and users don’t need to provide it.
  • Fallible collection allocation is also being pursued (rust-lang/rust#48043), and not at all in opposition to OOM hooks. We can have both.
  • And hooks can still be useful for e.g. crash reporting in Firefox for all those other allocations that do not have a dedicated failure code path.

OK so the thing we really want is for the allocator polymorphic types and std types to unify.

Also off-topic.

I think all the collections could be converted in a month.

Still off-topic, but: are you volunteering to do this work? Consider working with glandium in rust-lang/rust#50882. (Not in this thread!)

We could definitely stabilize alloc by the end of the year, but alloc might look quite different by then.
[…]
I deem alloc too ugly compared to where it could be with just a little effort to warrant stabilization at this time.

As mentioned in the RFC, everything being stabilized here already exists (re-exported) as stable under std::. The only new thing proposed by this RFC is making a subset of those already-stable std API available when std as a whole is not available for unrelated reasons. We cannot modify those APIs, so the only thing that can be “quite different” is new APIs added in the future. And that’s still totally possible with this RFC! Stabilizing a crate doesn’t mean we can’t add stuff to it later.

If you’d like to object to the RFC please concentrate on what it actually proposes compared to the current situation, and keep other things you’d also like to happen to the relevant threads.

@joshtriplett

This comment has been minimized.

Member

joshtriplett commented Jun 21, 2018

This seems reasonable to me.

One request: in the RFC, could you add a recommendation for conventions that crates supporting builds with or without the use of alloc should follow? (Either a feature named no_alloc, or a feature named alloc, depending on defaults.)

@Nemo157

This comment has been minimized.

Contributor

Nemo157 commented Jun 21, 2018

Either a feature named no_alloc, or a feature named alloc, depending on defaults

Please no more negative features, no_std features are a real pain to work with. Either an alloc feature, or a default-enabled alloc feature, depending on defaults. What could be really useful is a good example of how to have a crate supporting both an alloc and a std feature, where each enables slightly more functionality (at first glance having std = ["alloc"] and just always referring to items using their most “fundamental” path would cover most of what you need to do).

@SimonSapin

This comment has been minimized.

Contributor

SimonSapin commented Jun 21, 2018

@joshtriplett The RFC already gives this example:

#![no_std]

extern crate alloc;

use alloc::vec::Vec;

One of the motivation points for the RFC is to allow this on stable Rust without using a Cargo feature at all. (Is that not clear? I can try to rephrase it.)

@Nemo157 The example I had in mind was a no_std feature that does not change the public API at all, it only opts into using the (before this RFC) unstable alloc crate in order to avoid std. In a sense, it enables the "feature" of being compatible with targets where std does not exist. But regardless, this RFC enables such libraries to use #![no_std] unconditionally.

For other libraries that can work with only core and alloc but that extend their own public API or functionality when they can also depend on the std crate, I agree that a Cargo feature should only add stuff, not remove it. (Unfortunately default features are not great because they need to be explicitly disabled by ever dependent in order to not be included, but this is getting off-topic.)

@Centril

This comment has been minimized.

Contributor

Centril commented Jun 21, 2018

@SimonSapin

#![no_std]
extern crate alloc;
use alloc::vec::Vec;

I only want to say: Thank you so much for this! This will be awesome for anyone who has gone through the pain of conditional compilation of alloc feature gates.

@Ericson2314

This comment has been minimized.

Contributor

Ericson2314 commented Jun 21, 2018

Still off-topic, but: are you volunteering to do this work? Consider working with glandium in rust-lang/rust#50882. (Not in this thread!)

Thanks for the link. I commented there.

The only new thing proposed by this RFC is making a subset of those already-stable std API available when std as a whole is not available for unrelated reasons.

I disagree these stabile APIs are good enough for alloc. They might be good enough for the narrower set of situations std supports, but the no_std situation is broader, and for me that raises the bar.

For example I want to be able to look a library's dependencies and know that it may use allocation but it doesn't have any weird ambient authority. Being able to set hooks and use a global allocator constitute that ambient authority. Yes the hook interface isn't stable yet, but the global alloc stuff would be: I'd want to move it out to alloc and into another create prior to stabilization. Then I know a library that just depends on alloc and not that other create can't do those other things.

Taking a quote from rust-lang/rust#50882

Ideally, all inherent methods of Box<T> could be applied to
Box<T, A: Alloc + Default>, but, as of now, that would be backwards
incompatible because it would require type annotations in places where
they currently aren't required, per #50822 (comment)

If we make std do a type alias of it, then that problem might be avoided. But it would be breaking change for that same inference reason if alloc was already stable.

Overall, the portable work requires some realm for experimenting (not even for too long!), and stabilizing every crate behind facade ties are hands. (we can make more, but the good names will be taken). I'll never understand this rush to stabilize concurrent with no clear path (to me at least) for the working group to coalesce around proposals for an official roadmap.

@joshtriplett

This comment has been minimized.

Member

joshtriplett commented Jun 21, 2018

@Nemo157 I completely agree, I'd rather see this always done as a positive lint rather than a negative one.

@SimonSapin

This comment has been minimized.

Contributor

SimonSapin commented Jun 21, 2018

Being able to set hooks and use a global allocator constitute that ambient authority.

Having a global allocator (and possibly OOM hooks) is everything that the alloc crate is about, in the current core v.s. alloc v.s. std split. The Alloc trait is available in core::alloc.

If/when we have something similar to Vec or Box that doesn’t (even through a default type parameter) assume the existence of a global allocator, then such types are "pure" library code that don’t assume any dependency and therefore can live in libcore. (Or whatever other crate if we decide that core should be split up.)

@SimonSapin

This comment has been minimized.

Contributor

SimonSapin commented Jun 21, 2018

stabilizing every crate behind facade ties are hands. (we can make more, but the good names will be taken).

Is that what this is about? You’re advocating to block real use cases for months or years over a name?

@Ericson2314

This comment has been minimized.

Contributor

Ericson2314 commented Jun 22, 2018

I sort of presumed there was zero appetite to split core up and make a second facade. But I'd love to be wrong—that's a big reassurance! My mental model for stabilization was sort or bottom-up: working up carve out all the pure code and then deal with the lang items. But maybe this is better thought of as top down: strip things out of std leaving self-supporting subsets, while opening the door to future dividing-and-conquering down to pure bits.

If the preceding paragraph sounds good to you, then yes that does just leave the name. If that is "everything" this create is about, can we call it global_alloc?

@SimonSapin

This comment has been minimized.

Contributor

SimonSapin commented Jun 22, 2018

I am not in favor of renaming the alloc crate − which has existed in pretty much this form for several years with no sign of something else concrete happening − to a longer name now just because of unspecific plans that we might do something else in this area some day.

In what order you would like to focus your attention on other things doesn’t seem relevant. This thread is about this RFC making this proposal at this time.


Off-topic:

My earlier comment was not in support of nor against splitting up libcore. Whether that is possible or desirable is a discussion for another thread. It was only a statement of possibility not prevented by our stability policy. I only included it in an attempt to preemptively defuse an off-topic objection I suspected would be coming from you.

@Ericson2314

This comment has been minimized.

Contributor

Ericson2314 commented Jun 22, 2018

Forget about my plans for a moment. You pointed out core::alloc exists, so allocation is already a cross-crate concern. (c.f. collections and sync were basically the sole source of their corresponding std::* modules.) Likewise alloc::alloc is kind of a weird path, but follows from the interface subset policy you espouse with the alloc:: collection PR. Renaming alloc clarifies the purpose of this crate, and avoids that odd-looking path while retaining the subset interface policy.

which has existed in pretty much this form for several year

Let's no pretend these intermediate crates haven't changed because people have been broadly happy with their existance. collections and std-unicode also hadn't changed much, and then were merged, so there's plenty of precedent of shaking things up right before the merge.

@dhardy

This comment has been minimized.

Contributor

dhardy commented Jun 22, 2018

The alloc crate does not have a prelude (items that are implicitly in scope).

Would it not be possible to add a prelude and use it automatically when extern crate alloc; is specified? This would make it easier to use alloc.

PR #51569 moves them around so that the module structure matches that of std, and the public APIs become a subset

which implies that it would also be possible to make alloc a superset of core. This would allow writing extern crate alloc as std; and pretending that one is in fact using std (except that many parts of it are unavailable). I am not convinced this is a good idea however; aside from maintenance overhead, the current structure of core/alloc/std encourages users wanting to target no_std with partial functionality (like we have in Rand, like Regex is trying to do now, etc.) to import from the most appropriate module, and thus makes it clearer which parts need to be cfg-ed on feature flags.

@SimonSapin

This comment has been minimized.

Contributor

SimonSapin commented Jun 22, 2018

The alloc crate does not have a prelude (items that are implicitly in scope).

Would it not be possible to add a prelude and use it automatically when extern crate alloc; is specified? This would make it easier to use alloc.

The quoted part of the RFC might be giving a misleading framing. I think it’s better to think of current prelude(s) as the default one and the #![no_std] one, rather than associated to crates. If we want to have more preludes I’d prefer a language proposal like #890 that also works with crates.io rather than something ad-hoc for alloc.

Until such a proposal is accepted, I think #![no_std] crates can live with writing use alloc::vec::Vec; explicitly.

@SimonSapin

This comment has been minimized.

Contributor

SimonSapin commented Jun 22, 2018

The quoted part of the RFC might be giving a misleading framing.

I pushed a commit that rewrite that paragraph.

@Nemo157

This comment has been minimized.

Contributor

Nemo157 commented Jun 22, 2018

Could the alloc crate have an explicit prelude. Looking at the std prelude it could be appropriate to have

pub mod prelude {
    pub mod v1 {
        pub use crate::boxed::Box;
        pub use crate::borrow::ToOwned;
        pub use crate::string::String;
        pub use crate::string::ToString;
        pub use crate::vec::Vec;
    }
}

This would hopefully be forward compatible with any implicit prelude proposal since it's a pattern used in a lot of crates now.

@eddyb

This comment has been minimized.

Member

eddyb commented Jun 22, 2018

Minor note: we should probably be naming the module rust2015 cf. rust-lang/rust#51418.

@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Jul 9, 2018

@Amanieu ok thanks for the info.

In some sense though I'm a little worried here if we don't have a great motivational factor beyond "we'd like to stabilize this crate as it's been unstable for awhile". I see stabilizing alloc as a pretty large step for us that shouldn't be taken lightly, and for me at least it's best to have some strong motivation to both take the step and provide guidance for how it should be stabilized and expected to be used. To help sell this to others, is there perhaps a use case that can be found which provides strong motivation for stabilizing alloc? (one beyond "it removes a nightly feature here" if possible).

For HashMap I would personally very much prefer to not add a third assumption to alloc, the ability to get randomness. Instead, though, we can step back and reevaluate why this even exists! Currently HashMap has randomness for DoS protection, which afaik is primarily motivated by servers which are fed things like arbitrary HTTP headers and stuff like that. From what I've seen, though, no embedded system/kernel/etc wants or needs DoS protection on hash maps. Instead they can have a similar strategy to the wasm target, no randomness and a functional hash map by default.

In that sense I don't quite agree that if we stabilize alloc we can never have HashMap in alloc. Instead I think it's pretty viable to have "optional randomness" in the sense where you can opt-in to having randomized hash maps but it's not necessary. All our tier 1 platforms would opt in (via libstd) and other platforms would choose whether they're able to do so (like wasm choosing "no").

@Ericson2314

This comment has been minimized.

Contributor

Ericson2314 commented Jul 9, 2018

@alexcrichton The natural thing to do with HashMap is leave the default random state type elsewhere, so just as collections should have a default parameter for the allocator, a remotely defined Global, HashMap's existing default BuildHasher instead becomes another defined-elsewhere type which may be implemented in whatever way, provided it matches the BuildHasher bound. HashMap then has no direct knowledge of randomness at all. In #2492, I propose the mechanism to do this.

@tarcieri

This comment has been minimized.

tarcieri commented Jul 9, 2018

@alexcrichton the main reason I'd like alloc stabilized is to unlock the parts of the Rust crate ecosystem which are effectively #![no_std] + Box/String/Vec for use in these environments. Since using alloc in this way is presently a nightly-only feature, crate maintainers are reluctant to support it. The result has been an awful lot of forking and tweaking to support the nightly-only API, e.g.:

https://github.com/microcrates
https://github.com/baidu/rust-sgx-sdk/tree/master/third_party

It would be really nice not to have to fork these crates in order to support these environments, but instead get the needed tweaks upstream, e.g. carllerche/bytes#75

@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Jul 9, 2018

@tarcieri ok cool, so that I think makes sense. The purpose of stabilizing alloc would be to start building an ecosystem around it. That way maintainers of crates can start providing an alloc "version" of their crate on stable and don't have to worry too much about maintaining it. This will hopefully act as a catalyst for many crates being workable on more platforms (in theory) if I understand it right.

For a platform like SGX though, isn't that intended to be full-featured enough to use std? Are there examples of other platforms where an ecosystem is needed but std is "fundamentally not available"?

@Ericson2314

This comment has been minimized.

Contributor

Ericson2314 commented Jul 9, 2018

@tarcieri Right, but that's just stability for stability's sake.

@joshtriplett

This comment has been minimized.

Member

joshtriplett commented Jul 9, 2018

@alexcrichton

From what I've seen, though, no embedded system/kernel/etc wants or needs DoS protection on hash maps.

That's not a safe assumption; most embedded/IoT devices have some kind of connection, and have to process untrusted data.

For a platform like SGX though, isn't that intended to be full-featured enough to use std?

Not exactly. While there are variations on SGX that could support std, normally an SGX enclave doesn't have direct access to the filesystem, I/O, network, process control, etc.

@tarcieri

This comment has been minimized.

tarcieri commented Jul 9, 2018

The Baidu rust-sgx-sdk supports a full std alike (sgx_tstd) so it perhaps isn't the best example.

@alexcrichton I think most environments that support #![no_std]+alloc are probably good candidates for a future stripped down std which doesn't depend on things like I/O or threading primitives. The main counterargument for keeping things #![no_std]+alloc instead of going with a minified std would probably be cases where developers want to minimize code size/complexity, either to meet some strict persistent storage requirements, or due to other platform/toolchain limitations. I don't think std is particularly -Os (in a hypothetical sense) friendly right now, and there may be cases where even a minified std is too heavy a dependency.

That said, and like you said earlier, the main motivation for stabilizing alloc now would be to provide a short-term solution to this problem rather than waiting for a maximally portable std to materialize. If there's a short path to a minimum viable/maximally portable std, I think it would be preferable to stabilizing alloc. But if that's going to be awhile, I think there's definitely benefit in unlocking things like protocol crates (protocol parsers or full-blown implementations like rustls, snow, and strobe) which depend on Vec for developers working in #![no_std]+alloc-friendly environments.

Here's an example of a crate that just added #![no_std]+alloc support which implements an encrypted channel protocol (Strobe):

rozbb/strobe-rs@15af069

@SimonSapin

This comment has been minimized.

Contributor

SimonSapin commented Jul 9, 2018

@alexcrichton

is there perhaps a use case that can be found which provides strong motivation for stabilizing alloc? (one beyond "it removes a nightly feature here" if possible).

I don’t understand what you mean here, it seems there is no possible answer to this question. Stability is the whole point of stabilization.

Some people or projects choose not to use Nightly. I think that extending the realm of what is possible to build in Rust for them is not insignificant.

@Ericson2314

This comment has been minimized.

Contributor

Ericson2314 commented Jul 9, 2018

I think what was meant is a comparison of the pain of keeping it on unstable vs the benefits of what further auditing/modifications could give us.

@tarcieri

This comment has been minimized.

tarcieri commented Jul 9, 2018

The main pain point for me, as I've said before, is leveraging the Rust crate ecosystem in these environments. I will probably remain on nightly in my projects in these environments due to other reasons, but stabilization allows changes necessary to support these environments to be upstreamed, rather than maintained in forks.

This is especially tricky for transitive dependencies, e.g. a protocol library that depends on a Protocol Buffers implementation (e.g. prost) that depends on bytes. The only options now are to convince every project maintainer in that tower of dependencies to support an unstable nightly feature in their crate, or to fork every single crate to add alloc support.

Speaking as someone who tried unsuccessfully before to do the former, and ended up going with the latter... it is a huge pain.

@Ericson2314

This comment has been minimized.

Contributor

Ericson2314 commented Jul 9, 2018

@tarcieri Perhaps requesting those crates support a nightly feature would be easier now that we have an alloc prelude?

@BurntSushi

This comment has been minimized.

Member

BurntSushi commented Jul 9, 2018

@Ericson2314 I can't speak for other crate authors, but I'm very unlikely to do that. I've been around long enough to know what it's like to track nightly APIs. It ain't fun. Asking crate authors to maintain unstable APIs is a massive uphill battle.

@derekdreery

This comment has been minimized.

derekdreery commented Jul 9, 2018

Stabilizing is shifting burden from the crate developers (who have to track unstable APIs) to the language (who have to make alloc work for ever). That's the trade-off. I have no idea if it's worth it or not in this case, just wanted to state the arguments for/against.

@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Jul 10, 2018

@tarcieri yeah I definitely agree that alloc is a short-term solution for "let's build an ecosystem around the alloc crate". To me it's currently sort of nebulously defined as to what exactly is using that ecosystem (your SGX crates for sure, for example, but it's unclear to me at least how much else) and the thrust of my question was whether we have sufficient motivation to take the short-term solution over a possible different longer-term solution.

@SimonSapin sorry what I mean to say is that there's clear motivation in the sense of "stabilize for stabilization's sake" but I'd prefer to find deeper motivation which justifies what @tarcieri was saying in terms of going with the short-term solution instead of sticking it out for a possibly different/possibly better future (which is of course very uncertain). Right now it seems the main motivation is @tarcieri's platform of SGX and upstreaming work into crates, but is there other motivation as well? Do we know of more concrete use cases for alloc-without-std? (use cases which aren't "X sent a PR asking me to do so", more like "why did X send a PR?")

@lexxvir

This comment has been minimized.

lexxvir commented Jul 10, 2018

I'd like to add my use-case to @tarcieri. I developing no_std application but have enough memory on device (1Mbyte ROM, 384Kbyte RAM) to use dynamically allocated memory, and now I obligated to use nightly. Also I need to fork and "tune" almost every crate that uses collections from alloc crate to implement "fake" std facade in lib.rs file.

@tarcieri

This comment has been minimized.

tarcieri commented Jul 10, 2018

@alexcrichton the HSM platforms I'm working with are a lot better example than SGX. As I said earlier, SGX is just an unusual case of a full-blown i686-unknown-linux-gnu environment, so I think I was somewhat mistaken to even bring it up, aside from the cases of people wanting to write "bare metal" SGX applications which don't use the Intel SDK (as it were, Fortanix is developing a KMS application in Rust that way).

The HSM platforms I'm working on are much more applicable, and are really just a special case of "RTOS whose SDK exposes malloc(), free(), and some sort of proprietary system call API". FreeRTOS is another example:

https://github.com/hashmismatch/freertos.rs/blob/master/src/lib.rs#L56

It'd be nice to hear experiences from people in the Embedded WG on these sorts of platforms.

Would stabilizing alloc just be a temporary hack until std can be made maximally portable, and then everyone who could would naturally switch over to std? I'm not so sure. Again, I think std was not designed with minimal code size in mind, and in many of these applications minimizing code size is critical.

Some developers I know working in these environments have adopted a dialect of Rust that avoids things like generics/monomorphization, instead heavily leveraging traits/trait objects, and in that way optimizing for minimal code size instead of maximal performance. I think std has generally gone the opposite direction, which is great for higher classed devices, but may be a poor fit for constrained ones.

My best guess would be if alloc were stabilized, and std were made available in these environments somehow, you would still see people using alloc alone over full std for code size reasons (and in my case, to simplify the linking process). That is to say, I don't see a stable alloc as a stopgap solution which people will immediately abandon when a more portable std is available. I think it's something developers in these sorts of constrained environments might actually prefer. That's just my own not-entirely-informed opinion though.

@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Jul 10, 2018

Ok cool, great to hear about all these!

For my own edification, I'm curious, @lexxvir and @tarcieri how are y'all compiling libcore/liballoc? Are you using something like xargo?

@lexxvir

This comment has been minimized.

lexxvir commented Jul 10, 2018

@alexcrichton Yes, I use xargo to cross compile core and alloc crates. Also I known about rust-std-thumbv*-none-eabi component available on recent stable but now I use certain nightly version which have not that component.

@tarcieri

This comment has been minimized.

tarcieri commented Jul 10, 2018

@alexcrichton using xargo to produce a static library for the target ISA (ARM32 or PPC32) then linking the static library together with a small C shim and a minimal libc-ish library provided by the SDK (where I get the malloc and free implementations) using the vendor provided toolchain (which is a repackaging of GNU ld), using --gc-sections to prune unused code.

@Amanieu

This comment has been minimized.

Contributor

Amanieu commented Jul 10, 2018

@alexcrichton I'm using xargo with a custom target json derived from the aarch64-unknown-linux-musl one, but which does not use any libc. All the startup code, including an asm stub for _start, self-relocation for PIC and thread-local storage setup is done using custom rust code.

liballoc: rewrite on motivation and alloc crate requirements
* Largely rewrite the motivation section to concentrate on the impact
  on the (crates.io) library ecosystem.

* Mention the assumptions/requirements made by the alloc crate:
  allocator and OOM handler, but not randomness.

* Add stabilization of `alloc::prelude`

* Discuss alternatives for HashMap in no_std
@SimonSapin

This comment has been minimized.

Contributor

SimonSapin commented Jul 11, 2018

I just pushed a rewrite of parts of the RFC text:

  • Concentrate in the Motivation section on the impact on the (crates.io) library ecosystem.
  • Mention the assumptions/requirements made by the alloc crate: allocator and OOM handler, but not randomness.
  • Add stabilization of alloc::prelude (implemented in rust-lang/rust#52159)
  • Discuss possible futures for HashMap in no_std
@gnzlbg

This comment has been minimized.

Contributor

gnzlbg commented Jul 12, 2018

@alexcrichton this is a bit off-topic but "my vision" for std is very different with what you seem to mention, yet still 100% compatible from what your goals seem to be.

I think that making std being super configurable is a good idea, and I also think that we should move to a model where users compile more and more of std themselves, because a lot of features are blocked on this (from using debug_assert! in std, to using sanitizers, to compiling std with target features like avx2, etc.). So I pretty much agree with everything you mention, maybe even going a bit further, and saying we should move to a model where everyone compiles most of std themselves by default.

The moment I start compiling std myself all the time, however, I'll start asking questions like "My crate only needs Vec, why do I have to compile HashMap and fs (file-system)? Why do I have to pay for that even though I am not using it?"

So my vision for how std should look is basically "the same way that everything else in the Rust ecosystem looks: small crates that are combined to form larger crates, and so on, until std".

That is, at the lowest level we have one or more core libraries with lang items (e.g. floats, f16, ...). Then we have crates exposing platform-specific APIs, like libc, mach, winapi, and crates building on top of them, e.g. jemallocator, and crates building on top of them liballoc, libvec, libhashmap, libstring, ... and crates building on top of them libcollections, up to libstd.

For me personally, ideally, all those pieces would be independent crates, living in the nursery, and just using cargo to manage the dependency graph. Each would have their own CI, examples, benchmarks, docs, ... making it easier to hack on self-contained parts of std without having to hack on rustc, and offloading rust-lang/rust CI. I don't think this will make it harder to perform ecosystem wide upgrades though: when we update the Alloc trait in liballoc to a new version, we then have to go through all dependent crates, updating things, update their dependency on the liballoc crate, and continue going up, until we end up in libstd, where we can just release a new version that works, and then we update the libstd dependency in rustc.

Of course, we cannot split everything from rust-lang/rust, because some pieces of std are really tightly integrated with the compiler, and splitting them would just not be pragmatic (e.g. GlobaAlloc trait being integrated with the compiler). I don't know how many pieces like this there are (primitive types...), or whether they do represent the bulk of code in std, but many pieces do not really have that much "magic" behind them beyond being able to use nightly features, rustc_ features, etc.

None of this is incompatible with a super configurable std nor the portability lint. I don't know if it would make things easier or harder, but for at least some std components it would make it easier for people to hack on them without having to hack on rustc, and also it would make it easier to test these components beyond what we can feasibly test in rust-lang/rust. And of course, if you don't need many parts of std, you wouldn't have to recompile those yourself: you could just manually add dependencies on the pieces that you need (we'll just need a way to specify these dependencies in cargo, e.g., for users specifying a libvec dependency version might not make much sense, since what they want is for the version that is shipped with their std source code to be used).

@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Jul 12, 2018

Ok thanks for the info @lexxvir, @tarcieri, and @Amanieu! That makes sense to me. It also means that if we do one day decide to go the route of making std more configurable then you'd be poised to take advantage of it with xargo (as xargo uses cargo basically)

@gnzlbg a compelling vision! I agree that the end goals are aligned for sure. I'm sure we'll have fun times debating the details over time :)

@SimonSapin the updates look good to me!

mati865 added a commit to mati865/rust that referenced this pull request Jul 24, 2018

Add the `alloc::prelude` module
It contains the re-exports that are in `std::prelude::v1`
but not in `core::prelude::v1`.

Calling it prelude is somewhat of a misnomer since (unlike those modules
in `std` or `core`) its contents are never implicitly imported in modules.
Rather it is intended to be used with an explicit glob import like
`use alloc::prelude::*;`.
However there is precedent for the same misnomer with `std::io::prelude`,
for example.

This new module is unstable with the same feature name as the `alloc` care.
They are proposed for stabilization together in RFC
rust-lang/rfcs#2480

@pepyakin pepyakin referenced this pull request Dec 11, 2018

Open

Compile the runtime with a stable toolchain #1252

0 of 1 task complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment