Tracking issue for RFC #2116: fallible collection allocation

[aturon]

This is a tracking issue for the try_reserve part of the RFC "fallible collection allocation" (rust-lang/rfcs#2116).

Steps:

• Implement the RFC #48648
• Adjust documentation (see instructions on forge)
• Stabilization PR (see instructions on forge)

[snf]

Hi, this feature will require oom and try_reserve to be implemented. I'm needing try_reserve and I'm willing to work (slowly) on it if noone else does.
I see that the previous commit by @Gankro is here and seems to match the RFC description.
Is there anything else I should consider before rebasing that PR?

[snf]

Hi, this feature will require oom and try_reserve to be implemented. I'm needing try_reserve and I'm willing to work (slowly) on it if noone else does.
I see that the previous commit by @Gankro is here and seems to match the RFC description.
Is there anything else I should consider before rebasing that PR?

[snf]

I can use some mentorship on what's the best way to implement oom=panic/abort . Should I emit a global variable through rustc depending on this case?

I was taking inspiration from #32900 but there might be an easier way to emit a boolean global variable.

[snf]

I can use some mentorship on what's the best way to implement oom=panic/abort . Should I emit a global variable through rustc depending on this case?

I was taking inspiration from #32900 but there might be an easier way to emit a boolean global variable.

[aturon]

cc @Gankro, can you help mentor?

[aturon]

cc @Gankro, can you help mentor?

[SimonSapin]

try_reserve methods are implemented in Nightly, but the accepted RFC also includes the ability to make allocation failure cause a panic rather than abort and that part is not implemented. Unfortunately the RFC does not discuss how this would be implemented, other than being controlled by a flag in Cargo.toml. It’s not clear to me how this can work.

[SimonSapin]

try_reserve methods are implemented in Nightly, but the accepted RFC also includes the ability to make allocation failure cause a panic rather than abort and that part is not implemented. Unfortunately the RFC does not discuss how this would be implemented, other than being controlled by a flag in Cargo.toml. It’s not clear to me how this can work.

[glandium]

Unfortunately the RFC does not discuss how this would be implemented, other than being controlled by a flag in Cargo.toml. It’s not clear to me how this can work.

We've been talking about panic::set_hook-like hooking for the new oom lang item in #49668, this could be how this works.

[glandium]

Unfortunately the RFC does not discuss how this would be implemented, other than being controlled by a flag in Cargo.toml. It’s not clear to me how this can work.

We've been talking about panic::set_hook-like hooking for the new oom lang item in #49668, this could be how this works.

[SimonSapin]

@alexcrichton, how does #51041 fit with the goal of implementing some way to opt into oom=panic?

[SimonSapin]

@alexcrichton, how does #51041 fit with the goal of implementing some way to opt into oom=panic?

[alexcrichton]

@SimonSapin we'd decide that the regression is acceptable, and we'd revert the change.

[alexcrichton]

@SimonSapin we'd decide that the regression is acceptable, and we'd revert the change.

[SimonSapin]

You use a conditional tense, but oom=panic is already in an accepted RFC (though the implementation strategy is not quite settled).

[SimonSapin]

You use a conditional tense, but oom=panic is already in an accepted RFC (though the implementation strategy is not quite settled).

[alexcrichton]

Ok sure, we can figure it out in the implementation details? AFAIK there's no need to revert the change as it has basically no benefit today. It's a trivial change to revert as well, it's one line in the standard library

[alexcrichton]

Ok sure, we can figure it out in the implementation details? AFAIK there's no need to revert the change as it has basically no benefit today. It's a trivial change to revert as well, it's one line in the standard library

[3n-mb]

This example in playground asks for 1024*1024*1024*1000 bytes on a server, via v.try_reserve(len)? without giving an error.

Is this RFC implementation is not at the point, at which errors in such cases will start to show up?
Or, is this a gross over-commit by underlying OS, and this state of affairs will stay like this?
Can try_reserve in principle reserve only realistic things?

[3n-mb]

This example in playground asks for 1024*1024*1024*1000 bytes on a server, via v.try_reserve(len)? without giving an error.

Is this RFC implementation is not at the point, at which errors in such cases will start to show up?
Or, is this a gross over-commit by underlying OS, and this state of affairs will stay like this?
Can try_reserve in principle reserve only realistic things?

[SimonSapin]

When an error is returned is entirely dependent of the underlying allocator. As far as I know Linux usually overcommits, Windows doesn’t.

https://play.rust-lang.org/?gist=aa668f38117983d1951b58307df04880&version=nightly allocates 120 PB but fails at 130 PB, it is exhausting 48 bits of address space.

Even on Linux though, this API can be useful for 32-bit programs when exhausting the address space is much more realistic.

[SimonSapin]

When an error is returned is entirely dependent of the underlying allocator. As far as I know Linux usually overcommits, Windows doesn’t.

https://play.rust-lang.org/?gist=aa668f38117983d1951b58307df04880&version=nightly allocates 120 PB but fails at 130 PB, it is exhausting 48 bits of address space.

Even on Linux though, this API can be useful for 32-bit programs when exhausting the address space is much more realistic.

[SimonSapin]

#43596 was the pre-RFC feature request for oom=panic. I’ve repurposed it as the tracking issue for that feature, since it and try_reserved are likely to be stabilized at different times. This leaves this issue as tracking only try_reserve.

[SimonSapin]

#43596 was the pre-RFC feature request for oom=panic. I’ve repurposed it as the tracking issue for that feature, since it and try_reserved are likely to be stabilized at different times. This leaves this issue as tracking only try_reserve.

[snf]

#52420 will change CollectionAllocErr to CollectionAllocErr<AllocError>

[snf]

#52420 will change CollectionAllocErr to CollectionAllocErr<AllocError>

[SimonSapin]

Will, or would. I’ve comment there that such a change in public API design merits being discussed separately from a large PR.

[SimonSapin]

Will, or would. I’ve comment there that such a change in public API design merits being discussed separately from a large PR.

[Ericson2314]

Yeah definitely. Even if it weren't for the 2018 delaying considering these things, I wouldn't expect it to be merged quick. I just make the PR to prove the feasibility and make my ideas clearer to / interactive for others.

Ignoring the benefits of allocator polymorphism itself (that would be #42774), I see 4 big benefits:

• Self discipline: Applications can prevent themselves statically from forgetting to handled allocations failures.

• Race freedom: allocation divergence due to try_reserving insufficient space are statically prevented.

• Expressiveness: Unclear how to do try_reserve-like things for "deep" data structures (those containing indirecation).

• Library flexibility: Implement functionality just once with a parameteric error type, and you get fallibility polymorphism: downstream can instantiate with ! to get today's divergence, or AllocError to allow them to be explicitly handled.

I thus advocating going with error polymorphism and abandoning try_reverve (I rather reuse try_ for the Result<(), E> copies of every method we'll need for backwards compat). (But, my PR keeps the existing meaning of try_reserve so as to allow a transition period where people can try both approaches.)

[Ericson2314]

Yeah definitely. Even if it weren't for the 2018 delaying considering these things, I wouldn't expect it to be merged quick. I just make the PR to prove the feasibility and make my ideas clearer to / interactive for others.

Ignoring the benefits of allocator polymorphism itself (that would be #42774), I see 4 big benefits:

• Self discipline: Applications can prevent themselves statically from forgetting to handled allocations failures.

• Race freedom: allocation divergence due to try_reserving insufficient space are statically prevented.

• Expressiveness: Unclear how to do try_reserve-like things for "deep" data structures (those containing indirecation).

• Library flexibility: Implement functionality just once with a parameteric error type, and you get fallibility polymorphism: downstream can instantiate with ! to get today's divergence, or AllocError to allow them to be explicitly handled.

I thus advocating going with error polymorphism and abandoning try_reverve (I rather reuse try_ for the Result<(), E> copies of every method we'll need for backwards compat). (But, my PR keeps the existing meaning of try_reserve so as to allow a transition period where people can try both approaches.)