Stack overflow with Boxed array

[opus111]

This is possibly the same bug as

#40862

Using the latest version of Rust

rustc 1.27.2 (58cc626 2018-07-18)

The following code causes a stack overflow

#[test]
fn test_boxed() {
    let a = Box::new([-1; 3000000]);
}

Workarounds

Using Vec<T>

This does not have overhead.

let boxed_slice: Box<[u8]> = vec![0; 100_000_000].into_boxed_slice();
let boxed_array: Box<[u8; 100_000_000]> = vec![0; 100_000_000].into_boxed_slice().try_into().unwrap();

Unstably using new_uninit

This requires an unstable API and unsafe, but is more flexible.

let mut b = Box::new_uninit();
unsafe { std::ptr::write_bytes(b.as_mut_ptr(), 0, 100_000_000) };
let boxed_array: Box<[u8; 100_000_000]> = unsafe { b.assume_init() };

[nagisa]

Duplicate of #28008.

[Havvy]

But #28008 is closed. So we should probably keep one issue open for this. Especially since box syntax isn't coming any time soon AFAICT.

[opus111]

Just 2 cents from the peanut gallery... This bug has been around for 3 years. Surely Rust needs a stable way to allocate large arrays.

…

On Sat, Sep 1, 2018, 3:58 AM Ryan Scheel ***@***.***> wrote: But #28008 <#28008> is closed. So we should probably keep one issue open for this. Especially since box syntax isn't coming any time soon AFAICT. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#53827 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADUBIyLz22NknasbYL72xHVmMtxQTmpks5uWj4agaJpZM4WTto6> .

[nagisa]

We did make an attempt at solving this with placement-in, but that didn’t work out overall. Same problems are ailing the box syntax. An entirely different design is necessary.

[opus111]

If Box causes stack overflow, and box is not stable (and isn't coming soon), what is the recommended way to allocate a large block of memory in Rust?

[drewm1980]

@opus111 I am most certainly not a rust expert, but I bet the standard response would be to use the container types in the standard library, i.e. Vec. Depending on the application, increasing the allowable stack size may also be an option.

Is there a standard workaround for this (probably using unsafe for the cast) that works for any fixed size custom type?

[opus111]

@drewm1980 Thanks. I am also not an expert, but wouldn't a container also be allocated on the stack? My understanding is that Box or box is how one specifies that memory is not on the stack.

Surely, somebody is using Rust for large objects (e.g. images). How is it done? Does one have to be unsafe and call C?

[sfackler]

You could use a Vec rather than a fixed size array, or use the allocation APIs to have lower level control over initialization if you do need the fixed size: https://doc.rust-lang.org/std/alloc/index.html

[drewm1980]

@opus111 for images consider the ndarray crate. The standard libraries, and libraries like ndarray are doing heap allocations internally. They're probably using unsafe{} blocks internally, but it's OK, writers of standard libraries tend to know what they're doing.

Rust, like C, kept the core language very small. Like in C memory allocation is handled in a library, not at the language level. Syntactically speaking, to get an object on the heap, you have to first create it (on the stack like everything else in the core language) and pass it into a library function that does the heap allocation and a copy.

If you want to really double-down on using the stack for everything, I just found:

https://stackoverflow.com/questions/44003589/how-to-increase-the-stack-size-available-to-a-rust-library

It would be neat if rust had an ergonomic way to statically specify the max size of your stack (or have it automatically grow), and have the stack "just work" even for large objects for the main thread. If the rust devs focus on making the heap more ergonomic, people are going to use the heap more. If they focus on making the stack more ergonomic, people will use the stack more. Some people actually need to allocate their big objects on the heap, but some users are just being bitten by a default stack size limit that hasn't kept pace with the hardware, and a stack that can't grow transparently like the heap can.

[drewm1980]

I googled a bit more, and discussion around stack size issues in Rust's design unsurprisingly goes waaay back, i.e. :

#8345

"The conclusion was that we would always use split stacks, but on 64-bit platforms with overcommit we would make the initial segment large enough that it doesn't need to grow in typical use. This was also already implemented in the old runtime."

They apparently ripped out the split stack stuff, but I guess the "make the initial segment large enough that it doesn't need to grow in typical use" didn't happen, since by default Rust's stack overflows long before you actually run out of physical RAM.

Another recently commented on ancient thread asking for configurable stack size:
#5768

[opus111]

Thanks drew and steven. I am trying to avoid putting this object on the stack, because it is too big. I am surprised to learn that "to get an object on the heap, you have to first create it (on the stack...and pass it into a library function that does the heap allocation and a copy.". That doesn't sound very efficient :-(

Thanks for the pointer. Its not an image, but I'll try ndarray

[memoryruins]

Vec's method into_boxed_slice() returns a Box<[T]>, and does not overflow the stack for me.

vec![-1; 3000000].into_boxed_slice()

A note of difference with the vec! macro and array expressions from the docs:

This will use clone to duplicate an expression, so one should be careful using this with types having a nonstandard Clone implementation.

There is also the with_capacity() method on Vec, which is shown in the into_boxed_slice() examples.