Parallel xz/gz tarball generation triggers capacity overflow on i686-unknown-gnu-linux with bundled llvm (regression from adea17e)

[akien-mga]

When building rust 1.22.1 on i686-unknown-gnu-linux with the bundled LLVM (due to too old system LLVM - RHEL 6 - or too recent - Mageia 7), the tarball generation fails as it tries to allocate more than 2GB in memory.

Here's a complete log of the issue on Mageia 7 i586 (posing at i686... but that's another issue, the bug can be reproduced on real i686 too).

Dist std stage2 (i686-unknown-linux-gnu -> i686-unknown-linux-gnu)
thread 'main' panicked at 'capacity overflow', src/liballoc/raw_vec.rs:714:8
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::imp::backtrace::tracing::imp::unwind_backtrace
   1: std::sys_common::backtrace::_print
   2: std::panicking::default_hook::{{closure}}
   3: std::panicking::default_hook
   4: std::panicking::rust_panic_with_hook
   5: std::panicking::begin_panic
   6: std::panicking::begin_panic_fmt
   7: rust_begin_unwind
   8: core::panicking::panic_fmt
   9: core::panicking::panic
  10: <alloc::vec::Vec<T>>::reserve
  11: std::io::impls::<impl std::io::Write for alloc::vec::Vec<u8>>::write_all
  12: tar::builder::append
  13: installer::tarballer::Tarballer::run
  14: installer::generator::Generator::run
  15: fabricate::main
command did not execute successfully: "/builddir/build/BUILD/rustc-1.22.1-src/build/i686-unknown-linux-gnu/stage0-tools/i686-unknown-linux-gnu/release/fabricate" "generate" "--product-name=Rust" "--rel-manifest-dir=rustlib" "--success-message=std-is-standing-at-the-ready." "--image-dir" "/builddir/build/BUILD/rustc-1.22.1-src/build/tmp/dist/rust-std-1.22.1-i686-unknown-linux-gnu-image" "--work-dir" "/builddir/build/BUILD/rustc-1.22.1-src/build/tmp/dist" "--output-dir" "/builddir/build/BUILD/rustc-1.22.1-src/build/dist" "--package-name=rust-std-1.22.1-i686-unknown-linux-gnu" "--component-name=rust-std-i686-unknown-linux-gnu" "--legacy-manifest-dirs=rustlib,cargo"
expected success, got: exit code: 101
failed to run: /builddir/build/BUILD/rustc-1.22.1-src/build/bootstrap/debug/bootstrap install
Build completed unsuccessfully in 0:02:41

@cuviper helped me debug it and could confirm the issue on RHEL 6 in similar conditions (bundled LLVM):

<jistone> Akien, oh lovely - I just got the same error on rhel6 i686
* jistone shakes his fist in Akien's direction...
<jistone> but I guess I can use that to inspect how it got so huge
<jistone> (the irony is that we don't really even need the dist tarballs, but that's how it gets installed locally)
<jistone> 898176352 Jan 12 17:41 librustc_llvm-4f58def34f5809eb.so
<jistone> that's the main culprit, if you're using the bundled/static llvm
<jistone> I measure rust-std at 1214680252 bytes total
<jistone> and rust's doubling vec capacity means going over 1GB tries 2GB, which is more than isize::MAX, so it dies
<jistone> rust-lang/rust#31446

@cuviper found that it would likely be related to adea17e, and I confirm that reverting this patch in rust 1.22.1's rust-installer allows a successful build on Mageia 7 i586 (build log).

[ignatenkobrain]

We had similar issue in RPM which got fixed by rpm-software-management/rpm@a60f36a

[cuviper]

I like that the previous method was fully streaming. I also expect that the gzip time would be dwarfed by the xz time, making parallel computation not so useful. @alexcrichton, did you measure much benefit from this?

A small fix might be to seed the vector with a capacity that has more room to double, like Vec::with_capacity(125_000_000) -> 250, 500, 1000, 2000, still less than 32-bit isize::MAX. (Assuming the allocator can find that much contiguous address space...)

[alexcrichton]

@cuviper unfortunately I don't remember, but I'd be down for alternative fixes!