libcore panic on ridiculous compilation

[AuroransSolis]

I had a friend try to compile my solution for Advent of Code day 2 with 100k inputs instead of the 250 given in the puzzle, since I was unable to compile it myself. Because it ate up so much memory. He ended up allowing an external 2TB HDD to be a swap disk to compile it (on a Lenovo IdeaPad Yoga 11S), and it took eight days before it spat out this error (with said friend's name replaced with user for sake of privacy):

user@Computer:~/Desktop/aoc_2018/day_2$ cargo bench
warning: profiles for the non root package will be ignored, specify profiles at the workspace root:
package:   /home/user/Desktop/aoc_2018/day_2/Cargo.toml
workspace: /home/user/Desktop/aoc_2018/Cargo.toml
    Updating crates.io index
   Compiling cfg-if v0.1.6
   Compiling ugly_array_decl v0.1.0 (/home/user/Desktop/aoc_2018/day_2/ugly_array_decl)
   Compiling packed_simd v0.3.1
   Compiling galaxy_brain v0.1.0 (/home/user/Desktop/aoc_2018/day_2/galaxy_brain)
   Compiling day_2 v0.1.0 (/home/user/Desktop/aoc_2018/day_2)

thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: "SendError(..)"', src/libcore/result.rs:1009:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.
error: Could not compile `day_2`.

Caused by:
  process didn't exit successfully: `rustc --crate-name day_2 day_2/src/main.rs --color always --emit=dep-info,link -C opt-level=3 --test -C metadata=32bd32e061311793 -C extra-filename=-32bd32e061311793 --out-dir /home/user/Desktop/aoc_2018/target/release/deps -L dependency=/home/user/Desktop/aoc_2018/target/release/deps --extern galaxy_brain=/home/user/Desktop/aoc_2018/target/release/deps/libgalaxy_brain-ac99af5f18838f56.so --extern packed_simd=/home/user/Desktop/aoc_2018/target/release/deps/libpacked_simd-61a0a6a6c31c29f4.rlib --extern ugly_array_decl=/home/user/Desktop/aoc_2018/target/release/deps/libugly_array_decl-add69c2d264ca168.so` (signal: 9, SIGKILL: kill)

I don't want to trouble him to compile it again, so the code can be found here.

Here's the output of rustc --version --verbose:

user@Computer:~/Desktop/aoc_2018/day_2$ rustc --version --verbose
rustc 1.32.0-nightly (14997d56a 2018-12-05)
binary: rustc
commit-hash: 14997d56a550f4aa99fe737593cd2758227afc56
commit-date: 2018-12-05
host: x86_64-unknown-linux-gnu
release: 1.32.0-nightly
LLVM version: 8.0

Note on memory use: we don't actually know how much memory it took to compile this thing. All I know is that a different friend tried compiling it and it ate up the better part of their 40GB of RAM plus writing a 60GB swap file. As for the 2TB swap disk, I don't know how much of that it ate up since friend 1's laptop was frozen for five of the eight days it was compiling.

[steveklabnik]

Triage: i had to apply this patch to get things to compile:

diff --git a/day_2/galaxy_brain/src/lib.rs b/day_2/galaxy_brain/src/lib.rs
index 208c3b0..55e68e4 100644
--- a/day_2/galaxy_brain/src/lib.rs
+++ b/day_2/galaxy_brain/src/lib.rs
@@ -14,7 +14,7 @@ pub fn construct_galaxy_brain(_input: TokenStream) -> TokenStream {
 use packed_simd::u8x32;
 use ugly_array_decl::ugly_array_decl;

-const BYTES: [u8; 6750] = *include_bytes!("../../zesterer-input.txt");
+const BYTES: [u8; 7000] = *include_bytes!("../../zesterer-input.txt");
 const LINES: usize = 250;
 const ZEROS: u8x32 = u8x32::splat(0);
 const ONES: u8x32 = u8x32::new(1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
@@ -40,4 +40,4 @@ fn part_2_ff() -> String {
         }
     }
     unreachable!();
-}
\ No newline at end of file
+}
diff --git a/day_2/src/main.rs b/day_2/src/main.rs
index 9e2f244..957fab6 100644
--- a/day_2/src/main.rs
+++ b/day_2/src/main.rs
@@ -60,7 +60,7 @@ use std::hint::unreachable_unchecked;
 use packed_simd::u8x32;
 use ugly_array_decl::big_ugly_array_decl;

-const BYTES: [u8; 2700000] = *include_bytes!("../fixed-big-input.txt");
+const BYTES: [u8; 2800000] = *include_bytes!("../fixed-big-input.txt");
 const LINES: usize = 100_000;
 const ZEROS: u8x32 = u8x32::splat(0);
 const ONES: u8x32 = u8x32::new(1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
@@ -241,4 +241,4 @@ fn zesterer(b: &mut Bencher) {
 fn og_zesterer(b: &mut Bencher) {
     let i = include_bytes!("../ameo-input.txt");
     b.iter(|| black_box(original_zesterer(i)));
-}*/
\ No newline at end of file
+}*/

I didn't go the whole way through a whole compile, so I don't know if it will take a long time or not, but yeah.

[Mark-Simulacrum]

This crate expands to 60MB of source code, primarily repeating a bunch of expressions that look like this:

    [u8x32::new(BYTES[0], BYTES[1], BYTES[2], BYTES[3], BYTES[4], BYTES[5],
                BYTES[6], BYTES[7], BYTES[8], BYTES[9], BYTES[10], BYTES[11],
                BYTES[12], BYTES[13], BYTES[14], BYTES[15], BYTES[16],
                BYTES[17], BYTES[18], BYTES[19], BYTES[20], BYTES[21],
                BYTES[22], BYTES[23], BYTES[24], BYTES[25], 0, 0, 0, 0, 0, 0),
            u8x32::new(BYTES[27], BYTES[28], BYTES[29], BYTES[30], BYTES[31],
                BYTES[32], BYTES[33], BYTES[34], BYTES[35], BYTES[36],
                BYTES[37], BYTES[38], BYTES[39], BYTES[40], BYTES[41],
                BYTES[42], BYTES[43], BYTES[44], BYTES[45], BYTES[46],
                BYTES[47], BYTES[48], BYTES[49], BYTES[50], BYTES[51],
                BYTES[52], 0, 0, 0, 0, 0, 0),
            u8x32::new(BYTES[54], BYTES[55], BYTES[56], BYTES[57], BYTES[58],
                BYTES[59], BYTES[60], BYTES[61], BYTES[62], BYTES[63],
                BYTES[64], BYTES[65], BYTES[66], BYTES[67], BYTES[68],
                BYTES[69], BYTES[70], BYTES[71], BYTES[72], BYTES[73],
                BYTES[74], BYTES[75], BYTES[76], BYTES[77], BYTES[78],
                BYTES[79], 0, 0, 0, 0, 0, 0),
            u8x32::new(BYTES[81], BYTES[82], BYTES[83], BYTES[84], BYTES[85],
                BYTES[86], BYTES[87], BYTES[88], BYTES[89], BYTES[90],
                BYTES[91], BYTES[92], BYTES[93], BYTES[94], BYTES[95],
                BYTES[96], BYTES[97], BYTES[98], BYTES[99], BYTES[100],
                BYTES[101], BYTES[102], BYTES[103], BYTES[104], BYTES[105],
                BYTES[106], 0, 0, 0, 0, 0, 0),
            u8x32::new(BYTES[108], BYTES[109], BYTES[110], BYTES[111],
                BYTES[112], BYTES[113], BYTES[114], BYTES[115], BYTES[116],
                BYTES[117], BYTES[118], BYTES[119], BYTES[120], BYTES[121],
                BYTES[122], BYTES[123], BYTES[124], BYTES[125], BYTES[126],
                BYTES[127], BYTES[128], BYTES[129], BYTES[130], BYTES[131],
                BYTES[132], BYTES[133], 0, 0, 0, 0, 0, 0),
[...]
            u8x32::new(BYTES[2699973], BYTES[2699974], BYTES[2699975],
                BYTES[2699976], BYTES[2699977], BYTES[2699978],
                BYTES[2699979], BYTES[2699980], BYTES[2699981],
                BYTES[2699982], BYTES[2699983], BYTES[2699984],
                BYTES[2699985], BYTES[2699986], BYTES[2699987],
                BYTES[2699988], BYTES[2699989], BYTES[2699990],
                BYTES[2699991], BYTES[2699992], BYTES[2699993],
                BYTES[2699994], BYTES[2699995], BYTES[2699996],
                BYTES[2699997], BYTES[2699998], 0, 0, 0, 0, 0, 0)];

Looking at some logging from the compiler, I think this is primarily slow and using memory because we are (a) type checking each of these index expressions separately, which involves re-proving various bounds. Each of the indexing expressions desugars to have a unique lifetime assigned to parts of it so we're not able to deduplicate the proofs as well as we perhaps could.

That said, I think this input is pretty pathological. 60MB of code that is primarily concentrated around a few huge consts/statics isn't going to perform well. I'm not sure there's much to learn from looking into this further, beyond what we already know about wanting to improve caching/normalization around the trait solver. I'm going to go ahead and close this issue even though this particular case isn't fixed since I don't think anyone is likely to care much about the particulars and the larger problem IMO isn't going to get helped by keeping it open.