Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LLVM ERROR: out of memory building rustc on powerpc ppc64 #60294

Open
donbright opened this issue Apr 26, 2019 · 13 comments

Comments

@donbright
Copy link

@donbright donbright commented Apr 26, 2019

howdy

this is on the GCC Compile Farm,

cat /proc/cpuinfo
...
cpu             : POWER7 (architected), altivec supported
clock           : 3550.000000MHz
revision        : 2.1 (pvr 003f 0201)

timebase        : 512000000
platform        : pSeries
model           : IBM,8231-E2B
machine         : CHRP IBM,8231-E2B

donb@gcc1-power7:~/rust$ cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (AltArch)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (AltArch)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7:server"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Updating only changed submodules
Submodules updated in 0.06 seconds
   Compiling bootstrap v0.0.0 (/home/donb/rust/src/bootstrap)
LLVM ERROR: out of memory
error: Could not compile `bootstrap`.

Caused by:
  process didn't exit successfully: `
/home/donb/rust/build/powerpc64-unknown-linux-gnu/stage0/bin/rustc 
--edition=2018 --crate-name bootstrap src/bootstrap/lib.rs 
--color always --crate-type lib --emit=dep-info,link 
-C debug-assertions=off -C overflow-checks=on -C metadata=9f05f37567d58316 
-C extra-filename=-9f05f37567d58316 --out-dir /home/donb/rust/build/bootstrap/debug/deps 
-C incremental=/home/donb/rust/build/bootstrap/debug/incremental 
-L dependency=/home/donb/rust/build/bootstrap/debug/deps 
--extern build_helper=/home/donb/rust/build/bootstrap/debug/deps/libbuild_helper-e331fe5eaa037b7f.rlib 
--extern cc=/home/donb/rust/build/bootstrap/debug/deps/libcc-8a7c326a946223a3.rlib 
--extern cmake=/home/donb/rust/build/bootstrap/debug/deps/libcmake-520764bc3cb193af.rlib 
--extern filetime=/home/donb/rust/build/bootstrap/debug/deps/libfiletime-0a60bd675cad0447.rlib 
--extern getopts=/home/donb/rust/build/bootstrap/debug/deps/libgetopts-82500d29ceaf20c3.rlib 
--extern lazy_static=/home/donb/rust/build/bootstrap/debug/deps/liblazy_static-0307d963e4e609d3.rlib 
--extern libc=/home/donb/rust/build/bootstrap/debug/deps/liblibc-8856cbc7f48cd073.rlib 
--extern num_cpus=/home/donb/rust/build/bootstrap/debug/deps/libnum_cpus-c26408f25cdfd471.rlib 
--extern petgraph=/home/donb/rust/build/bootstrap/debug/deps/libpetgraph-54385ef12605e7df.rlib 
--extern serde=/home/donb/rust/build/bootstrap/debug/deps/libserde-178b37a1543723d2.rlib 
--extern serde_derive=/home/donb/rust/build/bootstrap/debug/deps/libserde_derive-d1253535fb5b2579.so 
--extern serde_json=/home/donb/rust/build/bootstrap/debug/deps/libserde_json-e5a1f4fba42098bb.rlib 
--extern time=/home/donb/rust/build/bootstrap/debug/deps/libtime-e101a1c489935147.rlib 
--extern toml=/home/donb/rust/build/bootstrap/debug/deps/libtoml-9a6584354dc2c335.rlib 
-Cdebuginfo=2` (signal: 6, SIGABRT: process abort signal)

failed to run: /home/donb/rust/build/powerpc64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /home/donb/rust/src/bootstrap/Cargo.toml

this is a very recent clone of github.com/rust-lang/rust

thanks

@jonas-schievink

This comment has been minimized.

Copy link
Member

@jonas-schievink jonas-schievink commented Apr 26, 2019

@donbright How many cores and how much RAM does this system have?

@donbright

This comment has been minimized.

Copy link
Author

@donbright donbright commented Apr 27, 2019

hello. It has 64 cores and 64 GB of RAM.

@donbright

This comment has been minimized.

Copy link
Author

@donbright donbright commented Apr 27, 2019

to be more specific it is machine 110 at https://gcc.gnu.org/wiki/CompileFarm shared amongst many people, so the RAM availability goes up and down

@cuviper

This comment has been minimized.

Copy link
Member

@cuviper cuviper commented Apr 30, 2019

When I've seen memory errors before, especially on 32-bit hosts, it's usually on the large crates like librustc -- but bootstrap is pretty small!

@andreasbaumann

This comment has been minimized.

Copy link

@andreasbaumann andreasbaumann commented Aug 22, 2019

Can confirm this is also a problem with rust 1.37.0 on 32-bit Archlinux32:

   Compiling syntax_ext v0.0.0 (/build/rust/src/rustc-1.37.0-src/src/libsyntax_ext)
LLVM ERROR: out of memory
error: Could not compile `rustc`.
@andreasbaumann

This comment has been minimized.

Copy link

@andreasbaumann andreasbaumann commented Aug 22, 2019

I see a -j8 here

command did not execute successfully: "/usr/bin/cargo" "build" "--target" "i686-unknown-linux-gnu" "-j" "8" "--release" "--frozen" "--features" "" "--manifest-path" "/build/rust/src/rustc-1.37.0-src/src/rustc/Cargo.toml" "--message-format" "json"

I wonder how you can tame parallelity, with RUSTFLAGS (analog to MAKEFLAGS='-j1')?

@andreasbaumann

This comment has been minimized.

Copy link

@andreasbaumann andreasbaumann commented Aug 22, 2019

When monitoring the rustc process using top I see:

9708 user  20   0 1150852 944228 106960 S  96.7   5.8   1:04.96 rustc
...
 9708 user  20   0 3709340   3.3g 119936 S  99.7  21.2  13:24.16 rustc
...
 9708 user  20   0 4034952   3.6g 119936 S 100.0  23.3  13:54.19 rustc

This must fail in 32-bit address space.

Is there a way to compile and link incrementally?

Or differently put: how are you producing the prebuilt binaries for i686? Via
cross-compiling in a 64-bit address space?

@mati865

This comment has been minimized.

Copy link
Contributor

@mati865 mati865 commented Aug 22, 2019

I wonder how you can tame parallelity, with RUSTFLAGS (analog to MAKEFLAGS='-j1')?

I think ./x.py build -j<num> should work.

Or differently put: how are you producing the prebuilt binaries for i686? Via
cross-compiling in a 64-bit address space?

Yes, they are cross-compiled from 64 bit Docker container: https://github.com/rust-lang/rust/blob/2ec6f340cd29f289fea1c5a672195ceeaf44c475/src/ci/docker/dist-i686-linux/Dockerfile

@andreasbaumann

This comment has been minimized.

Copy link

@andreasbaumann andreasbaumann commented Aug 22, 2019

There is a python ./x.py build -j"$(nproc)" in PKGBUILD, so yes, this is not really good. :-)

Reported as:

https://bugs.archlinux.org/task/62952?project=1&string=rust

@andreasbaumann

This comment has been minimized.

Copy link

@andreasbaumann andreasbaumann commented Aug 22, 2019

   Compiling fmt_macros v0.0.0 (/build/rust/src/rustc-1.37.0-src/src/libfmt_macros)
LLVM ERROR: out of memory
error: Could not compile `rustc`.

command did not execute successfully: "/usr/bin/cargo" "build" "--target" "i686-unknown-linux-gnu" "-j" "1" "--release" "--frozen" "--features" "" "--manifest-path" "/build/rust/src/rustc-1.37.0-src/src/rustc/Cargo.toml" "--message-format" "json"

So basically, the rust compiler is not compilable from source on 32-bit.. can you confirm?

Is it possible to remove LTO optimization as a possible cause in config.toml?

Looking through some config.toml (https://github.com/rust-lang/rust/blob/master/config.toml.example) options I see two promizing options to reduce memory usage:

#thin-lto = false
#link-jobs = 0

Having binutils/gcc is LTO enabled by default? Can I switch it off?

@mati865

This comment has been minimized.

Copy link
Contributor

@mati865 mati865 commented Aug 22, 2019

So basically, the rust compiler is not compilable from source on 32-bit.. can you confirm?

No idea, I haven't ran 32 bit OS for years.

Is it possible to remove LTO optimization as a possible cause in config.toml?

AFAIK it's not enabled by default.

Looking through some config.toml (https://github.com/rust-lang/rust/blob/master/config.toml.example ) options I see two promizing options to reduce memory usage:

Those two options only affect LLVM build, your build fails when building Rust parts.

@cuviper

This comment has been minimized.

Copy link
Member

@cuviper cuviper commented Aug 24, 2019

For 32-bit hosts, I suggest checking your debuginfo settings and possibly turning them down. The rustc crate in particular requires a ton of memory with full debuginfo.

The official upstream builds use --debuginfo-level-std=1 for all targets, for just line info, leaving all other debuginfo implicitly off. x86_64 and i686 Linux builds run on x86_64 Linux CI hosts with native target toolchains -- meaning the i686 build is using i686 rustc even though the kernel is 64-bit. All other Linux architectures are cross-compiled from x86_64 rustc.

In Fedora, all of our builds are native. I use full --debuginfo-level=2 for 64-bit hosts, and then a narrower --debuginfo-level-std=2 for 32-bit hosts, and this has been good enough.

@andreasbaumann

This comment has been minimized.

Copy link

@andreasbaumann andreasbaumann commented Sep 9, 2019

Thanks, --debuginfo-level=1 works around the huge memory usage and I was able to build it for
Archlinux32 (IA32) in a 32-bit chroot on a Archlinux 64-bit host and kernel (which should be the
same as on a real IA32).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.