Use jemalloc to compile dist on CI (x86_64-linux)

[blyxyas]

Based on this Zulip thread

I added Jemalloc for building LLVM, this should cut some CI time.

r? @Kobzol

[blyxyas]

I think this is good, and haven't gotten a bad review. Let's hope that I'm right!
@bors try @rust-timer queue

[bors]

⌛ Trying commit f6093dc with merge 1e6e02a...

[bors]

💔 Test failed - checks-actions

[blyxyas]

@bors try @rust-timer queue

[bors]

⌛ Trying commit 9780abb with merge df44909...

[Kobzol]

I'm pretty sure that this only applies to the first command, and not to the whole chain (commands after &&) 😆. Bash is annoying.

[bors]

☀️ Try build successful - checks-actions
Build commit: df44909 (df449090857e2b2b79b3969dd5df26b691e1877f)

[Kobzol]

Also for the next attempt, we don't need to run the benchmarks, just try :)

[rust-timer]

Finished benchmarking commit (df44909): comparison URL.

Overall result: no relevant changes - no action needed

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

@bors rollup=never
@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	3.9%	[3.7%, 4.1%]	2
Regressions ❌ (secondary)	2.5%	[2.2%, 3.6%]	6
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	3.9%	[3.7%, 4.1%]	2

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 646.206s -> 646.285s (0.01%)
Artifact size: 175.06 MiB -> 175.06 MiB (0.00%)

[blyxyas]

Hmmm... How should I check if this implies an improvement 😅 The action took 2h 24m, we can compare that with an arbitrary merge, and compared to e.g. #122054, this one seems to take an 40 minutes (1h 44m), is there any other number that I can check? I don't think that installing a simple jemalloc dependency could increase that much CI time (it should cut some minutes.)

[Kobzol]

We have to do two (or more) try builds in a row, so that the Docker cache kicks in.

@bors try

[bors]

⌛ Trying commit 9780abb with merge e0fba3d...

[bors]

☀️ Try build successful - checks-actions
Build commit: e0fba3d (e0fba3d79bdab3255decd88b77ff63eac7a3abfb)

[Kobzol]

Ah, the environment command is still wrong. We need to use it for the invocation of opt-dist, not for its build.
So it should look something like this:

ENV SCRIPT python3 ../x.py build --set rust.debug=true opt-dist && \
    LD_PRELOAD=`jemalloc-config --libdir`/libjemalloc.so.`jemalloc-config --revision` ./build/$HOSTS/stage0-tools-bin/opt-dist linux-ci -- python3 ../x.py dist \
    --host $HOSTS --target $HOSTS \
    --include-default-paths \

[blyxyas]

Sorry for wasting so much time. It's a little bit embarrassing to mess up a simple 2-line change so many times.

[Kobzol]

Nah, you're not wasting time, this could actually help in the end :) You should check out some of my PRs that modify CI workflows, I have run try builds several hundred times at least 😆 It's normal.

@bors try

[bors]

⌛ Trying commit ccadda5 with merge d6bf0f6...

[Kobzol]

Hmm, this prints an empty line:

FOO=BAR echo $(printenv) | grep FOO

so that would suggest that <key>=<val> command doesn't propagate the env to subprocesses. But that's maybe just some shell specialty, for normal processes it should hopefully work, since if the parent process has the env, it should propagate it to children by default (I hope).

[tgross35]

Hmm, this prints an empty line:

FOO=BAR echo $(printenv) | grep FOO

so that would suggest that <key>=<val> command doesn't propagate the env to subprocesses. But that's maybe just some shell specialty, for normal processes it should hopefully work, since if the parent process has the env, it should propagate it to children by default (I hope).

I think subshells only get exported environment, set -a makes that the default for assignments. Regular processes get the inline environment though, e.g. FOO=BAR sh -c 'printenv' | grep FOO works.

Seems like inline env only gets sent to the first process? I.e. echo but nothing else, also doesn't go through | pipes.

Feels like I learn something new about bash every day :)

[Kobzol]

Ah, makes sense. Yeah, for a Rust process, it should be enough if the parent gets the env, and then it should (hopefully) propagate through everything (I think that bootstrap does indeed propagate ENVs by default).

Feels like I learn something new about bash every day :)

Yeah, same 😆

[bors]

☀️ Try build successful - checks-actions
Build commit: d6bf0f6 (d6bf0f60201023352eef570a8989395f65cc66ec)

[cuviper]

You don't need -devel -- the base package includes a jemalloc.sh wrapper script that will set LD_PRELOAD for you.

[blyxyas]

Sorry for the delay, do you know where this jemalloc.sh script is located?

[cuviper]

It's /usr/bin/jemalloc.sh, so that's even in the PATH.

[Kobzol]

@bors try

[bors]

⌛ Trying commit ccadda5 with merge e3253dd...

[bors]

☀️ Try build successful - checks-actions
Build commit: e3253dd (e3253ddeecd1d4c4571563a5a33737dac4d63653)

[Kobzol]

Doesn't seem to have helped. It's hard to say whether it was used at all though. Maybe we could check e.g. in bootstrap.

[blyxyas]

I'll do some debugging in my machine to make sure that it is 100% being compiled with jemalloc.

[blyxyas]

Hmmm...
I'm having some problems with GLIBC (GLIBC_<version> version not found on /lib64/libc.so.6). I've tried upgrading my system (arch-based), but the problems seems to lie on Docker.

But I'm not sure how Docker would have this issue, if the Dockerfile does a yum upgrade on immediately, and installs glibc-devel 🤔

[Kobzol]

Inside the x64 dist Linux Docker, an old version of GLIBC (2.17, I think) is used, to make sure that the built artifacts are compatible with such old GLIBC distributions.

[blyxyas]

Hmmmm... I've been trying for a while (for like a month and a half) to just get this to run in my machine, including changing distro and getting a new hard drive (and changing from HDD to SSD). I'll close this to get the issue free for people that have the workflow already set up. It seems like my PC is just not having it :/