Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rustc: enable parallel building #46406

Closed
wants to merge 1 commit into from
Closed

Conversation

@oxij
Copy link
Contributor

oxij commented Sep 8, 2018

Motivation for this change

The upstream issue is closed, all mentions of the issue on NixOS GitHub are old, I rebuilt it several times with this enabled, no issues.

rustc is on the critical path to firefox, without this firefox is always the very last package to be built on my machine when doing a mass rebuild because rustc take ages to build on a moderately slow single core.

Things done
  • Rebuilt >3 times without issues. Good enough for me.
@oxij oxij requested review from LnL7 and Mic92 as code owners Sep 8, 2018
@oxij oxij mentioned this pull request Sep 8, 2018
4 of 7 tasks complete
@LnL7
Copy link
Member

LnL7 commented Sep 8, 2018

The upstream issue was closed but not fixed AFAIK. I've never been able to reproduce the problems locally either but hydra builds regularly failed.

@GrahamcOfBorg
Copy link

GrahamcOfBorg commented Sep 8, 2018

Failure on aarch64-linux (full log)

Attempted: rustc

Partial log (click to expand)

tar: A lone zero block at 9623
install -d -m 755 '/nix/store/k9h9x4b1ncdy205dcx6qq7q9cg18xv60-git-2.18.0/lib/perl5/site_perl/5.28.0'
(cd perl/build/lib && tar cf - .) | \
(cd '/nix/store/k9h9x4b1ncdy205dcx6qq7q9cg18xv60-git-2.18.0/lib/perl5/site_perl/5.28.0' && umask 022 && tar xof -)
tar: Skipping to next header
tar: Exiting with failure status due to previous errors
make: *** [Makefile:2733: install] Error 2
builder for '/nix/store/0c80jp9nlf16f2kywfsy7sfhlh3kgpvl-git-2.18.0.drv' failed with exit code 2
cannot build derivation '/nix/store/k9p6bn9s3h3r241hnrcldi3mz7w8m1bw-rustc-1.27.0.drv': 1 dependencies couldn't be built
error: build of '/nix/store/k9p6bn9s3h3r241hnrcldi3mz7w8m1bw-rustc-1.27.0.drv' failed

@GrahamcOfBorg
Copy link

GrahamcOfBorg commented Sep 9, 2018

Timed out, unknown build status on x86_64-linux (full log)

Attempted: rustc

Partial log (click to expand)

   Compiling rustc_mir v0.0.0 (file:///build/rustc-1.27.0-src/src/librustc_mir)
   Compiling rustc_typeck v0.0.0 (file:///build/rustc-1.27.0-src/src/librustc_typeck)
   Compiling rustc_traits v0.0.0 (file:///build/rustc-1.27.0-src/src/librustc_traits)
   Compiling rustc_allocator v0.0.0 (file:///build/rustc-1.27.0-src/src/librustc_allocator)
   Compiling rustc_resolve v0.0.0 (file:///build/rustc-1.27.0-src/src/librustc_resolve)
   Compiling rustc_save_analysis v0.0.0 (file:///build/rustc-1.27.0-src/src/librustc_save_analysis)
   Compiling rustc_privacy v0.0.0 (file:///build/rustc-1.27.0-src/src/librustc_privacy)
   Compiling rustc_plugin v0.0.0 (file:///build/rustc-1.27.0-src/src/librustc_plugin)
building of '/nix/store/31iqgzrqi9jdnkp60zynym7q91l68grm-rustc-1.27.0.drv' timed out after 1800 seconds
error: build of '/nix/store/31iqgzrqi9jdnkp60zynym7q91l68grm-rustc-1.27.0.drv' failed

@Mic92
Copy link
Contributor

Mic92 commented Sep 9, 2018

This was the pull request, where we tried to enable it earlier: #21742

@oxij
Copy link
Contributor Author

oxij commented Sep 9, 2018

@andir
Copy link
Member

andir commented Sep 10, 2018

I did build the entire rebuild on one of my machines during the last day. It finished some time this morning. rustc was compiled without any visible issue. I had a load of around 48 to 600 during the entire process.

I also just build rust on it's own again. I think we should give this a try. None of us able to produce an error and if hydra does we are doing something wrong...

@LnL7
Copy link
Member

LnL7 commented Sep 10, 2018

Dealing with intermittent failures that have a large dependency tree like rustc is a pain in hydra. There are still thinks like literal sleeps in the rust makefiles, I'm almost certain we're going to revert this anyway. So this seems a bit pointless to me, even tho I'd like to have it enabled, since there's no indication that the problems might be resolved.

I suspect lower the io bandwidth in combination with concurrent builds are triggering problems that can't be reproduced on a machine with SSD or something similar like that.

@xeji
Copy link
Contributor

xeji commented Sep 24, 2018

If we're not sure about this we could put it in its own little Hydra jobset and restart the build multiple times to see what happens.

@Ekleog
Copy link
Member

Ekleog commented Oct 24, 2018

(triage) Anyone knows who to poke to put this PR in its Hydra jobset so that we can give log files to upstream and they can fix the issue?

@jonringer
Copy link
Contributor

jonringer commented Aug 8, 2019

I don't think this is an issue anymore, rustc-1.36 was able to max out my cores during the last build.

@andir @oxij @LnL7

@oxij
Copy link
Contributor Author

oxij commented Aug 9, 2019

@drewrisinger
Copy link
Contributor

drewrisinger commented May 4, 2020

In fact, this was not an issue even once on any of my builders since I opened this PR. I'm living happily with thing change applied for almost a year now.

@oxij are you ok with closing this PR then?

@JohnAZoidberg
Copy link
Member

JohnAZoidberg commented May 4, 2020

I think he meant that the concerns about the problems with this PR are now invalid.
I.e. we can probably merge this PR.

@drewrisinger
Copy link
Contributor

drewrisinger commented May 4, 2020

@GrahamcOfBorg eval
@GrahamcOfBorg build rustc

@Ekleog
Copy link
Member

Ekleog commented May 4, 2020

I've just tried nix-build --check -E 'let pkgs = import ./. { overlays = []; }; in pkgs.rustc' on ed755a6 ; looks like at least the time before the build gets oom-killed does use all my cores, though intermittently, probably due to limits in what's parallelizable.

Maybe upstream just forced parallel builds on?

@jonringer
Copy link
Contributor

jonringer commented May 5, 2020

Selection_035
seems to fan in and out depending on workload. At least most of it is already parallel

@jonringer jonringer closed this May 5, 2020
@jonringer
Copy link
Contributor

jonringer commented May 5, 2020

either way, seems to take about as long as specifying enableParallelBuilding = true; or not

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

10 participants
You can’t perform that action at this time.