Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

erlang: disable parallel building #51644

Merged
merged 1 commit into from Dec 8, 2018

Conversation

@ivan
Copy link
Member

ivan commented Dec 7, 2018

Motivation for this change

All of the Erlang runtimes fail to build on my Hydra with parallel building enabled (implemented in #45484). The same problem was observed on an official Hydra machine (and again), but ignored as a temporary issue.

This PR disables parallel building to avoid this build failure.

<3> ERLC   ../ebin/asn1ct_gen_ber_bin_v2.beam
<3> ERLC   ../ebin/asn1ct_rtt.beam
<3> ERLC   ../ebin/asn1ct_constructed_ber_bin_v2.beam
<3> GEN    asn1ct_eval_ext.erl
<3>{"init terminating in do_boot",{undef,[{asn1ct_func,start_link,[],[]},{prepare_templates,gen_asn1ct_eval,1,[{file,"prepare_templates.erl"},{line,58}]},{init,start_it,1,[]},{init,start_em,1,[]}]}}
<3>init terminating in do_boot ()
<3>
<3>Crash dump is being written to: erl_crash.dump...done
<3>make[2]: *** [Makefile:139: asn1ct_eval_ext.erl] Error 1
<3>make[2]: Leaving directory '/build/source/lib/asn1/src'
<3>make[1]: *** [/build/source/make/otp_subdir.mk:29: opt] Error 2
<3>make[1]: Leaving directory '/build/source/lib'
<3>make: *** [Makefile:550: secondary_bootstrap_build] Error 2
<3>builder for '/nix/store/5565imavv7imy514fycp2r1qi61qx2m6-erlang-18.3.4.8.drv' failed with exit code 2
<3> ERLC   ../ebin/asn1ct_gen_ber_bin_v2.beam
<3> ERLC   ../ebin/asn1ct_rtt.beam
<3> ERLC   ../ebin/asn1ct_constructed_ber_bin_v2.beam
<3> GEN    asn1ct_eval_ext.erl
<3>{"init terminating in do_boot",{undef,[{asn1ct_func,start_link,[],[]},{prepare_templates,gen_asn1ct_eval,1,[{file,"prepare_templates.erl"},{line,58}]},{init,start_em,1,[]},{init,do_boot,3,[]}]}}
<3>init terminating in do_boot ()
<3>
<3>Crash dump is being written to: erl_crash.dump...done
<3>make[2]: *** [Makefile:139: asn1ct_eval_ext.erl] Error 1
<3>make[2]: Leaving directory '/build/source/lib/asn1/src'
<3>make[1]: *** [/build/source/make/otp_subdir.mk:29: opt] Error 2
<3>make[1]: Leaving directory '/build/source/lib'
<3>make: *** [Makefile:550: secondary_bootstrap_build] Error 2
<3>builder for '/nix/store/sd9kp905syyfqb7gczrp3qpzxvhvl1pj-erlang-19.3.6.11.drv' failed with exit code 2
<3> ERLC   ../ebin/asn1ct_gen_ber_bin_v2.beam
<3> ERLC   ../ebin/asn1ct_rtt.beam
<3> ERLC   ../ebin/asn1ct_constructed_ber_bin_v2.beam
<3> GEN    asn1ct_eval_ext.erl
<3>{"init terminating in do_boot",{undef,[{asn1ct_func,start_link,[],[]},{prepare_templates,gen_asn1ct_eval,1,[{file,"prepare_templates.erl"},{line,58}]},{init,start_em,1,[]},{init,do_boot,3,[]}]}}
<3>init terminating in do_boot ({undef,[{asn1ct_func,start_link,[],[]},{prepare_templates,gen_asn1ct_eval,1,[{_},{_}]},{init,start_em,1,[]},{init,do_boot,3,[]}]})
<3>
<3>Crash dump is being written to: erl_crash.dump...done
<3>make[2]: *** [Makefile:138: asn1ct_eval_ext.erl] Error 1
<3>make[2]: Leaving directory '/build/source/lib/asn1/src'
<3>make[1]: *** [/build/source/make/otp_subdir.mk:29: opt] Error 2
<3>make[1]: Leaving directory '/build/source/lib'
<3>make: *** [Makefile:557: secondary_bootstrap_build] Error 2
<3>builder for '/nix/store/2dq6bp93q788bnz4gbbhhiyzrpgh38nz-erlang-20.3.8.9.drv' failed with exit code 2
<3> ERLC   ../ebin/asn1ct_gen_ber_bin_v2.beam
<3> ERLC   ../ebin/asn1ct_rtt.beam
<3> ERLC   ../ebin/asn1ct_constructed_ber_bin_v2.beam
<3> GEN    asn1ct_eval_ext.erl
<3>{"init terminating in do_boot",{undef,[{asn1ct_func,start_link,[],[]},{prepare_templates,gen_asn1ct_eval,1,[{file,"prepare_templates.erl"},{line,58}]},{init,start_em,1,[]},{init,do_boot,3,[]}]}}
<3>init terminating in do_boot ({undef,[{asn1ct_func,start_link,[],[]},{prepare_templates,gen_asn1ct_eval,1,[{_},{_}]},{init,start_em,1,[]},{init,do_boot,3,[]}]})
<3>
<3>Crash dump is being written to: erl_crash.dump...done
<3>make[2]: *** [Makefile:138: asn1ct_eval_ext.erl] Error 1
<3>make[2]: Leaving directory '/build/source/lib/asn1/src'
<3>make[1]: *** [/build/source/make/otp_subdir.mk:29: opt] Error 2
<3>make[1]: Leaving directory '/build/source/lib'
<3>make: *** [Makefile:575: secondary_bootstrap_build] Error 2
<3>builder for '/nix/store/f9h1xf6k0sjri4h0wkgx28b5z1xwxqkw-erlang-21.1.3.drv' failed with exit code 2

I don't know why these builds succeed elsewhere, but they fail reliably on my 8 core/16 thread Intel(R) Xeon(R) W-2145 CPU @ 3.70GHz with a NixOS (4.19.7 kernel, btrfs) in VMware on a Windows 10 1803 host.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Assured whether relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@hedning
hedning approved these changes Dec 7, 2018
Copy link
Contributor

hedning left a comment

So we hit this close to a 100% on aarch_64 which are mostly build with at least -j8, but much less on x86_64, which is mostly built with -j4, but it does happen: here's a x86_64 failure from staging-next with -j8. That suggests we could limit jobs to 4, but I can't find any simple way to do that. So I'd say it's best to just disable.

This fixes build errors like

 GEN    asn1ct_eval_ext.erl
{"init terminating in do_boot",{undef,[{asn1ct_func,start_link,[],[]},{prepare_templates,gen_asn1ct_eval,1,[{file,"prepare_templates.erl"},{line,58}]},{init,start_em,1,[]},{init,do_boot,3,[]}]}}
init terminating in do_boot ({undef,[{asn1ct_func,start_link,[],[]},{prepare_templates,gen_asn1ct_eval,1,[{_},{_}]},{init,start_em,1,[]},{init,do_boot,3,[]}]})

that occur on some machines but not others.
@ivan ivan force-pushed the ivan:erlang-disable-parallel-building branch to 82e5590 Dec 8, 2018
@Mic92 Mic92 merged commit b4a2ed8 into NixOS:staging Dec 8, 2018
9 checks passed
9 checks passed
grahamcofborg-eval ^.^!
Details
grahamcofborg-eval-check-meta config.nix: checkMeta = true
Details
grahamcofborg-eval-nixos-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A manual
Details
grahamcofborg-eval-nixos-options nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A options
Details
grahamcofborg-eval-nixpkgs-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A manual
Details
grahamcofborg-eval-nixpkgs-tarball nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A tarball
Details
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A unstable
Details
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
Details
trofi added a commit to trofi/otp that referenced this pull request Jan 17, 2020
The build failure is reproducible on a 4-core CPU when using
massive parallelism:

```
$ make -j21 -l4
...
 GEN    asn1ct_eval_ext.erl
{"init terminating in do_boot",
  {undef,[{asn1ct_func,start_link,[],[]},
          {prepare_templates,gen_asn1ct_eval,1,
           [{file,"prepare_templates.erl"},{line,58}]},
            {init,start_it,1,[]},{init,start_em,1,[]}]}}
init terminating in do_boot ()

Crash dump is being written to: erl_crash.dump...done
make[2]: *** [Makefile:139: asn1ct_eval_ext.erl] Error 1
```

Here `asn1ct_eval_ext.erl` attempts to load `asn1ct_func.beam`
via `prepare_templates.erl` before it's compiled into `.beam` file.

The change pulls in a dependency on `asn1ct_func.beam`.

Reported-by: WGH
Bug: NixOS/nixpkgs#51644
Bug: https://bugs.gentoo.org/705500
Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.