Is make -j (parallel building) supported?

[joealden]

I am attempting to build opam for Solus (https://getsol.us) and by default, we use have a build system macro that, amongst other things, sets the max number of make jobs automatically based on the number of threads the machines has (on my machine, the macro executes make -j8 as I have a 4 core, 8 thread CPU).

The issue is that if I do not manually pass -j1 to make to make it build serially, the build will fail every time with the same error message, Error: mkdir: _build: File exists. Here is the build log if it is of any use:

+ make -j8
cd src_ext/jbuilder && ocaml bootstrap.ml && ./boot.exe
'/usr/bin/ocamllex.opt' -q src/meta_lexer.mll
'/usr/bin/ocamldep.opt' -modules src/action.ml src/action_intf.ml src/arg_spec.ml src/artifacts.ml src/bin.ml src/binary_kind.ml src/build.ml src/build_interpret.ml src/build_system.ml src/clflags.ml src/cm_kind.ml src/colors.ml src/config.ml src/context.ml src/dep_path.ml src/env.ml src/errors.ml src/exe.ml src/file_tree.ml src/findlib.ml src/gen_meta.ml src/gen_rules.ml src/glob_lexer.boot.ml src/import.ml src/inline_tests.ml src/install.ml src/install_rules.ml src/installed_dune_file.ml src/interned.ml src/jbuild.ml src/jbuild_load.ml vendor/boot/jbuilder_opam_file_format.ml vendor/boot/jbuilder_re.ml src/js_of_ocaml_rules.ml src/lib.ml src/loc.ml src/log.ml src/main.ml src/menhir.ml src/merlin.ml src/meta.ml src/meta_lexer.ml src/ml_kind.ml src/mode.ml src/module.ml src/module_compilation.ml src/modules_partitioner.ml src/ocaml_flags.ml src/ocamldep.ml src/odoc.boot.ml src/opam_file.ml src/ordered_set_lang.ml src/package.ml src/path.ml src/per_item.ml src/preprocessing.ml src/print_diff.ml src/process.ml src/report_error.ml src/scheduler.ml src/scope.ml src/setup.boot.ml src/sexp.ml src/string_with_vars.ml src/sub_system.ml src/sub_system_intf.ml src/sub_system_name.ml src/super_context.ml src/syntax.ml src/top_closure.ml src/utils.ml src/utop.ml src/variant.ml src/vfile_kind.ml src/watermarks.ml src/workspace.ml > boot-depends.txt
'/usr/bin/ocamldep.opt' -modules src/stdune/caml/caml.ml src/stdune/caml/result.ml src/stdune/caml/result_compat.ml > boot-depends.txt
'/usr/bin/ocamldep.opt' -modules src/fiber/fiber.ml > boot-depends.txt
'/usr/bin/ocamldep.opt' -modules src/ocaml-config/ocaml_config.ml > boot-depends.txt
'/usr/bin/ocamldep.opt' -modules src/stdune/stdune.ml src/stdune/ansi_color.ml src/stdune/array.ml src/stdune/char.ml src/stdune/common.ml src/stdune/comparable.ml src/stdune/either.ml src/stdune/exn.ml src/stdune/filename.ml src/stdune/hashtbl.ml src/stdune/import.ml src/stdune/int.ml src/stdune/io.ml src/stdune/list.ml src/stdune/map.ml src/stdune/map_intf.ml src/stdune/option.ml src/stdune/or_exn.ml src/stdune/ordering.ml src/stdune/pp.ml src/stdune/result.ml src/stdune/set.ml src/stdune/set_intf.ml src/stdune/staged.ml src/stdune/string.ml > boot-depends.txt
'/usr/bin/ocamldep.opt' -modules src/usexp/usexp.ml src/usexp/parser_automaton_internal.ml src/usexp/sexp_ast.ml src/usexp/table.ml > boot-depends.txt
'/usr/bin/ocamldep.opt' -modules src/xdg/xdg.ml > boot-depends.txt
'/usr/bin/ocamlc.opt' -g -w -40 -o boot.exe unix.cma boot.ml
src_ext/jbuilder/_build/install/default/bin/jbuilder build  opam-installer.install opam.install
src_ext/jbuilder/_build/install/default/bin/jbuilder build  src/tools/opam_installer.exe
Error: mkdir: _build: File exists
make: *** [Makefile:104: opam.install] Error 1
make: *** Waiting for unfinished jobs....

I found issue #2706, where @AltGr said the following:

This is a known bug; for parallel builds, use make fast, which relies on ocp-build. (OCaml)Makefiles are only really there for bootstrapping, they are not reliable for either parallel or incremental builds.

Although this issue is almost 2 years old, does this still hold true that parallel builds are not reliable? Also, I cannot see a fast entry in the Makefile which @AltGr suggested to use, but I'm guessing that it has since been removed.

If parallel make builds are not supported, that is fine. I just want to make sure that I am not being stupid!

[dra27]

make is just a frontend to calling jbuilder which is parallel. So there’s no value in calling make -j - we should add .NOTPARALLEL: to formalise this.

[joealden]

@dra27 I see, thanks for the explanation. So that would force make to run serially even if -j was passed? That sounds like the perfect solution to me. What should I do with this issue? Should I leave it open and possibly rename it or just close it?

[dra27]

.NOTPARALLEL causes GNU make to ignore it, yes - I don’t think it’s portable, but the build system already requires GNU make. I’d leave this open and let the pull request which fixes it close it via commit message. It’s most definitely a bug that running with -j doesn’t work.

[dra27]

(See the GNU make manual for full info)

[joealden]

Okay, that sounds like a good idea. Thanks for your time.

[MSoegtropIMC]

A note on parallel build on 4.07: as far as I can see on Windows there is nothing parallel about the build. Is the build process different if there exists a binary jbuilder / dune in the path?

I am building with the following command in a pretty much untouched environment:

    make world -f Makefile.nt
    make bootstrap -f Makefile.nt
    make opt -f Makefile.nt
    make opt.opt -f Makefile.nt
    make install -f Makefile.nt

[dra27]

@MSoegtropIMC - the build of opam is parallel, those are the commands for compiling OCaml. -f Makefile.nt hasn't been needed since OCaml 4.05 (and will fail in 4.08). It's also faster to do make world.opt, unless you have a definitely need to perform a bootstrap.

I can't remember what the state of parallel building was for 4.07 (it's certainly fine in 4.08+). However, the build is handled by a separate script which itself invokes make again, so make -j cold should not cause a parallel build of OCaml.

[MSoegtropIMC]

@dra27 : thanks for the info. This is the way I build OCaml for Coq for Windows releases since years and didn't really review it over time. I prefer to do the bootstrap for building Coq.

I anyway planned to move the build to opam - on Windows this was still fairly tricky when I tried it 1/2 year back, but doable. One more reason for picking up this work.