Not optimal parallel build of dependencies

[vicuna]

Original bug ID: 5754
Reporter: meyer
Status: resolved (set by @damiendoligez on 2017-03-03T10:48:10Z)
Resolution: suspended
Priority: high
Severity: feature
Version: 4.00.0
Target version: undecided
Category: -for ocamlbuild use https://github.com/ocaml/ocamlbuild/issues
Has duplicate: #5796
Related to: #4934
Monitored by: @gasche @ygrek @jberdine @hcarty

Bug description

For this example:

a1.ml:
a2.ml:
a3.ml:
a4.ml:
b1.ml:
open A1
open A2
b2.ml:
open A3
open A4
c.ml:
open B1
open B2

we have this dependency graph:

A1    A2   A3    A4
  \  /       \  /
   B1         B2
     \__   __/
        \ /
         C

Of course it's apparent that for -j 4 option, ocamlbuild should trivially recognize this case and build A1 & A2 & A3 & A4 in parallel, and later B1 & B2 also in parallel and then finally C, but currently it does not.

What it does is to build A1 & A2 in paraller, then build B1, then build A3 & A4 and the build B2, and finally C.

[vicuna]

Comment author: berenger

Ocamlbuild doesn't parallelize the build.
Using -j 1 to -j 8 with a step size of 1 shows that
there is almost no parallelization of the build.

[vicuna]

Comment author: meyer

I agree that the tool currently is does not extremely parallelize the build. This fact mostly comes from the reason of handling dynamic dependencies automatically.

[vicuna]

Comment author: @hhugo

Can someone give some detail/explanation/example on why handling dynamic dependencies prevent from doing proper parallel build ?

[vicuna]

Comment author: @gasche

hhugo, there is my understanding of things:

http://gallium.inria.fr/blog/ocamlbuild-parallelization/

Do not hesitate to ask about any further details.

[vicuna]

Comment author: @gasche

(I also built some tools, on top of unrelated work of Arthur Charguéraud, to explore ocamlbuild's parallelization efficiency on actual runs. If someone actually starts working on this again, please ping me back to make me release them, it's very helpful.)

[vicuna]

Comment author: @damiendoligez

ocamlbuild is now a separate project that lives on GitHub.
PR transferred to ocaml/ocamlbuild#207