Add monorepo benchmark

[gridbugs]

This adds a current-bench benchmark that measures the time it takes to build a large monorepo composed from opam packages.

[rgrinberg]

Would it be possible to move monorepo-bench to bench/monorepo?

[rgrinberg]

Why do we need this mega library? It's not enough to build everything individually?

[gridbugs]

The monorepo is made up of many packages and some of them contain libraries that don't build. I've made a list of the libraries that don't build and used them to construct the list of libraries in this file.

[rgrinberg]

What do you mean don't build? As in fail to build, or aren't being requested to build at all?

[gridbugs]

I mean that they fail to build in a monorepo setting.

[rgrinberg]

I see. So building this mega library is supposed to fail as well then?

[gridbugs]

No, the list of libraries in this file are just the ones that build in a monorepo setting.

[rgrinberg]

Shouldn't this depend on $(DUNE_TO_BENCHMARK) as well?

[gridbugs]

I don't think so. $(DUNE_TO_BENCHMARK) isn't a target of this makefile; it's assumed that it's already been built when make bench is run.

[rgrinberg]

I don't think this works since we have a top level dune-file instead. In any case, I'd like to see this under bench/ if possible.

[gridbugs]

Ah you're right, even with this file when I run dune build it attempts to build the project in monorepo-bench. I'll move the monorepo benchmark inside the "bench" directory but I still need to find a way to prevent the monorepo bench project being seen by dune. My intention is that it should only ever be built inside a docker container when running the benchmark.

[rgrinberg]

You could exclude it with (dirs ..).

[gridbugs]

I ended up moving the monorepo-bench dir to bench/monorepo and adding (dirs :standard \ monorepo) to bench/dune. I don't understand why (dirs ..) works. Did you mean that putting (dirs ..) in a dune file would exclude that file's directory from the build? Why is it necessary to refer to the parent directory?

[gridbugs]

I've moved the monorepo benchmark to bench/monorepo and fixed the dune files so it's excluded from normal builds. As an experiment I tried getting rid of the (library ...) stanza with the long list of dune files, but afterwards running dune build doesn't build any of the packages in duniverse. Is there different command I should be using to exercise dune in this situation? It's probably moot though because if I was to, say, add a library dependency for all the libraries in all the packages included in the monorepo, there would be some that don't build in a monorepo setting. I maintain a list of libraries that don't build in such a setting and have excluded them from the list of library dependencies of the monorepo library.

[rgrinberg]

but afterwards running dune build doesn't build any of the packages in duniverse. Is there different command I should be using to exercise dune in this situation?

Which command did you try?

[gridbugs]

I tried running dune build and dune build @all but neither of them built targets inside "duniverse". If I remove (vendored_dirs *) from that directory then both of those commands bulid targets inside it.

[rgrinberg]

If I remove (vendored_dirs *) from that directory then both of those commands bulid targets inside it.

How annoying. I really regret adding all this special behavior to vendored directories. We can either:

1. Remove vendored_dirs and your library stanza
2. Keep things as is and leave a note as to why.

Your choice.

[gridbugs]

I'm going to elect to keep things as they are (I'll update this PR with a note explaining why) as some libraries don't build in a monorepo setting, so without the specific list of libraries to build we would have to remove packages so that every library in every remaining package builds in a monorepo setting.

[gridbugs]

After discussing this with @snowleopard I'm moving most of this benchmark to a separate repo (currently https://github.com/gridbugs/ocaml-monorepo-benchmark/tree/main/benchmark but the plan is to move it to https://github.com/ocaml-dune). I'll keep the bench.Dockerfile and benchmark runner tool in this PR and update the dockerfile to download the rest of the benchmark from its external location. The reasoning behind this is to avoid adding the ~1mb opam-monorepo lockfile to dune, consolidating the benchmarks with the tool that generates the benchmarks (also in the ocmal-monorepo-benchmark repo) and so that the monorepo can be used for non-dune purposes in the future.

[snowleopard]

To add one more reason: I think it's worth planning to have more than one monorepo benchmark in future, and pushing all of them to the Dune repo doesn't seem right.

[gridbugs]

I've moved all the files specific to the content of the benchmark to https://github.com/ocaml-dune/ocaml-monorepo-benchmark and made a tag of that repo 2023-03-13.0 which gets downloaded when building the docker image for the monorepo benchmark.

[gridbugs]

@rgrinberg regarding the dune file listing all the library dependencies, I've moved it to the ocaml-monorepo-benchmark repo (here https://github.com/ocaml-dune/ocaml-monorepo-benchmark/blob/main/benchmark/dune). I've also added to that repo a description of why the file is necessary, in https://github.com/ocaml-dune/ocaml-monorepo-benchmark/blob/main/benchmark/README.md.

Are there any other changes needed before we merge this?

[Alizter]

@gridbugs Could you add a note in the benchmark section of hacking.rst how to use this?

[gridbugs]

@gridbugs Could you add a note in the benchmark section of hacking.rst how to use this?

Yep, here: #7361