Disable bytecode compilers in ocamltest

[stedolan]

Current, ocamltest compiles every test case four times, using all of ocamlc, ocamlc.opt, ocamlopt and ocamlopt.opt. The only reason to compile with both ocamlc and ocamlc.opt is to compare their output, and likewise for ocamlopt.

However, since #128 , we don't actually compare the output of ocamlc and ocamlc.opt, and I'm not sure there much need to compare ocamlopt and ocamlopt.opt either. So, this PR stops ocamltest from using ocamlc and ocamlopt at all, greatly speeding the upstream testsuite. We still compile every case in bytecode and native modes, but only once.

[mshinwell]

@dra27 does this look right to you?

[stedolan]

CI times:

	before	after
closure	14m 44s	12m 32s
closure_cfg_local	15m 7s	11m 45s
flambda1	15m 52s	14m 47s
flambda1_cfg_local	16m 22s	16m 43s
flambda2	19m 27s	14m 43s
flambda2_cfg	19m 10s	12m 14s
macos	33m 10s	27m 14s

(no idea what's going on in flambda1_cfg_local beyond "CI times are noisy")

[mshinwell]

It's a bit of a shame the macos one takes so long, but it is useful to check this. I wonder if we could speed that up somehow. @xclerc do you remember from previous macos problems whether they were usually on the testsuite, or on the build? One option might not be to run the testsuite on that platform until we can get a faster (arm64/M1?) worker.

[xclerc]

What prompted the pull request was a build issue (#411), but
I can imagine catching problems with the testsuite.

I am wondering: (i) is merge blocked until all checks pass? and
(ii) are checks cancelled when a pull request is merged?
If "no" is the answer to both questions, our policy may be
to merge when the macOS check is still in progress and
the other ones are green. We would not wait, and if anything
breaks we would quite easily find the first breakage and work
on a smaller diff.

[xclerc]

Another obvious possibility to to only have a daily run of
the check on macOS.

[mshinwell]

@xclerc maybe we should run the testsuite on macos only for the "push" events, leaving the "pull_request" events to check only the build on macos? I'd still like to make sure PRs don't break the build.

[xclerc]

I am not sure I follow, don't you want to build on every "push"
if you want to be sure the tip is always buildable on macOS?

Looking at the definitions of the jobs, I wonder whether we could
save some time when building the compiler for dune. By building
the strict minimum there -- i.e. no documentation, no debugger, etc.

[xclerc]

Looking at the definitions of the jobs, I wonder whether we could save some time when building the compiler for dune. By building the strict minimum there -- i.e. no documentation, no debugger, etc.

For the avoidance of doubt, that would benefit to all jobs, not only the macOS one.

[mshinwell]

There may well be some simplifications there, indeed.

For macos, just build on pull request pushes, but for pushes to the actual branch, build and then run the testsuite.

[xclerc]

This might be what we are looking for: https://github.community/t/trigger-workflow-only-on-pull-request-merge/17359/2
i.e. run the job on merge.

[mshinwell]

Yeah, that's what I was thinking of, but just the build step should also happen as it does at the moment (i.e. every push to a pull request).

[xclerc]

Great, I will try to set up this configuration tomorrow.

[dra27]

LGTM! For vague reference, I did something similar - but Windows-specific - in ocaml-multicore/ocaml-multicore@b84109e (which intentionally wasn't merged).

[mshinwell]

@dra27 thanks