ocamlopt defaults to system install on OCaml 5 bytecode-only architectures

[art-w]

I'm running into a compilation issue on the ppc64 architecture, which is bytecode-only since OCaml 5 (= does not provide ocamlopt, only ocamlc):

$ eval $(opam env --switch=5.1.0)
$ opam install cmdliner
$ opam install fmt
# Error: ~/.opam/5.1.0/lib/cmdliner/cmdliner.cmi
#        is not a compiled interface for this version of OCaml.

This machine happens to have a system install of OCaml 4.13.1, so ocamlopt is still found during the compilation of cmdliner but with a version that's incompatible with the opam switch expectation:

$ ocamlc -version
5.1.0
$ ocamlopt -version
4.13.1

$ .opam/5.1.0/bin/ocamlobjinfo .opam/5.1.0/lib/cmdliner/cmdliner.cmi
Wrong magic number:
this tool only supports object files produced by compiler version
	5.1.0
This seems to be a compiled interface file (cmi) for an older version of OCaml.

$ /bin/ocamlobjinfo .opam/5.1.0/lib/cmdliner/cmdliner.cmi
Unit name: Cmdliner
Interfaces imported:
	2f078433075037f22235ff84d6f9b58c	Cmdliner
  ...

I'm not sure if opam could prevent this... I found a couple of different strategies in the wild for how people are checking for native support:

• dune appears to be unaffected by this issue... I heard rumors that it searches for ocamlopt in $OPAM_SWITCH_PREFIX/bin (?)
• cmdliner is only checking for the existence of an ocamlopt executable (cc @dbuenzli)
• topkg delegates to ocamlfind and ocamlbuild (?)
• ocamlfind doesn't have the concept that ocamlopt could be missing (? cc @gerdstolpmann)
• ocamlbuild checks if the ARCH is unsupported from the $(ocamlc -where)/Makefile.config, which isn't quite correct as here ARCH=power is supported but with NATIVE_COMPILER=false (cc @gasche)
• (most likely some custom makefiles in the wild doing their own stuff?)

I'm afraid a fix would be required in each of those build systems (I don't think opam could hide the system install of ocamlopt?). But does someone has recommendation on the right way to address this issue? (do we need to check the ocamlc/ocamlopt versions explicitly or trust the NATIVE_COMPILER or check that the binaries are in the same directory or ...?)

[dbuenzli]

Well yes in general having a mix of tools of the same toolchain from different installs in your PATH is going to be a disaster. Welcome to the ruins we build our software on. And that's not limited to ocaml, I wrote here a few other scenarios for a meltdown (in a same kind of issue reported by @jhjourdan about ocamlnat in down). The solution in general is to isolate your build environments and avoid multiple installs of the same toolchain in these build environments.

If you really want something reliable that copes with these mixed installs we would need a) A binary that is guaranteed to be present in every configuration of an OCaml install b) a way for this binary to point to a file or output that has precise information about every bit of the OCaml install that is configurable so that build system only access the tools that actually exist in this configuration.

Note that ages ago, already well aware of this exact issue, I had requested for ocamlc -config to report about the availability of ocamlopt, it was demeed to hard to provide at that point but it seems that this finally has come with OCaml… 5.1. Given that cmdliner still supports 4.08, that's not going to be a fix for the foreseable future.

• topkg delegates to ocamlfind and ocamlbuild (?)

No topkg constructs a Topkg.Conf.OCaml.t value by reading ocamlc -config. According to the doc string of Topkg.Conf.OCaml.native it looks if the standard library directory has a libasmrun static library. I guess something similar could be done in cmdliner if that's actually a reliable procedure.

(Also given the mess we see in opam end-users environments I rather not do something as brittle as checking that tools are in the same directory)

[kit-ty-kate]

opam could detect this particular case (has ocamlc but no ocamlopt in the current switch, but ocamlopt is in the previous PATH) and display a warning, but i don't think we can do much more

[art-w]

Yeah the system install is definitively an issue here, but as a non-root user it's awkward to vanish /bin/ocamlopt (Could opam install hide it thanks to bwrap?... I'm kidding. But perhaps this has some ground in security/reproducibility?)

a) Can we assume that ocamlc is always available?
b) I agree that solutions would have to be backward compatible, but also maintained in the forseeable future (... until OCaml 5.1 becomes the new lowerbound). Perhaps different tricks are required to detect which features are available in the mean time?

Regarding the native compiler support, which options should be favored?

1. Any issue with checking for the existence of $(ocamlc -where)/libasmrun.$EXT_LIB? (Thanks for the details on how topkg does it!)
2. Otherwise, any issue with checking the versions [ "$(ocamlc -version)" = "$(ocamlopt -version 2> /dev/null)" ] ?
3. I liked $(ocamlc -where)/Makefile.config until discovering your comment that it only reflects the configuration and not the real outcome. Can it be trusted for NATIVE_COMPILER? Or is this recent and do we also need to check for ARCH=none?

[gerdstolpmann]

@art-w Responding to your question: ocamlfind should work when ocamlopt is missing (i.e. not found in the search path), though I think this hasn't been tested for quite a while. Also, I think you can convince ocamlfind that ocamlopt is not existing if you link it to /usr/bin/false, e.g.

$ mkdir fakecmds
$ cd fakecmds
$ ln -s /usr/bin/false ocamlopt
$ ln -s /usr/bin/false ocamloptp
$ cd ..
$ PATH="$(pwd)/fakecmds:$PATH" ./configure ...

but you will have have to customize findlib.conf, because it includes pointers to the .opt variants of the tools by default.

[kit-ty-kate]

opam could detect this particular case (has ocamlc but no ocamlopt in the current switch, but ocamlopt is in the previous PATH) and display a warning, but i don't think we can do much more

As pointed out during the dev meeting this might not be so easy as this would be against the idea of opam being OCaml agnostic (modulo FormatUpgrade and opamInit)