ocamlbuild -where is not relocatable

[gridbugs]

ocamlbuild -where always returns the value of the $OCAMLBUILD_LIBDIR makefile variable set during build time. This means that if a pre-compiled instance of ocamlbuild is ever relocated, ocamlbuild -where will return the wrong result.

Practically, this causes problems for dune package management. Dune builds opam packages by running their build and install commands in a transient sandbox, after which all installed artifacts are moved into a shared opam-like filesystem and the sandbox is deleted. This results in errors like:

Warning: Won't be able to compile a native plugin
Failure: Cannot find "ocamlbuild.cmo" in ocamlbuild -where directory.

...because the directory returned by ocamlbuild -where was inside the (now deleted) build sandbox. (possibly related: #321)

My plan for addressing this is to implement ocamlbuild -where by locating the library directory relative to the path to the ocamlbuild executable (by means of Sys.executable_name), possibly keeping the current behaviour as a fallback.

See also: ocaml/dune#10290

[gasche]

You report the following behavior on the dune issue:

$ _build/_private/default/.pkg/ocamlbuild/target/bin/ocamlbuild -where
/home/s/tmp/bonsai/_build/.sandbox/7158a9da761d39477e414a6f2dd9d3fc/_private/default/.pkg/ocamlbuild/target/lib/ocamlbuild

Two comments:

1. The standard way to deal with this issue in Linux distributions is to set the correct path at configure-time, that is, the path where the software will be installed and not the paths where the software is built. The configure logic of ocamlbuild does the following to set OCAMLBUILD_LIBDIR:

   OCAMLBUILD_LIBDIR ?= \
     $(or $(shell opam config var lib 2>/dev/null),\
          $(shell ocamlfind printconf destdir 2>/dev/null),\
          $(OCAML_LIBDIR))
   

   That is, it tries to ask standard tools about where ocamlbuild is going to be installed, and it lets the people invoking the script override the default logic by defining the OCAMLBUILD_LIBDIR variable yourself.

   Maybe the dune packaging script for ocamlbuild (where is it? can you point to the sources?) is not doing the right thing, by configuring ocamlbuild with a temporary sandbox prefix instead of using the final installation prefix?

   (Note that this discussion above is not about relocatibility, as setting the correct installation directory at configure-time does not make the software relocatable. I don't know if you want to fix the build of ocamlbuild by dune, or if you actually need to make ocamlbuiild relocatable for other reasons.)

2. The logic to compute ocamlbuild -where is in fact more complex already than just fetching OCAMLBUILD_LIBDIR, see the source code at https://github.com/ocaml/ocamlbuild/blob/d72d55397689bfc6a90e28a0fd49db3e860b76b0/src/ocamlbuild_where.ml and in particular the last change made to the logic (in 2017) at 81f8495 . It may be the case that it is already possible to configure the environment, at ocamlbuiild invocation time, to get the behavior that you want -- and in particular to get a form of relocatability.

[gridbugs]

Dune builds opam packages by running the build and install commands from the package spec. For ocamlbuild this is:

build: [
  [
    make
    "-f"
    "configure.make"
    "all"
    "OCAMLBUILD_PREFIX=%{prefix}%"
    "OCAMLBUILD_BINDIR=%{bin}%"
    "OCAMLBUILD_LIBDIR=%{lib}%"
    "OCAMLBUILD_MANDIR=%{man}%"
    "OCAML_NATIVE=%{ocaml:native}%"
    "OCAML_NATIVE_TOOLS=%{ocaml:native}%"
  ]
  [make "check-if-preinstalled" "all" "opam-install"]
]

The path variables such as bin and lib are set within the sandbox where the build is performed rather than within the destination filesystem but perhaps that is something we (dune devs) should consider changing.

[gasche]

The path variables such as bin and lib are set within the sandbox where the build is performed rather than within the destination filesystem but perhaps that is something we (dune devs) should consider changing.

This sounds like the wrong choice indeed. What does the opam client do?

A quick git grep indicates that there are 91 packages in the opam-repository that use %{lib} in their opam file, and a quick look at the uses suggest that most of them will be broken by the current choice. For example:

post-messages: [
  "The file 'hevea.sty' has been installed in %{lib}%/hevea but latex won't see it by itself" {success}
]