Relocatable OCaml - --with-relative-libdir

[dra27]

This is the second of three PRs which implement Relocatable OCaml as proposed in ocaml/RFCs#53. The series of changes in this PR combine to allow the absolute location of the Standard Library (e.g. /usr/lib/ocaml) to be removed from both the C runtime (ocamlrun and libcamlrun.a, etc.) and also from the Config module in the ocamlcommon compiler-libs library. The patches address sections 3 & 4 of the RFC.

The key changes are:

• A new primitive %standard_library_default allows an OCaml program to determine the default of the Standard Library as a compile-time constant. In particular, it means that rather than Config.standard_library_default being fixed once when config.ml itself is compiled, it allows the compiler to determine its value each time a program is linked.
• A new command line option, -set-runtime-default allows the calculated value for the %standard_library_default primitive to be overridden when linking an executable.
• A configure option, --with-relative-libdir, which allows the compiler to be configured to expect to find the Standard Library in a location specified relative to where the compiler is running from. When ./configure is run with no arguments, the default location of the Standard Library is /usr/local/lib/ocaml and binaries are installed to /usr/local/bin. The equivalent relative compiler would be configured with --with-relative-libdir=../lib/ocaml.
• Utilisation of both the BUILD_PATH_PREFIX_MAP support added in #1515 and usage of -fdebug-prefix-map-style options to the C compiler to make the compiler's artefacts considerably more reproducible when --with-relative-libdir has been specified.

These changes necessitate considerable churn in the start-up routines for the bytecode runtime, and in the way argv[0] is being processed. While this code is in motion, there are several additional changes which aren't a strict requirement of the main change, but are here because this is all being shaken up:

• Building the mingw-w64 port using the MSYS2 improves slightly, in that configure now correctly recognises that it is a Cygwin-like environment, and uses cygpath et al as necessary.
• On Windows, if backslashes are included in --prefix (e.g. ./configure --prefix='C:\OCaml') then these are preserved in the resulting compiler (this implements a slightly more sensible version of #658). Apart from grinding my own axe where this is concerned, it prevents "mixed" slash paths (which look particularly amateur) from being generated when the compiler is configured --with-relative-libdir.
• There is an obscure, but nonetheless present, bug in the order in which the bytecode runtime processes the various options for starting up. In particular, if an executable has been compiled with -custom, it is possible to direct the resulting executable to load a different bytecode image by manipulating argv[0]. This issue is fixed here, and on normal systems (where caml_executable_name is implemented), an executable compiled with -custom only loads the bytecode image it was compiled with and can no longer be directed to a load a different one.
• As part of improving reproducibility, the Cygwin version of ocamlopt now calls as directly, just as Linux does, rather than going via gcc.

Technical background

The crux of this PR is this pair of innocuous-looking lines:

ocaml/runtime/dynlink.c

Line 91 in 0728f6a

if (stdlib == NULL) stdlib = OCAML_STDLIB_DIR;

and

ocaml/utils/config.generated.ml.in

Line 23 in 0728f6a

let standard_library_default = {@QS@|@ocaml_libdir@|@QS@}

which, by default, are:

  if (stdlib == NULL) stdlib = "/usr/local/lib/ocaml";

and

let standard_library_default = {|/usr/local/lib/ocaml|}

Here, /usr/local is the installation prefix. A consequence of being relocatable, as defined in the RFC, is that this string cannot appear (in any encoding!) in the compiler binaries. At present, the string is present in the Config module and in the bytecode runtime. Its presence in the bytecode runtime means that essentially all bytecode executables contain the location of the OCaml Standard Library, either directly, if compiled with -output-obj, -output-complete-exe, etc., or through the ocamlrun executable, if compiled with default options or -custom. Conversely, native executables only contain the location of the OCaml Standard Library if they link the Config module from the ocamlcommon.cmxa¹.

For the compiler or runtime, installed to /usr/local/bin/ocamlopt or /usr/local/bin/ocamlrun, it is straightforward for ocamlopt/ocamlrun to determine the directory containing the running executable (/usr/local/bin) and to instead contain the location ../lib/ocaml and thus combine the two to derive /usr/local/lib/ocaml. In the code, ../lib/ocaml is the default value, which is static and may be explicit-relative or absolute, where the calculated /usr/local/lib/ocaml is the effective value, which is dynamic and must be absolute. Thus, at module initialisation in:
https://github.com/ocaml/ocaml/blob/0728f6af2aae32a97c2a7a1214c25736a26a479b/utils/config.common.ml.in#L23-L30
we can instead calculate the effective value "/usr/local/lib/ocaml" from the default standard_library_default = {|../lib/ocaml|}.

If only it were that straightforward! The calculation of the effective location requires the caller to agree to be invoked from a binary located in a specific place relative to the effective location. The compiler is not the only consumer of Config.standard_library. Consider the trivial program display.ml:

Printf.printf "Config.standard_library = %S\n" Config.standard_library

compiled with:

$ pwd
/home/dra/work
$ command -v ocamlopt
/usr/local/bin/ocamlopt
$ ocamlopt -o display -I +compiler-libs ocamlcommon.cmxa display.ml
$ ocamlopt -where
/usr/local/lib/ocaml
$ ./display
/usr/local/lib/ocaml

Today, the two final commands display the same result. If Config.standard_library always uses the effective value as the default for the Standard Library location, ocamlopt -where will continue to display /usr/local/lib/ocaml but ./display will suddenly display /home/dra/work/../lib/ocaml (or something similar, but nonetheless not /usr/local/lib/ocaml).

Finally, bytecode executables pose an additional problem. Let us extend the trivial program slightly:

Unix.realpath Config.standard_library
|> Printf.printf "Config.standard_library = %S\n"

compiled with:

$ /usr/local/bin/ocamlc -o display -I +unix -I +compiler-libs unix.cma ocamlcommon.cma display.ml

Supposing we have two installations of the same version and configuration of OCaml, one in /usr/local and another in ~/.opam/default. In this case, we have:

$ /usr/local/bin/ocamlrun -config
standard_library_default: /usr/local/lib/ocaml
...
shared_libs_path:
  /usr/local/lib/ocaml/stublibs

$ /usr/local/bin/ocamlc -where
/usr/local/lib/ocaml

$ ~/.opam/default/bin/ocamlrun -config
standard_library_default: /home/dra/.opam/default/lib/ocaml
...
shared_libs_path:
  /home/dra/.opam/default/lib/ocaml/stublibs

$ /home/dra/.opam/default/bin/ocamlc -where
/home/dra/.opam/default/lib/ocaml

Now, executing ./display involves loading dllunixbyt.so from the stublibs directory. In these two invocations:

$ /usr/local/bin/ocamlrun ./display
/usr/local/lib/ocaml

$ ~/.opam/default/bin/ocamlrun ./display
/usr/local/lib/ocaml

there are two important things we reasonably expect to happen:

1. ./display will display the same path in each case, which will be /usr/local/lib/ocaml (Config.standard_library for the compiler it was built with)
2. Each ocamlrun will load dllunixbyt.so from its own installation - i.e. /usr/local/bin/ocamlrun will load /usr/local/lib/ocaml/stublibs/dllunixbyt.so and ~/.opam/default/bin/ocamlrun will load ~/.opam/default/lib/ocaml/stublibs/dllunixbyt.so

This leads to what I think is not an entirely obvious distinction. There are two locations to consider: the default location of the Standard Library for the runtime, and the default Standard Library location for the mutator. Config.standard_library_default refers to the mutator (i.e. program's view), but this may not necessarily be the same value as the runtime needs for OCAML_STDLIB_DIR in runtime/dynlink.c. In practice, this only affects standalone bytecode images - i.e. the situation where the runtime executable and the bytecode image are in separate locations. In all other compilations (including native code; though the native runtime doesn't ever care about the location of the Standard Library), there are still two values, but they are always the same.

The changeset is best reviewed commit-by-commit (and, I'm afraid, armed with the "Technical background" explanation...):

• enum caml_byte_program_mode is augmented with a new APPENDED option which is used for -custom. caml_main then uses caml_byte_program_mode to differentiate between a #!-style "standalone" bytecode image running via ocamlrun and a -custom executable. While moving the code around, I renamed the COMPLETE_EXE enumeration constant to EMBEDDED as the mode is also used with -output-obj.
• The runtime-launch-info is one of two places where the binary directory, rather than library directory is embedded in a file. The format for runtime-launch-info is trivially extended to recognise . as referring to the directory containing the compiler (note that runtime-launch-info is not a generally-configurable file - supporting arbitrary relative paths here would be a hypothetical installation where the runtime executables are installed to a different location from the compiler, which isn't supported or needed at present).
• A symbol caml_runtime_standard_library_default for the runtime default value, principally so that OCAML_STDLIB_DIR is only referred to in once place.
• %standard_library_default is the most involved change, introducing a compile-time constant to retrieve the mutator default value. In native code, the Standard Library location is only present if the Config module is linked. ocamlopt therefore has to create the string as part of linking an executable, which is done using a similar mechanism to the caml_apply, caml_curry and caml_send functions - a new field in the cmx header records that the compilation unit uses the %standard_library_default primitive. At link-time, if any of the compilation units has set this flag, the linker creates caml_standard_library_nat containing the location of the standard library and synthesises references to it. For bytecode, a similar technique is used, except that the linker already has the full list of %-primitives which are used by the program, so there is no need for cmo format to be changed. There are various strategies on offer for exactly where and how the value is stored. It is needed to link the bytecode runtime (i.e. the value has to be included in the same places where C tables of primitives and so forth are generated). It is also has to be included in bytecode images where no C is being produced. Given that -output-complete-exe and so forth share the same value for both runtime and mutator, for bytecode images I've opted to add a new OSLD bytecode section containing the mutator value, which is read to caml_standard_library_default (not to caml_runtime_standard_library_default). Then, as with the other compile-time constants, it's just a matter of adding another caml_sys_const_ primitive in runtime/sys.s which returns an OCaml copy of that string. I've proposed this as a %-primitive, rather than a "known" C primitive for two reasons. Firstly, the behaviour of this is a compile-time constant (i.e. something which the compiler must work out when linking the program), and the other compile-time constants are % primitives. Secondly, the standard library location only wants to be embedded when it's actually needed; if the native runtime used a C primitive, there would have to be a sentinel value or a default for caml_standard_library_nat - it seems worse to have a C primitive sat in the runtime which could return an invalid value (i.e. caml_sys_const_standard_library_default always returns a correct value in bytecode, but if exposed in native code, it would be possible to end up with a program which called it, but got a default empty string back). Given that these primitives are never intended to be called by user-code (because of the difference in linking), unlike in #13465, I think it's better to have the C primitive for bytecode only, and the completely synthesise the function in ocamlopt only when needed.
• %standard_library_default provides a mechanism which means that the string constant carved into utils/config.ml when the compiler was built can now be determined when the compiler is run. The default value determined by the compiler is simply that same absolute path. The -set-runtime-default provides a mechanism to change that default value when linking a specific program.
• These two features together combine to allow Config.standard_library_default to be changed to be the result of %standard_library_default. This mechanism is a necessary consequence of %standard_library_default for cross-compilation. When linking a cross-compiling version of ocamlopt, that compiler is built with a compiler which uses a host standard library but the resulting cross-compiler it's linking should default to a different target standard library (see the change in Makefile.cross).
• At this point, %standard_library_default means that "/usr/local/lib/ocaml" has now moved the Config module to the executables themselves. Where before, ocamlopt and ocamlc.byte both had the string location linked in code via Config.standard_library, it's now instead embedded in caml_standard_library_nat for ocamlopt (and ocamlc.opt) and in an OSLD section for ocamlc.byte. The next commit then allows explicit-relative values of this path to be interpreted by both the runtime and compiler. This is then activated using -set-runtime-default, so that caml_standard_library_nat is changed to be "../lib/ocaml". Most of the complexity here arises from the fact that the bytecode C runtime needs to be able to do this. Especially in the light of the work on ld.conf in #PR1, rather than implementing the logic in both C and OCaml, the logic is implemented just in C and the Config module uses a new caml_sys_get_stdlib_dirs primitive, which is passed the result of %standard_library_default and returns both the effective value and also the directory containing the executable (which is used to implement Config.bindir for ocamlmklib). Note, in passing, that ocamlmklib on Windows now searches for tools in the same way as on Unix, as there's no need for the PATH-search previously there. The C implementation itself is in caml_locate_standard_library which is implemented separately for Unix and Windows in runtime/unix.c and runtime/win32.c. The tools used to implement this (dirname, realpath, etc.) are not equivalently available on Windows, by which I mean that the functions don't have exactly equivalent semantics. In particular, an exact dirname is not available on Windows, but GetFullPathName has the required semantics for this specific use and actually has an option to return the dirname (albeit indirectly). On this occasion, it therefore seemed easier to make the entire "work out the location of the Standard Library" operation platform-specific, than to go to more effort to construct platform-specific building blocks for a generic version of this function.
• The CI matrix for pull requests is altered to test a mix of absolute/relative builds by default. An additional test is added to deal with the "cross-runtime" example the Technical background above. Specifically, after building the compiler, it is duplicated to a new location, and a check is added to ensure ocamlc.byte -where produces the expected result regardless of whether the original or copied ocamlrun is used. If the CI: Full matrix label of #14013 is added, then each of the jobs builds an additional compiler with the alternate configuration (i.e. the jobs which by default are --without-relative-libdir then have an additional --with-relative-libdir build created, and vice versa). These two compilers are likewise used for the "cross-runtime" ocamlc.byte -where test.
• Finally, there's a little bit of additional plumbing added to go from "Relocatable" to "Reproducible" by utilising BUILD_PATH_PREFIX_MAP and directly adding -ffile-prefix-map to our internal C flags. Ignoring #! lines and RNTM sections (which are addressed in the final PR), when building with --with-relative-libdir, none of the binaries in the build contain either the build path or the installation prefix on Windows (with mingw-w64) or Linux.

Footnotes

1. Note that prior to#11996 5.3.0, native code executables which linked dynlink.cmxa also contained the location of OCaml Standard Library through the copy of the Config module in the Dynlink_compilerlibs library. ↩

[MisterDA]

Suggested change

AC_CHECK_HEADERS([unistd.h],[AC_DEFINE([HAS_UNISTD])])

This isn't needed.

[dra27]

This is part of a much worse merge/rebase artefact!

[MisterDA]

Suggested change

	@since 5.1 *)
	@since 5.3 *)

[dra27]

Low down the list to fix for now - but it certainly won’t be 5.3!! 🤣

[MisterDA]

Suggested change

	@since 5.1 *)
	@since 5.3 *)

[MisterDA]

Suggested change

	@since 5.1 *)
	@since 5.3 *)

[dra27]

Thanks for looking at this, of course, but these PRs are for my own purposes and aren’t ready for review