Construct the bytecode executable header in ocamlc

[dra27]

This PR is a prerequisite for the "Relocatable Compiler" project, but the changes here are independent of it. It addresses five currently known bugs (see below) in #! ("shebang") handling in the bytecode compiler and its implementation considerably simplifies stdlib/Makefile, in advance of @shindere merging that into the root Makefile.

Background: Bytecode Executables

When not using one of its C-based linking modes (-custom, -output-complete-exe, etc.), ocamlc creates bytecode executables by prepending a launch header to the bytecode image. This header's sole responsibility is to locate the actual OCaml runtime and transfer execution to it. There are three ways in which this can be done:

1. A #! (or "shebang") header is used with the full path to the runtime, e.g. #!/usr/local/bin/ocamlrun. This is the default on Unix systems (except Cygwin, at least before this PR).
2. A small shell script is used to exec the runtime. Presently this is only used with -use-runtime when the path given is too long or contains a space, for example:

#!/bin/sh
exec '/home/David Allsopp/OCaml/bin/ocamlrun' "$0" "$@"

3. A small C program (in stdlib/header{,nt}.c), which is able both to execute a runtime at an absolute location (i.e. /usr/local/bin/ocamlrun) or to search $PATH for a runtime (i.e. to search for ocamlrun in $PATH).

At present, the choice between shebang scripts (mechanisms 1 and 2) and executable (mechanism 3) is made at configure time ($(SHEBANGSCRIPTS) and $(LONG_SHEBANGS)), and the result is written by the build system to the file camlheader which is kept in the Standard Library directory. This file is either the compiled executable header or it is the full path to where ocamlrun will be installed.

The runtime variants (-runtime-variant d and -runtime-variant i) are supported by building multiple versions of this file, so there are in fact 3 of them: camlheader, camlheaderd and camlheaderi.

Finally, in order to support the -use-runtime option, a different file camlheader_ur is created. ocamlc copies this file and immediately starts the bytecode TOC recorder. It then writes the name of the runtime followed by a newline and then marks the RNTM section.

Now, when shebang headers are supported, camlheader_ur is exactly the string #!. This means that ocamlc's procedure writes a shebang header, though pointlessly records it in the RNTM section. When mechanism 3 (the small C program) is in use, camlheader_ur is exactly the same as camlheader. The C program, in addition to knowing how to search $PATH, is also able to read the RNTM section of the bytecode image. It reads this data, cunningly converts the newline character which ocamlc wrote into a nul character, in order to make the RNTM payload a valid C string, and then proceeds to execute that runtime.

For -use-runtime only, ocamlc performs some validation on the runtime path to check if it's valid to use in a shebang line. If it's not, then it elects to write a mechanism 2 header (using /bin/sh).

That gets us to OCaml 4.02.1. In OCaml 4.02.2, in order to assist the iOS and Android cross-compilation projects, an additional set of headers was added: target_camlheader, target_camlheaderd and target_camlheaderi. These are the same as their unprefixed counterparts, except that the directory written to them is $(TARGET_BINDIR) (which can be overridden when calling make) instead of $(BINDIR). There is no target_camlheader_ur because there are no paths embedded in the file (so it never differs).

The 🐛s

As announced above, the five following bugs are present in this mechanism in OCaml. In decreasing order of severity, they are:

1. Not all valid paths are valid to use in a shebang line. If OCaml is configure'd with a bindir which is not valid in a shebang line, at present the bytecode tools installed correctly use mechanism 2. However, the target_camlheader* files are incorrectly generated, and bytecode executables produced by the installed ocamlc will have invalid shebang lines. (this is a very real bug, originally identified by the Sandmark project; see also Add -with-runtime/-without-runtime options #2309 (comment) and compiling ocaml in a folder with space in its name creates a binary with an invalid shebang #12709)
2. On Windows, passing an argument to -use-runtime which is longer than 125 characters causes ocamlc to generate a corrupt executable since it uses a #!/bin/sh header.
3. The check for whether a path is valid for a shebang line is duplicated between configure.ac and bytecomp/bytelink.ml, these have subsequently diverged and are (still) both incorrect. In particular, configure.ac only checks the length (and, even then, in a conservative check which rejects one otherwise valid possible header) and while bytecomp/bytelink.ml checks for space characters, both places fail to check for tabs and newline characters, which are also not permitted in a shebang line. (this issue has been separately reported in Avoid shebang fallback on 128 on linux #10724, and is also one reason that Windows CI actually includes a stronger test of strange characters in --prefix than the Linux/macOS one)
4. Both stdlib/Makefile (generating the headers) and bytecomp/bytelink.ml (processing -use-runtime) assume sh resides in /bin/sh, which is not guaranteed by POSIX (and, indeed, is not the case on some, admittedly obscure, systems)
5. When camlheader_ur is just the string #!, bytecomp/bytelink.ml still (unnecessarily) records the RNTM section. This in itself isn't a bug, but when writing a #!/bin/sh script version, the RNTM section incorrectly contains the entire /bin/sh script, rather than just the name of the runtime.

Current implementation

The current implementation of all this goes to some lengths to ensure that it is enough for bytecomp/bytelink.ml to copy the header blindly and never actually have to inspect its content. Regardless, the header is a relatively subtle piece of configuration state. The compiler and most of the tools will be compiled with boot/ocamlc which is built with a generic Config module. With the current setup of the build, therefore, it is not possible for boot/ocamlc to use values from Config (although Config.bindir exists, boot/ocamlc will see the value it was built with during the bootstrap, not the value used during configuration and, for this reason, there is no Config.shebangscripts to mirror the $(SHEBANGSCRIPTS) variable in the build system). Although ocamlc doesn't at present actually analyse the content of the header it's copying, the decisions it takes as to which file to read (based on -use-runtime and -runtime-variant) mean that the header is effectively acting as a series of "ghost" command-line arguments! While this is sort of neat, it's causing a few problems:

• In -use-runtime when camlheader_ur is #!, ocamlc ends up writing the full path of the runtime twice (even allowing for bug 5)
• In -use-runtime mode, ocamlc only needs to mark the RNTM section for the executable header, but it's unnecessarily marking it even when camlheader_ur is just #! (which ocamlc has in fact read!)
• Even more nefariously, even when the RNTM section is needed, the string ends with the wrong terminator in order to keep the format valid for the shebang case (i.e. the RNTM section is written unnecessarily in one case and, in order to ensure that the string is correct in that unnecessary case, the string has to be mangled in the necessary case 🤯)
• For largely historical reasons, the Windows ports prefer to search to %PATH% for the runtime, and never use an absolute path. This is very subtly encoded in stdlib/Makefile.
• Storing the "state" of the shebangs in camlheader et al means that the code for validating shebang lines is at present implemented in m4sh (in configure.ac), in OCaml (in bytecomp/bytelink.ml) and should be being implemented in GNU make (in stdlib/Makefile).

Proposed implementation in this PR

Presently, the processing of the header in ocamlc is simple, because it boils down to copying the correct file. I think it's possible to fix these 5 bugs while maintaining that. However, the code (in GNU make and m4sh) won't be terribly tasteful and the various checks will still be duplicated in several places. It is not possible to do Relocatable's switch cloning this way, where camlheader instead of being #!/usr/local/bin/ocamlrun wants to be something akin to #!../../bin/ocamlrun with the ../ interpreted relative to the header itself.

So, at last, to the details.

The principle here is to allow ocamlc to do all of the work, being given only the information which it can't know in advance via the "header". Since the header is now really a data-file, it's called runtime-launch-info. It contains the following three pieces of information:

• Whether shebang headers may be used, and if they can be used, how to specify sh in a shebang
• The full directory containing ocamlrun and its variants (this is usually $prefix/bin)
• The content of the executable header (compiled from stdlib/header.c, or stdlib/headernt.c on Windows)

ocamlc is responsible for:

• Constructing the full path to the runtime (based on -runtime-variant and the bindir read from runtime-launch-info)
• Finding sh, if runtime-launch-info doesn't contain an absolute path to it
• Writing either a valid shebang header or ultimately falling back to an executable header if this is not possible
• As a result, the RNTM section is used only with the executable header (and is now null-terminated). Furthermore, when using executable headers, it is required that the RNTM section is present in the image (this stops the executable header from ever containing an absolute path).

It makes sense while overhauling all this to implement (finally) the --with-target-bindir option, which was added in the switch to autoconf in 4.08 but never plumbed in. Previously, cross-compilation systems will have specified TARGET_BINDIR to make instead. Additionally, a new switch --with-target-sh has been added to complete the cross-compilation picture, at least in terms of the shell scripting. This allows every aspect of stdlib/target_runtime-launch-info to be controlled in the build. In particular, it allows an improvement to Cygwin's compilation (see below), also providing an immediate upstream use-case for this change.

While the implementation necessarily adds quite a lot of code to bytecomp/bytelink.ml, it removes a relatively complex bit of m4sh from configure.ac and an exceedingly complex mess from stdlib/Makefile. In passing, there are a couple of related issues which can be trivially fixed:

• There is a highly unlikely corner-case where sh may not be found. This is "solved" by always compiling the executable launcher, even on Unix. A minor side-effect of this is to reduce bit-rot in this file, which had started to happen (see first commit).
• The implementation of --with-target-sh allows the use of shebangs on Cygwin to be improved.

The approach I've adopted in this PR is to allow ocamlc to look at the data it reads from camlheader and act accordingly. There is more than one way to do this! I think it is possible to achieve this using a mix of -use-runtime in the build system and carefully ensuring that the Config module's values are only ever used by a compiler which has been actually installed. Likewise, we have previously discussed being able to dynamically load the complete Config module into the boot compiler (see #9291). I think the -use-runtime approach is likely to be a bit too brittle and although I have a possible approach for loading Config at runtime, it's not a trivial change, and we are also fixing bugs here and now.

[MisterDA]

We don't want with_open_bin here?

[dra27]

We're reading the output of a command, so morally it's text (not that this path is likely ever hit by a Windows build of the compiler, even with cross-compilation)

[xavierleroy]

Thank you for revisiting this old pile of hacks, and for the detailed write-up!

I read the diff and understand most of it. See below for a suggestion for more comments.

The one part I'm not sure about is the late determination of the path to sh by ocamlc. Hard-coding /bin/sh would work on 99.99% of Unix systems. But even if you prefer not to hard-code, why not determine the path at configure-time?

[xavierleroy]

I'm not sure why the shell location needs to be determined so late (when ocamlc is generating a bytecode executable), as opposed to being determined by the configure script and stored in runtime-launch-info as /path/to/bin/sh.

[xavierleroy]

Nice use of command -p -v, but again I was expecting this to take place at configure-time.

[xavierleroy]

You lost me :-) The printf I understand, but not the sed-scripting. A short comment would help...

[dra27]

Ouch, yes - doubly confusing by the use of & instead of ' which I think came from an earlier version. It's our old friend the escaped quotation mark. I added some comments. Overall it means that ./configure --with-target-bindir="/home/'quoted'/bin" actually makes it to the substitution files correctly!

[shindere]

Many thanks for having gone thorugh the description and the code Xavier!

[dra27]

Thanks, @xavierleroy! For the location of /bin/sh, I took a while to make up my mind, but what I eventually settled on was that we're working down a series of options to ensure that ocamlc definitely produces a valid executable - starting off with a direct shebang to ocamlrun, falling back to a /bin/sh script otherwise but also having the executable header as the final "this will definitely work" option. My reasoning both for a dynamic check at all and also for checking for sh in ocamlc rather than configure was just about swayed by a combination of:

• Posix requiring it to be searched for (it's quite explicit, even in Issue 7!)
• If configure determined a path, I'd still want ocamlc to check that that it actually exists at link time
• There's not a vast amount of difference at that point between Sys.file_exists on a path we read and command -p -v sh for one we searched (very very very pedantically command -p -v sh also confirms that sh is executable)
• It only happens at link time: in the direct shebang case there's also a call to ocamlrun -p (which confirms that the runtime both exists and is executable as well).

I don't mind changing it, although I'd be happy if that bit of over-thinking is humourable 😁

[shindere]

The merge of #12843 has created a conflict here that now needs to be
resolved, in stdlib/Makefile but that should be an easy one.

[OlivierNicole]

I understand the code and it looks reasonable to me.

There is only one thing that I do not fully understand from the code and your (otherwise very clear) description, namely which runtime-launch-info is read by boot/ocamlc (IIUC, stdlib/runtime-launch-info?).

[OlivierNicole]

This file doesn’t seem to ever be generated. Or am I missing something?

[dra27]

The LIBFILES variable in the root Makefile includes runtime-launch-info (as $(HEADER_NAME)) and, along with stdlib.cma, std_exit.cmo and all the .cmi files, that gets copied to boot/ from stdlib/ in the coldstart target.

[OlivierNicole]

Like in .gitignore, boot/$(HEADER_NAME) should be removed here if it is never generated.

[dra27]

Thanks, @OlivierNicole! It is indeed stdlib/runtime-launch-info which gets promoted to boot/runtime-launch-info in coldstart (which exactly replaces what happens with stdlib/camlheader at the moment).

The fact that's not been clear to you says there should be more comments, but I'm not immediately clear what to put where. It does hint at a mild inconsistency that the camlheader files - which are morally part of the bytecode linker - are built in stdlib/ at all, although from a technical perspective it makes sense inasmuch as stdlib and runtime have to be processed as part of coldstart before the bytecode compiler itself is built.

[shindere]

David Allsopp (2024/01/18 15:42 -0800):

Thanks, @olivernicole! It is indeed `stdlib/runtime-launch-info` which gets promoted to `boot/runtime-launch-info` in `coldstart` (which exactly replaces what happens with `stdlib/camlheader` at the moment). The fact that's not been clear to you says there should be more comments, but I'm not immediately clear what to put where. It does hint at a mild inconsistency that the `camlheader` files - which are morally part of the bytecode linker - are built in stdlib/ at all, although from a technical perspective it makes sense inasmuch as `stdlib` and `runtime` have to be processed as part of `coldstart` _before_ the bytecode compiler itself is built.

I believe we _did_ discuss moving the header files to `runtime/` rather than leaving them in `stdlib/`, didn't we? I just can't remember why we endedup not doing it. Was it because we wanted to keep that for later / keep changes separate?

[dra27]

I'm not sure - moving those files is indeed related to #11198, and it's probably better to move the file as part of doing that (i.e. move where it's built at the same time as moving where it's installed).

[shindere]

David Allsopp (2024/01/19 02:54 -0800):

I'm not sure - moving those files is indeed related to #11198, and it's probably better to move the file as part of doing that (i.e. move where it's built at the same time as moving where it's installed).

Ah, I see. You mean that, install-wise, stdlib is a better choice (we do'nt even have a reasonable alternative, do we?)?

[dra27]

#11198 got the otherlibs into subdirectories of /usr/local/lib/ocaml ... the next stage would to have /usr/local/lib/ocaml/stdlib with just the standard library and /usr/local/lib/ocaml/runtime containing all the various libasmrun and libcamlrun files and camlheader/runtime-launch-info (not at present planned - it's useful, but not as useful as shifting unix, dynlink, etc. to subdirectories was)

[shindere]

David Allsopp (2024/01/19 05:45 -0800):

#11198 got the otherlibs into subdirectories of `/usr/local/lib/ocaml` #... the next stage would to have `/usr/local/lib/ocaml/stdlib` with #_just_ the standard library and `/usr/local/lib/ocaml/runtime` #containing all the various libasmrun and libcamlrun files _and_ #camlheader/runtime-launch-info (not at present planned - it's useful, #but not as useful as shifting unix, dynlink, etc. to subdirectories #was)

This target definitely sounds nice and pleasing to me!

[lthls]

#11198 got the otherlibs into subdirectories of /usr/local/lib/ocaml ... the next stage would to have /usr/local/lib/ocaml/stdlib with just the standard library and /usr/local/lib/ocaml/runtime containing all the various libasmrun and libcamlrun files and camlheader/runtime-launch-info (not at present planned - it's useful, but not as useful as shifting unix, dynlink, etc. to subdirectories was)

This sounds nice, but keep in mind that while testing the compiler, we often use -nostdlib -I stdlib and that magically finds the runtime and the header files. It's not a huge pain to have to also add -I runtime, but it's not a huge gain either.

[shindere]

Vincent Laviron (2024/01/19 06:11 -0800):

This sounds nice, but keep in mind that while testing the compiler, we often use `-nostdlib -I stdlib` and that magically finds the runtime and the header files. It's not a huge pain to have to also add `-I runtime`, but it's not a huge gain either.

That seems indeed very useful to keep in mind! Althogh I'm wondering whether it does make sense to use the same option, -I, for both OCaml files and other library files. Perhaps this is an area where relocatable willhelp because then the path to the runtime files will be relative to the path of the compiler and will thus no longer need to be specified?

[OlivierNicole]

I want to approve this PR, I just have one last, small incomprehension: shouldn’t the bootstrap commit add a new file boot/runtime-launch-info?

[OlivierNicole]

You were mentioning the possibility to add a brief comment to explain the role of runtime-launch-info and how it is bootstrapped. I think here would be a good place to put it.

[dra27]

Aha - yes! I have pushed some extra text there.

I want to approve this PR, I just have one last, small incomprehension: shouldn’t the bootstrap commit add a new file boot/runtime-launch-info?

I've clarified that in the comment, but no - boot/runtime-launch-info is the part which can't be committed (the platform- and configuration-specific information).

[OlivierNicole]

Thanks for the clarification.

[damiendoligez]

Looks good modulo one small suggestion.

[damiendoligez]

Wouldn't it be easier to read if you just tested kind?

Suggested change

	else if bindir_start > 1 && (kind.[0] = '/' || kind = "sh") then
	else if kind <> "" && (kind.[0] = '/' || kind = "sh") then

[dra27]

Yes, indeed! Changed, thanks.

[Octachron]

I fear that this comment might become confusing in the future, it seems better to document the current state rather than the history of this Makefile.

[dra27]

Ah, the idea was meant to be a reminder that if/when SHEBANGSCRIPTS is removed, it would also be possible to remove shebangscripts from configure.ac, but I see what you mean. I think it best therefore to remove the comment entirely and trust that whoever removes SHEBANGSCRIPTS to consider re-checking for any other uses of shebangscripts (and so remove it, if it is truly unused at that point)

[OlivierNicole]

Looks good to me as well.

[dra27]

Thank you, everyone... here goes...