Uses -custom

[glondu]

Looking at doc/dune-files.rst, dune uses ocamlc -custom to generate binaries on bytecode architectures. -custom is deprecated, please provide a way to avoid it.

One problem is that resulting binaries are not strippable. I once submitted a patch to OCaml to fix that, but it has been rejected by @xavierleroy saying that -custom was deprecated and should not be used.

The patch has been ported to OCaml 4.05.0, but I'd rather not port it to 4.08.0 as diverging from upstream this way is painful.

[rgrinberg]

Additionally, we now use -custom to bootstrap dune itself. That should be avoided if it is indeed deprecated.

[ghost]

It's actually not trivial to avoid. -custom provides a very convenient way to produce self contained executables when native compilation is not available.

This is the first time I hear that -custom is deprecated BTW.

[ghost]

We had a discussion on caml-devel about this.

Currently, we use -custom in Dune has a fallback for executables when native compilation is not available. Doing this for programs that are run during the build seems fine, as it makes Dune's life much easier. For this reason, -custom will stay for now.

However, using -custom for binaries that are installed on the system is not ideal and we could indeed install the pure byte code programs instead.

Regarding the patch to allow custom program to be stripped, integrating it in OCaml could be an option. However, looking at it it looks a little bit specific to Debian. Would it be possible to have something that is more portable?

Finally, before doing anything, I would also like to push back a little; why is bad to have binaries that can't be stripped? For instance, a valid option would be to change the stripping tool to skip custom byte programs.

[glondu]

However, looking at it it looks a little bit specific to Debian

In spirit, there is nothing specific to Debian in the idea. The Debian-specific bits are about activating the effect of the patch only while building packages, keeping the former behaviour elsewhere. I wrote the patch in 2008. Now, I would apply the new behaviour unconditionally since it's so easy to install a vanilla OCaml thanks to OPAM, so we can afford to not care about people wanting the original behaviour.

Would it be possible to have something that is more portable?

The patch currently doesn't apply to 4.08.0 in a way that I'll have to rewrite it basically from scratch.

why is bad to have binaries that can't be stripped

The policy in Debian is to build everything with debugging symbols, so that users can provide useful reports in case of crashes without having to recompile packages. The Debian toolchain automatically strips binaries and puts debugging symbols in separate packages to keep the main package small. (This could also benefit to custom OCaml runtimes by the way.)

a valid option would be to change the stripping tool to skip custom byte programs

I don't consider as valid an option to workaround a broken behaviour. And it sounds wrong to bake OCaml-specific knowledge in a language-agnostic tool.

[ghost]

What is the broken behaviour?

[glondu]

What is the broken behaviour?

Generating binaries that cannot be stripped.

[ghost]

I had a look at this. It's actually more difficult than expected to solve this problem in general. For instance, an executable might depend on a private library that has stubs and we currently don't install the .so files for stubs of private libraries. Plus in the future we'll probably allow executables to have stubs as well.

I had a look at how -custom is implemented. IIUC, the process is as follow:

1. the compiler links a custom ocamlrun binary, that in particular statically links in all the C stubs
2. it produces a bytecode object file
3. it appends the bytecode object file at the end the the binary produced by step 1

At runtime, the startup code:

• tries to open argv[0] as a bytecode file
• if it fails, it tries to open /proc/self/exe as a bytecode file

This can lead to weird behaviors:

$ echo 'print_endline "custom"' > custom.ml
$ echo 'print_endline "normal"' > normal.ml
$ ocamlc -custom custom.ml -o custom
$ ocamlc normal.ml -o normal
$ ./custom
custom
$ ./normal
normal
$ ocamlrun custom
custom
$ ocamlrun normal
normal
$ ( exec -a ./custom ocamlrun normal )
custom
$ echo 'print_endline "custom2"' > custom2.ml
$ ocamlc -custom custom2.ml -o custom2
$ ( exec -a ./custom2 ./custom )
custom2

Oh dear, I understand why custom is considered a dirty hack...

I wonder if we couldn't change the implementation of custom to instead generate the same kind of C file -output-obj -o blah.c produces and link that directly in the custom runtime where we would generate a main function that would reference the generated caml_code array directly.

[glondu]

I wonder if we couldn't change the implementation of custom to instead generate the same kind of C file -output-obj -o blah.c produces and link that directly in the custom runtime where we would generate a main function that would reference the generated caml_code array directly.

This is exactly what my patch did.

[nojb]

This is exactly what my patch did.

@glondu could you post a link to the source of your patch ? Thanks!

[ghost]

@nojb I think it's the link in @glondu's first comment.

/cc @dra27 who might be interested in this discussion as well.

In the end, the compiler patch seems to me like the way forward. It'd would turn -custom into a nice and clean feature. On the other hand, making sure Dune doesn't install -custom executable feels like opening a can of worms.

[nojb]

See ocaml/ocaml#8872

[nojb]

I think this has been fixed in #2692