Support for multiple native code builds of the stdlib, compilerlibs and otherlibs

[mshinwell]

Compilation with gprof has always required a different build of the stdlib, where the .o files but not the .cmx files differ. This pull request implements an extensible way of doing the same thing in the presence of multiple such builds. The new method is used for the gprof build. The next stdlib variant is likely to be for Spacetime, where we wish to have a non-instrumented and an instrumented version of the stdlib present, both to support dynamic selection of using Spacetime and so that the compiler itself can be built without instrumentation. (In that case, a change to the cmx format will ensure that non-instrumented and instrumented code is not linked together in the same program.)

The non-default builds of the stdlib for native code are conducted in subdirectories of stdlib/ with the .cmx files (only) installed into subdirectories of the installation lib/ocaml/... directory.

(Joint work with @damiendoligez )

[mshinwell]

It looks like we need to make a fix for Windows, I will do it on Monday.

[mshinwell]

Erm, this feature had an accident and now aims to support multiple builds of the stdlib, compilerlibs and otherlibs. There is nearly enough support to make tricky things such as compilation with frame pointers selectable at runtime. This should also be a significant step along the road to full cross compilation. So far the building and installation part is probably just about done, but the runtime selection of the correct libraries probably needs a bit more work. The configure script work is not complete. I am aiming to have this feature-complete this week or early next.

The intention is to have a configure-time flag that specifies which features (e.g. gprof support, Spacetime, frame pointers etc)---or maybe better, combinations of features---are to be included.

Previously for Spacetime we were envisaging a configure-time switch and an undocumented compiler option to disable instrumentation, the rationale being that the compiler would not be instrumented but the code it generated would be. Unfortunately this would mean that compiler wouldn't be usable in conjunction with its own compiler-libs, as they would be built without instrumentation, and everything must either be built with or without. Hence the change in approach for this patch.

[mshinwell]

A couple of other comments:

• All the juicy things are in Makefile.
• This shows up warts in the build system quite nicely (by the presence of special cases).

[mshinwell]

To clarify, in the installation directory you now get subdirectories (like GCC multilibs) such as:

lib/ocaml/x86_64-unknown-linux-gnu/p/

(p being the name for the feature that builds with gprof support. Another feature name might be spacetime+with-frame-pointers.)

[mshinwell]

Another part needed which I am going to work on now: the notion of runtime variant needs to change. I think that is mixed up at the moment as well. It should probably be something like: normal runtime, debug runtime or instrumented runtime. Runtimes that are just compiled differently, e.g. with profiling (-pg), with frame pointers etc. should just follow whatever compiler options were provided. So for example -runtime-variant d -p would use a runtime that is both (a) the debug one and (b) instrumented for gprof.

[mshinwell]

This patch now provides compiler options to select no-naked-pointers, frame pointers, gprof instrumentation and PIC at runtime. There are still some things to fix (e.g. missing libasmrun_shared.so in the subdirectories) but it is looking promising.

[gasche]

Would it be possible to rebase to clarify the history a bit? I'm not sure how I would go about reviewing this in the current state.

[mshinwell]

@gasche : I've done this

[lefessan]

I am really concerned by the length of this patch, its complexity (a Makefile now contains the list of almost all files in the distribution, how is it expected to be maintained correctly ?), for a purpose that doesn't sound very important to me: why not use different OPAM switches to build different builds of OCaml, i.e. one switch for gprof, one for +spacetime, one for -spacetime, etc. The only thing that we would need is a way to pass configure options while creating a new switch in OPAM.

[mshinwell]

This patch is not actually that complicated, even though it may look frightening. I think a GNU make expert could probably shorten it further. There may also be ways of avoiding the extra lists of files. It's worth noting that if that list isn't correct, there will be an error. Also, we do actually have similar lists for the things involved in compiler-libs, so it's not like this is really novel.

I explained above why something like this patch is needed for situations such as use of Spacetime where you do not want the compiler to be instrumented. For cross-compilation scenarios I think you also need something like this unless you build cross compilers without self hosting. I tend to think the combined self-hosted approach is probably better, especially if it could be made modular so that different OPAM packages could be installed to provide the extra functionality. (So for example there might be a base 4.04 compiler package and then a set of add-ons providing e.g. gprof, PIC, Spacetime etc.)

I don't know how many people use OCaml without OPAM---there are certainly some---and in that environment, the multilib approach in this patch is potentially much more convenient than the existing situation.

[mshinwell]

I'm going to remove this as a Spacetime prerequisite since I don't think it's realistically going to be polished up in time for 4.04, and there may be some arguments still to have about it. We have two other options for Spacetime:

1. Put up with the fact, for 4.04, that the compiler is instrumented.
2. Use ocamlopt instead of ocamlopt.opt, which is not instrumented.

[mshinwell]

@shindere This is really for you to look at, when you have time to investigate cross compilation.

[shindere]

Mark Shinwell (2016/12/27 00:37 -0800):

@shindere This is really for you to look at, when you have time to investigate cross compilation.

@mshinwell yeah it's on my list, thanks.

[shindere]

Mark Shinwell (2016/12/27 00:38 -0800):

Assigned #620 to @shindere.

Thanks. Feel free to assign to me any configure, build system or cross-compilation related {G,M}PR.

[XVilka]

With the removal of gprof, maybe it makes sense to close this PR?

[damiendoligez]

With the removal of gprof, maybe it makes sense to close this PR?

I think it will still be useful for the cross-compilation work, so let's keep it around for the moment.

[mshinwell]

@shindere @dra27 I'm going to close this, but it will still be here for reference!