Provide the option to bootstrap FlexDLL with OCaml

[dra27]

Preamble

This pull request contains a submodule which at present refers to a commit which has not yet been merged into FlexDLL (see ocaml/flexdll#14). If you wish to experiment with the PR, you need to do one of two things:

1. Run git submodule update --init, ignore the error message and then from the flexdll directory run git remote add -f dra27 https://github.com/dra27/flexdll.git followed by git checkout visual-studio
2. Run git submodule init, edit .git/config and change the URL for the flexdll repository to mine and then run git submodule update

Obviously, notwithstanding other reasons, this PR should not be merged until ocaml/flexdll#14 is resolved.

Abstract (!)

This PR provides the option of bootstrapping FlexDLL from sources as part of the general build process for native Windows OCaml. At present, this support is not offered for Cygwin (see below).

The philosophical arguments for doing this have already been aired on at least one occasion, but there are two more concrete reasons for the desirability of being able to do this:

1. Ease of compilation from sources - having the option of compiling both together results in much simpler instructions for the end-user (especially a beginner).
2. OPAM. OPAM, at least at present, is intended to be a source distribution of OCaml - at present, the only binary which has to be installed and managed is Cygwin. FlexDLL sits in an odd place, given that it must be awkwardly installed before the compiler in a switch - OPAM therefore benefits both philosophically and literally from being able to install OCaml and FlexDLL as one.

Cygwin

Cygwin OCaml uses FlexDLL to enable shared libraries - some of the changes I have made are in fact porting existing functionality in the Unix build system to allow Cygwin flexlink to be bootstrapped, because there is/was no binary distribution. I have not extended this full bootstrapping support to Cygwin yet for two reasons:

1. It is more effort (and, as it happens, for a platform I don't care about, but that's by the by) and involves (more) duplication of existing code between the Unix and Win32 code branches.
2. Cygwin is built using the configure rather than Makefile-editing route.

At some point in the coming months, if nobody else gets there first, I shall contribute a unified build system for the native Windows ports via configure and I would like to propose that full bootstrapping for Cygwin of FlexDLL be left until that work is done (by whoever).

Commentary

Overall, it is pleasing (to me, at least) how few changes are required to facilitate this - which I hope may aid in its acceptance.

The eight commits include an earlier partial effort (09feadf and 29f7d7d). This simpler approach allowed the user to say

make -f Makefile.nt flexdll install-flexdll

which allowed a byte-code flexlink to be compiled and pre-installed. However, it still assumes that the user has set their PATH appropriately so that that flexlink can then be used in the main build process for OCaml.

In that earlier work, the flexdll sources have to downloaded (either by source tarball or by git clone) and placed into the OCaml source tree prior to issuing make -f Makefile.nt flexdll. However, this is far from the best way for Git - both the fact that OCaml and FlexDLL are truly separate projects and the fact that it remains possible to compile OCaml without also compiling FlexDLL makes the repository a perfect candidate for a Git submodule which is what is then added in d8a35be. Error messages in Makefile.nt are added which alert the user to the absence of flexlink and advise how to fix the problem (this is part of the reason for not incorporating this to Cygwin at the moment, as such testing really should be in configure).

In order to allow the OCaml build procedure to use a locally bootstrapped copy of FlexDLL, it is necessary to add one feature to the compilers and to ocamlmklib - there needs to be a way to override the flexlink command (or, more specifically, Config.mkexe, Config.mkdll and Config.mkmaindll). The only alternative I could see would be to build two copies of the compiler and ocamlmklib which I don't think is an acceptable burden to add to the build process - both in terms of time and complexity of understanding. My solution instead in f4fd8b1 is to split the FLEXLINK Makefile variable into known constituent parts and then to allow it to be overridden at runtime by an OCAML_FLEXLINK environment variable (i.e. the way in which mkexe, mkdll and mkmaindll is less general than it was before).

Finally, 9b62404 utilises this new feature to allow a bootstrapped FlexDLL to be used if flexlink is not found in PATH. It also amends the opt.opt target to build a native code version of flexlink.exe only if FlexDLL was bootstrapped in the first place and the installbyt target also installs flexlink.exe and the required FlexDLL object files and manifests again, only if FlexDLL was bootstrapped in the first place. The change to the build process principally involves communicating either a revised OCAMLOPT variable or a OCAML_FLEXLINK=... line to be used, similar to the overriding already in various places to specify the boot compiler. At present, the expressions used are more verbose than would be ideal (especially the need to translate forward slashes to backslashes), however this would be vastly simplified by the use of a configure script for the native Windows ports, so I would crave your patience with the very long $(if $(shell ... )) lines!

This has been tested with all four native Windows ports (running make -f Makefile.nt [flexdll] world bootstrap opt opt.opt install) both with and without the submodule initialised and in all eight builds the test-suite (utilising various patches pending in #366 and #384) runs with 100% pass rate.

I haven't updated README.win32.adoc with instructions because I can't be bothered to update a guide which I'm already planning on completely rewriting once I've finished the Visual Studio testing which has been yielding all these patches...

Thank you for getting to the bottom 😄

[dra27]

Please note that Travis is failing because of the submodule (which yields the question of why the CI is doing a submodule init, given that there were no other submodules)

[dra27]

Travis build fixed

[alainfrisch]

An alternative approach to break the dependency cycle between ocaml and flexlink would be to only simplify building OCaml without flexlink (I think @damiendoligez does it regularly with the cygwin port), and add to the flexdll build system the required stuff to compile a local OCaml only to build flexlink itself.

This would reduce the impact on OCaml build system. And it would allow building flexlink more naturally with a different port than the one you want to use for OCaml itself; for instance, to create a 64-bit version of flexlink to allow it to work on larger bigger object files, even if you want to use a 32-bit port of OCaml; or, to create a version of flexlink compiled with the Cygwin port of OCaml so that it does not need to call the external cygpath command (which would probably give a significant speedup).

[dra27]

Or we could do both 😄

Indeed, copying over the required support for the -no-shared-libs option for bootstrapping Cygwin flexlink is in the first commit and it would only then require moving parts of the second commit to FlexDLL for that. Using a 64-bit flexlink with a 32-bit OCaml is an interesting config - I've added a 'flexlink-x64' package thought to my OPAM todo list!

Using, for example, a Cygwin flexlink with a native Windows port falls neatly into work I'm planning for later this year with Windows cross-compiling anyhow - again, for OPAM, I'd like to be able to produce:

• Compilers as Cygwin programs emitting MSVC/mingw (solving the symlinks and fs differences and sitting very well for users who use Cygwin bash as their environment) and vice versa (purely for completeness)
• 32-bit compilers emitting 64-bit programs (and possibly vice versa, though that's totally unnecessary given WoW)

(which also gives the side-goal of accessing Microsoft's ARM compilers, so I can get OCaml programs running on Windows 10 IoT!)

In all that, I'd find it natural and preferable still to have OCaml's build process able to bootstrap its required FlexDLL, but there is of course no bootstrapping argument for that - it's just a (big) convenience.

[alainfrisch]

copying over the required support for the -no-shared-libs option for bootstrapping Cygwin flexlink is in the first commit

I've pushed the first two commits to trunk. Let me think more about the rest!

[dra27]

I've just revised things slightly - in running my own testing, I hit a build-tree usability issue where I'd re-built a rebuilt bytecode flexlink but because ocamlopt.opt was still floating around the testsuite would attempt to invoke flexlink directly (i.e. without using ocamlrun) and so not work
My revision changes the way I do opt.opt to build flexdll/flexlink.opt (i.e. as opt.opt does for all the other tools) and leave flexdll/flexlink.exe as a bytecode executable. installopt copies flexdll/flexlink.opt as flexlink.exe if it exists (and so overwrites a bytecode executable which may have been installed).

[dra27]

I'm unsurprisingly eager for this to be considered for 4.03.0 - have you had time to think further on it?

[alainfrisch]

Is this change related to this PR?

[dra27]

Yes - because of the addition in tools/Makefile.nt because I need an extra line at the end of ocamlmklibconfig.ml for Windows only. You can't mix : and :: rules. This has the intentional effect of executing the instructions in Makefile.shared followed by the appending to that file in Makefile.nt (the clean targets in many of the Makefiles are constructed in the same way).

[alainfrisch]

I'd prefer to minimize the diff between Makefile and Makefile.nt. As long as common parts are not shared, it is useful to reduce the diff to help spot problems; and to allow sharing, it's good to avoid invasive changes such as the addition of REQUIRE_FLEXDLL as a dependency to many rules.

I don't think that all rules are supposed to be usable in arbitrary order. What do you think of including this requirement only to more specific places. Or, alternatively, to add a more pre-compilation requirement target (with a more generic name) also to Unix's Makefile (which would be empty there)?

[dra27]

I'd happily go along with removing the REQUIRE_FLEXDLL in most places and simply leave it on a ToDo list for when the native Windows ports work via configure (as really the whole thing should be selected with --with-flexdll allowing you to force it even when there's a system flexlink available)

[alainfrisch]

I'd happily go along with removing the REQUIRE_FLEXDLL in most places

I'm in favor of everything which still suits your needs while making the diff as small as possible and reducing divergence between Windows and non-Windows Makefiles.

[dra27]

Ok - I'll have a go at reducing the diff as far as possible. Thanks for reviewing it!

[dra27]

OK, here we go! Rebased on to this morning's trunk. I have:

• Removed the REQUIRE_FLEXDLL and FLEXDLL_SRC stuff completely
• As part of that, altered the semantics of having flexdll present slightly. If you initialise the submodule (which is detected by seeing if flexdll/Makefile exists) then the various Makefile's will always use flexdll/flexlink and will assume that its a bytecode image. If you haven't run make -f Makefile.nt flexdll, then later stages will produce an error. I could make it that make -f Makefile.nt world automatically does make -f Makefile.nt flexdll if flexdll/Makefile exists but I haven't done this just in case somebody initialises the git submodule without realising what they're doing (e.g. because they have set-up a git alias which automatically initialises submodules)
• Shifted the conversion from / to \ for OCAML_FLEXLINK to the Config module
• It is possible to change lex/Makefile.nt, etc. so that the extra settings are all made in the root Makefile.nt, but this has the downside of meaning that if you were working on ocamllex (or, more importantly, say, win32unix!) that you can't invoke make within the directory itself because OCAML_FLEXLINK won't be set, so I think the present placement is better. Granted, it would be better if all those Makefile's referred to a common shared root with all this stuff, but that's not really part of this PR!
• I have included one extra commit ae49015 for consideration - the forward-to-backslash stuff in b46843f results in OCaml code duplication between tools/ocamlmklibconfig.ml and tools/config.ml. This commit eliminates this duplication by linking ocamlmklib with config.cmo. If you approve of this change, I would rebase to swap these two commits around to create a more logical history (since at present the latter commit deletes the code duplication, where having them the other way would never introduce it).

I don't think I can sensibly make the diff too much smaller - what do you think of the revised version?

[alainfrisch]

(a reminder that this can only be merged after ocaml/flexdll#14 has been merged

Done!

[dra27]

Cool - I'm just testing a rebase onto current trunk so that Travis can comment on this...

[dra27]

OK - compilation tested. Native Windows trunk is broken at the moment (PR coming soon...), but this builds. I shall shortly verify that the testsuite is still at full pass, but I have to rebase another branch first (too many open PRs at the moment!)

[dra27]

And that's a full testsuite pass for MSVC x86 and x64

[alainfrisch]

Passing $(BOOT_FLEXLINK_CMD) to various calls to $(MAKEREC) increases the diff between Makefile and Makefile.nt. What about introducing a variant of MAKEREC (MAKERECBOOT?) for those calls, also for the Unix version (defined as MAKEREC there)?

[alainfrisch]

Ok, I've fixed the conflict and push the current version to trunk. I'm still interested in keeping the diff between Makefile and Makefile.nt under control as a preparation for more sharing between the two, but this is certainly not blocking.

[dra27]

Thank you - you've made my Monday 😄 I'll have a go at reducing that diff as you suggest shortly - there will also be a small update wanted to shift the SHA apropos of the two PRs on FlexDLL this morning.

[dra27]

Just looking at this - is defining MAKERECBOOT necessarily a good idea at this stage? Makefile.nt already has differences here since Makefile uses $(MAKE) and Makefile.nt has to use $(MAKEREC) - does having $(MAKERECBOOT) run a risk of looking too similar for being missed when they're merged later?

[damiendoligez]

OK, I can't say I understand everything here, but I have a warning:

There is an ocaml package in cygwin, which I'm maintaining, and it uses the flexdll package, also in cygwin, which somebody else is maintaining, and which is different from the upstream flexdll in ways that I don't understand. All I know is, if I build the ocaml cygwin package with the regular flexdll (instead of the flexdll provided by cygwin) some things will break.

I just want you guys to be aware of this workflow.

[dra27]

At present the submodule shouldn't interfere in any way - especially at the moment as Cygwin uses the Makefile, not Makefile.nt route.
And of course if the submodule isn't initialised (git submodule init; git submodule update) then nothing different should happen at all for any platform.