Record "weak dependencies" with -trans-mod

[lpw25]

Currently, -trans-mod simply ignores dependencies on unused module
aliases. This commit changes this behaviour so that unused module aliases are
included in the list of imported interfaces but without a CRC.

This change means that the imported interfaces list (as displayed by
ocamlobjinfo) is now an accurate representation of what was used during the
compilation of a module. Whilst the contents of unused module aliases is not
used, their existence is still required in order for the module to compile
successfully.

Previously, a simple file foo.ml containing an unused module alias to Bar
would have the following output from ocamlobjinfo for foo.cmo:

File foo.cmo
Unit name: Foo
Interfaces imported:
        4ad29aa1be509426919169d97aad0a82        Pervasives
        df1763e3e7e64b9b8ebea6f93b0a95b3        Foo
Uses unsafe features: no
Force link: no

with this patch it instead has output:

File foo.cmo
Unit name: Foo
Interfaces imported:
        --------------------------------        Bar
        6cda9672639792333f53de8e8ff9e71d        Pervasives
        df1763e3e7e64b9b8ebea6f93b0a95b3        Foo
Uses unsafe features: no
Force link: no

This is useful because now we can see that if bar.cmi is deleted then this
module will no longer compile successfully.

This commit achieves this change by making the digests optional in every list
of imported interfaces and implementations.

In the case of implementations this replaces the previous use of the
cmx_not_found_crc dummy value for implementations where a .cmx was not
available.

ocamlobjinfo and read_cmt print imports with an empty digest as a line of
dashes. All other uses of the imported interface lists ignore imports with an
empty digest, leaving their behaviour unchanged.

This commit also adds a (new) description of -trans-mod to the man pages.

[lpw25]

Note that I did not bump the magic numbers as I'm still not clear what the policy is on those.

[gasche]

minor nitpick, but I'd find the code more readable with

if `crco <> None && crco <> Some old_crc then
  raise ...;
match state with ...

[gasche]

The magic numbers need to change as soon as the format of compiled files changes (would a .cm*-consumer written against the previous interface work unchanged when passed files produced by the new implementation?). If I understand correctly, the serialized values have Some foo in stead of foo, so the magic numbers need to change.

I find the idea of making the dependency on the interface in the .cmi very good, but I'm not sure I understand why implementations are impacted as well. I understood .cmx files to be about cross-module inlining or optimizations, does -trans-mod actually change that in any way? I would have guessed that code inlined from a module-aliased dependency still need to recompiled if the dependency's implementation changes (while -trans-mod is about interface changes). Or is it that you thought the "optional CRC" could bring better structure to the .cmx-handling as well? What is the overlap with the -nocmx PR?

[lpw25]

The magic numbers need to change as soon as the format of compiled files changes (would a .cm*-consumer written against the previous interface work unchanged when passed files produced by the new implementation?)

I guess the thing I don't quite understand is that whenever the stuff in types.ml changes the .cmi files will have a different format, but that doesn't seem to result in a bump to the magic number. Anyway, I'll add a commit to bump the affected magic numbers.

Or is it that you thought the "optional CRC" could bring better structure to the .cmx-handling as well? What is the overlap with the -nocmx PR?

The change to the implementation imports lists is indeed unrelated to -trans-mod, but it does seem to be a better representation than what is used currently.

I considered making that a separate patch, but since the code changes are almost identical to those required for the interfaces lists I thought it would be easier to review as a single patch. Both changes also depend on the same change to Consistbl.extract.

A None value in the implementations list represents an implementation for which a .cmx file was not found. IIUC the -nocmx PR still creates a .cmx so I don't think the two are related.

[lpw25]

I've attached a commit that bumps the magic numbers for cmi, cmo, cma, cmx, cmxa, cmxs and cmt. I'm not certain the change to the cmxs magic number is needed. The dynunit type has changed, but it appears that this is stored as a symbol (caml_plugin_header) rather than saved directly into the file.

[lpw25]

Are any more changes required before this can be merged? I would like to see it in 4.02 as it is part of our solution to the "pack" problem (along with module aliases and the -prefix argument which Jeremie will hopefully add soon).

[dbuenzli]

@lpw25 is there any documentation about all that ? It may influence the design of a library I'll work on in a few weeks. If not, could you maybe make a blog post on what all this will mean for the future.

[lpw25]

@dbuenzli I was planning on doing a blog post once all the pieces were on trunk, so that I could show simple examples that would compile. Ideally that would include support from ocamlbuild, but I think that is too difficult to do before 4.02, so I'll probably do it once this patch and -prefix are merged.

[dbuenzli]

Excellent ! I'm especially interested in the "damned this will link in everything and produce huge executables so I have to pollute the toplevel module name space" problem.

[mshinwell]

We've been testing this at Jane Street, and shown that it (plus the prefix stuff) does solve the executable size problem. I'm going to commit this to trunk shortly.

[mshinwell]

Committed to trunk (rev 14719).
Leo, please close this.