Tidy up installation-directory/lib/ocaml

[v-gb]

The way most libraries of the compiler distribution are installed right now is by putting all their files in $PREFIX/lib/ocaml, or $PREFIX/lib/ocaml/compiler-libs for the libraries of the compiler (the exceptions being threads, ocamldoc and before camlp4).

This is annoying because it's desirable for a build system to prevent a library foo from building against a library bar if foo doesn't declare a dependency on bar (by not putting bar in the search path). And this works great, except when bar is a compiler library like str or unix because they are in the search path and there's nothing the build system can do about it.

This results in link time errors instead of "unbound module" errors, both with the normal ocaml compilers and with js_of_ocaml, and potentially for a user of the bad library rather than an author of the bad library.

The same thing happens with compiler-libs, when declaring a dependency on ocamlcommon and using ocamltoplevel.

I propose solving this by installing all libraries in their own subdirectory of $PREFIX/lib/ocaml (except stdlib). For backwards compatibility, the search path contains +unix, +str etc by default, which can be turned off with a new flag -nostdincludes which I would use.

I didn't implement any compatibility for compiler-libs, as code that links with these libs tends to break on every new compiler version anyway. I made -I +compiler-libs expand to -I +ocamlcommon -I +ocamltoplevel .. for compatibility.

[mshinwell]

Why would the stdlib not be in its own directory too?

[bobot]

When this MR is accepted, could you propose an MR in ocamlfind in order to update site-lib-src/? So that the ocamlfind packages for compiler-libs work. The other libraries could also be updated for when -nostdincludes would become the default.

[v-gb]

Why would the stdlib not be in its own directory too?

No strong reason. I had to stop somewhere to see what people think about this, I don't know if some people might be accessing this directory other than through the search path and I had some trouble last I tried making such a change because it wasn't clear what to do with the files in lib/ocaml that were installed there but not part of the stdlib like topdirs.cmi. Though for that last problem, I ran into the same thing with compiler-libs and driver/{main,opt-main}.* being installed for some unclear reason, so I left these in compiler-libs. Presumably the same thing would work.

When this MR is accepted, could you propose an MR in ocamlfind in order to update site-lib-src/? So that the ocamlfind packages for compiler-libs work. The other libraries could also be updated for when -nostdincludes would become the default.

I'm not very familiar with ocamlfind, but I can take a stab at it.

[v-gb]

I rebased this, thought it conflicted a lot so I could have introduced problems.

Any taker for review (I recommend commit by commit)? This would still be a useful change.

[nojb]

I did a first pass over the code, which seems fine, modulo the few small comments. More generally:

• Commits 1, 2, 6, and 7 are cleanups/fixes which could be merged independently of this PR. I would suggest splitting them into a separate PR which would be merged quickly.

• stdlib should be installed in its own directory as well.

• This patch will break any tool that assumes the current layout of the ocamlc -where directory. Am not sure how significant the risk is here.

[nojb]

-nostdincludes -> -nostdinclude

[v-gb]

I'm guessing you simply want the commit introducing the flag to be consistent about the plural vs singular, rather than necessarily use the singular version? Given that we have multiple include directories, it seems slightly better to keep the plural.

[nojb]

"-nostdincludes" -> "-nostdinclude"

[v-gb]

Same response as in Changes.

[nojb]

If reinstalling over an existing installation, the "old" otherlibs in $(INSTALL_LIBDIR) should be removed here to avoid confusing the compiler. See #1724 for a similar fix for the switch to the packed Stdlib.

Idem for dynlink and compilerlibs

[v-gb]

Hum. Is there any reason to install over an existing installation directory? Is deleting the whole $(INSTALL_LIBDIR) directory in the install phony target possible?

[nojb]

Reinstalling over an existing installation is common when installing ocaml manually (ie without opam, under Windows, etc): see e.g. https://caml.inria.fr/mantis/view.php?id=7773.

Not sure if it is very friendly to outright delete the destination directory. In any case it is a change that should be documented.

[v-gb]

Ok, I tried to delete only the relevant cmi,cmx.

[nojb]

Shorter and faster:

install_into = \
  $(MKDIR) "$(1)" && \
  for suf in $(3); do \
    $(INSTALL_DATA) $(addsuffix $$suf,$(2)) "$(1)"; \
  done

[nojb]

Personally I don't like make function much, I find they often make the code harder to read; but I don't have a very strong argument against using one here.

[v-gb]

Are you sure? When I try this, I get errors like install: cannot stat 'parsing/parsetree.cmt': No such file or directory.

[v-gb]

(this .cmt doesn't exist because there is no .ml, which is fine, we simply shouldn't fail trying to install it)

[nojb]

Indeed, it is a bit more complicated than what I suggested.

Still I think it would be a good idea to avoid having one invocation per filename -- process creation is notoriously inefficient on some platforms (e.g. Cygwin).

[v-gb]

Ok. I couldn't find any filter-existing-files function, but test -f should not spawn, so we can probably filter file by file, and then do one big install call.

[v-gb]

After much struggle trying to understand how makefiles work, I have something that does one call to install per call to install_into.

[nojb]

I would write

let std_include_flags prefix =
  List.map (fun dir -> prefix ^ Filename.quote dir) (std_include_dirs ())

for less repetition.

[v-gb]

Indeed, not sure why I didn't write it this way in the first place.

[v-gb]

Thanks. I should have either taken your remarks into account or replied. And see #1875.

I noticed that actually the ocamldoc library was installed alongside the rest, not as a subdirectory. So I created a subdirectory for it too, and added it to the list of directories treated in a backwards compatible way, since I think this is what ocamldoc plugins compile against.
There's an ocamldoc.hva file that moves in there too, I don't know what it's for, and if perhaps it should stay in lib/ocaml.

I made a separate directory the stdlib. Now the only ocaml code that's directly in lib/ocaml is std_exit.* (hardcoded in the compilers), and profiling.* (ocamlprof runtime). I haven't tested much, but at least I can link programs.

I've never written nor seen tools that depend on the layout of the installation directory. The change to the stdlib increases the risk of tooling built against compiler-libs breaking, but that's not very different to any other compiler-libs change.
At least the manual doesn't document the layout of the installation directory, only how to build, and that's the compatibility we preserve (when not using -nostdincludes).

[nojb]

I've never written nor seen tools that depend on the layout of the installation directory.

At the very least, ocamlfind will need to be updated since it ships META files for standard libraries with explicit paths. opam may need adapting as well.

[gasche]

Personally I am worried by the amount of user scripts that may break because of this, especially people who have maintained their software since the pre-widespread-ocamlfind days. I think that we should evaluate the impact of this PR on at least the opam universe before merging. (The opam universe is a debatable choice, as it contains much fewer "old buildsystem cruft" than the full OCaml ecosystem, but it is the street lamp under which to search our keys.)

[gasche]

Is there a step-by-step guide somewhere on how to test a PR against opam? (@trefis?) We should probably include one in HACKING.adoc somewhere.

[trefis]

Nope, as far as I can tell it's still mostly impossible to do this (in a reliable fashion) unless you're @avsm (or @samoht).

My hope is that once opam-ci is "ready", there'll be a way for maintainers of ocaml/ocaml to add a target to the build matrice for a given PR and diff the results against the base of the PR (i.e. trunk or a specific branch).

[trefis]

Also, I should say that this PR seems (to me) like a must have, and Valentin seems to have given some thought to the backward compatibility story (but I have thought about it, nor reviewed the PR).
Could you explain why it doesn't seem satisfactory to you?

Also, there are potentially breaking changes with every release, and I don't think this one deserves more worry than any other (on the contrary).

[v-gb]

@nojb I think ocamlfind only needs updating to support -nostdincludes (except for the compiler-libs libraries), doesn't it? If you say -package unix and you don't get -I +unix, nothing breaks as long as you don't use -nostdincludes.

What do you think needs adapting in opam?

What would break is: ocamlopt $(ocamlc -where)/unix.cmxa instead of ocamlopt unix.cmxa.

[gasche]

Could you explain why it doesn't seem satisfactory to you?

I'm not saying it is not satisfactory, but I still get goosebumps thinking about compatibility, so I would like to get it tested.

(Maybe once the Menhir thing has settled it would be time to get back to our project of merging the best parts of Fabrice and Damien's scripts to get an opam-checker for normal people.)

[v-gb]

@dbuenzli it's supposed to work fine, and I tried and it's fine. What does break currently is that topdirs.cmi is not in the search, making the opam generated .ocamlinit not type check. But that's a mistake.

[dbuenzli]

@sliquister yes I read too quickly and deleted my comment but apparently email notifications fire immediately.

[v-gb]

This latest batch of changes should have addressed all remarks, and fixed the topdirs.cmi issue.

I was thinking it's actually not hard to parse -I +compiler-libs as -I +ocamlcommon -I +ocamlbytecomp ... (though it makes Misc.expand_directory's type slightly weird, as it needs to return a string list), so perhaps that'd be better, as it's a bit more backwards compatible.

[xavierleroy]

Also I don't see the logic behind putting ocamlcommon, ocamltoplevel, etc, in separate places. These libraries are all part of the same (conceptual) package.

[ghost]

@xavierleroy on the other hand the current situation regarding bigarray is not great either: in order to use Stdlib.Bigarray, you are forced to write Stdlib.Bigarray because the bigarray.cmi file from the (compatibility) bigarray library shadows the Bigarray module from the stdlib. We wouldn't have this issue if it was installed in a separate directory.

[v-gb]

@xavierleroy if someone has -I +bigarray in their build flags, they need to specify bigarray.cmxa at link time, so removing the bigarray library would break compability because the cmxa is gone, not because of the extra pointless -I (which causes no issues). For the compiler libs, if you use them, you use and link a subset of ocamlcommon, ocamlbytecomp, ocamltoplevel etc, not all of them.
It seems to me that splitting things off has exactly the same effect as this pr: force people to say which library they depend on. So it should no easier or harder.

[xavierleroy]

It seems to me that splitting things off has exactly the same effect as this pr: force people to say which library they depend on.

To some extent, yes. It's the other evolution that breaks: moving some of the otherlibs into the standard library.

For the compiler libs, if you use them, you use and link a subset of ocamlcommon, ocamlbytecomp, ocamltoplevel etc, not all of them.

Yes, that's why I put the appropriate .cmas on the command line, not all of them. It doesn't follow that they should be installed separate directories. Those libraries share a bunch of .cmis. Everything should be in the same directory.

[ghost]

Specifying include directories and archive files manually is not very practical though, especially when you consider the transitive dependencies. Since ocamlfind was introduced, the vast majority of projects have been specifying their dependencies as library names. A library name cover both what is visible to the typer and what will be used during linking, so the two should match.

[v-gb]

It seems to me that splitting things off has exactly the same effect as this pr: force people to say which library they depend on.

To some extent, yes. It's the other evolution that breaks: moving some of the otherlibs into the standard library.

I don't understand. If bigarray is in the stdlib directory, then you build this way:

ocamlopt -c a.ml
ocamlopt bigarray.cmxa a.cmx

If bigarray gets merged, then you do:

ocamlopt -c a.ml
ocamlopt a.cmx

so either you keep bigarray.cmxa (empty is fine), or you break compatibility .

If bigarray is in a separate dir, you build this way:

ocamlopt -I +bigarray -c a.ml
ocamlopt -I +bigarray bigarray.cmxa a.cmx

If bigarray gets merged, then you do:

ocamlopt -c a.ml
ocamlopt a.cmx

but this works perfectly fine too:

ocamlopt -I +bigarray -c a.ml
ocamlopt -I +bigarray a.cmx

so the compatibility is the same regardless of whether bigarray is installed in the installation directory of the stdlib.

For the compiler libs, if you use them, you use and link a subset of ocamlcommon, ocamlbytecomp, ocamltoplevel etc, not all of them.

Yes, that's why I put the appropriate .cmas on the command line, not all of them. It doesn't follow that they should be installed separate directories. Those libraries share a bunch of .cmis. Everything should be in the same directory.

Do they share cmis? When I look at the modules of all cmxa in compiler-libs, I see no overlap. So the only possible overlap would be for mli without ml (since they wouldn't be listed), but I highly doubt we'd have overlap for specifically such files.

[ghost]

Another way to solve this problem would be to provide a DSL to represent the initial typing environment. We already have such a DSL, but it is limited to including directories and opening modules, which is not very flexible. What I'm proposing is simply to extend it and provide a syntax for it. This way we wouldn't have to change anything in the way the compiler install files and third-party tools such as dune or other build system could expose the model of their choice to the user.

Here is what such as DSL could look like:

module Stdlib__list = "+stdlib__list.cmi"
module Stdlib__string = "+stdlib__string.cmi"
...
module Stdlib = "+stdlib.cmi"
open Stdlib
module My_module = "./my_module.cmi"
...

[gasche]

This sounds like you are going in the direction of my old module namespace proposals.

[v-gb]

@diml currently I don't think you can easily tell which library the .cmi of .mli without .ml, like compiler-libs/annot.cmi, are from. There aren't hundreds of them, but it's kind of annoying, and the set does change from times to times. While not critical, it's also just simpler to understand the structure if not everything goes in one or two directories.

I could also remove all the changes other than to Makefiles, make the layout configurable and keep the default unchanged: make install SEPARATE=true. I find that worse, but for me, being able to split out libraries is more important than making it the default, and I need none of all the compatibility stuff, it's purely there to support the change of default.

[ghost]

This sounds like you are going in the direction of my old module namespace proposals.

@gasche indeed, your proposal looks ambitious though :)

@diml currently I don't think you can easily tell which library the .cmi of .mli without .ml, like compiler-libs/annot.cmi, are from. There aren't hundreds of them, but it's kind of annoying, and the set does change from times to times. While not critical, it's also just simpler to understand the structure if not everything goes in one or two directories.

Agreed.

I could also remove all the changes other than to Makefiles, make the layout configurable and keep the default unchanged: make install SEPARATE=true. I find that worse, but for me, being able to split out libraries is more important than making it the default, and I need none of all the compatibility stuff, it's purely there to support the change of default.

Yh, I find having an option here worse as well. It's more things to test and maintain.

[ghost]

I think the proper way to go about "tidying" the otherlibs is first to split some of them off the core distribution and into independent packages, and then and only then change their installation directory.

@xavierleroy I can provide feedback for this method as we tried to do that for Camlp4. The current release story for Camlp4 is a pain as it needs two opam packages for every new release of OCaml. It is painful for the person doing the Camlp4 releases (me) and it has been the cause of plenty of hard to debug problems in the past for users. About a year or two after Camlp4 was split off the distribution, we tried to use a more classic installation layout to avoid this duplication; it was a disaster so we had to go back.

Moving forward, installing things in separate directories will make the process of splitting things off from the core distribution a lot smoother.

[rgrinberg]

I'd like to express that there's strong user support for this. It's quite common to see users get late errors when linking executables due to the absence of archives for str or unix. Some opam package authors even publish libraries that do not express their dependencies on these libraries correctly.

[v-gb]

@xavierleroy given the replies to your remarks, do you still have reservations about this pull request? Knowing would help decide what to do with it.