From 1cf0315be1e9f35b1b1157d444252293d25b33f6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20B=C3=BCnzli?= Date: Thu, 23 Apr 2020 23:56:37 +0200 Subject: [PATCH 01/16] Add library linking proposal v2. --- rfcs/ocamlib.md | 400 +++++++++++++++++++++++++++++++----------------- 1 file changed, 259 insertions(+), 141 deletions(-) diff --git a/rfcs/ocamlib.md b/rfcs/ocamlib.md index ac54740..86f9f29 100644 --- a/rfcs/ocamlib.md +++ b/rfcs/ocamlib.md @@ -192,7 +192,7 @@ executable that uses `MYLIB`. We store library dependencies in library archives (`cma`, `cmxa` and `cmxs` files) as *library names* to be looked up in `OCAMLPATH` at (dyn)link time. They are specified at library archive creation time -via the repeatable `-lib-require` flag. +via the repeatable `-require` flag. It was decided to push library dependencies in the library archives themselves rather than have yet another compiler managed side-car @@ -202,201 +202,317 @@ file. Here are few points that motivated this decision: no codec for it to be implemented in the compiler. * It is one file less that can be missing or corrupted at the point where an archive has to be used. -* It makes library archives self-contained (e.g. this is nice for `cmxs` - application plugins). +* It makes library archives self-contained and allows bare + archives to declare their library dependencies (e.g. this is nice + for `cmxs` application plugins). * If needed, it makes it easier to change or migrate the information structure internally. The format is allowed to change on each version of the compiler. * Since it's not written as a separate file there's no fight about who gets to write it during parallel `cma`, `cmxa` and `cmxs` archive creation. * Systems that need to get or store the information separately can - extract it with an `ocamlobjinfo -lib-requires` invocation which provides a - simple line based and stable output accross versions of the OCaml compiler. + extract it with an `ocamlobjinfo` which provides a stable output + accross versions of the OCaml compiler. Executables that are produced with `-linkall` flag also get the name of the libraries that were used embedded in them so that the `Dynlink` API can properly load libraries with library dependencies matching those of the executable without double linking. +## Command line interface commonalites + +### Support for `OCAMLPATH` extension + +Every tool from the toolchain that interprets the `OCAMLPATH` +environment variable gets a new repeateable and and ordered `-L PATH` +option that allows to prepend `PATH` to `OCAMLPATH` from left to +right. For example with: + +``` +ocamlc -L dir1 -L dir2 ... +``` + +The effective `OCAMLPATH` for the invocation becomes `dir2:dir1:$(OCAMLPATH)`. + +### Distinction between library names and file paths + +The `-require ARG` option argument `ARG` allows, in certain cases, to specify +either a library name `LIB` or a file path `PATH` to an object. A +`PATH` is unconditionally detected by the presence of the directory +separator (`/` on Unix, `/` or `\` on Windows), otherwise the argument +is parsed as library name. To specify a file in the current directory +you must thus use `./FILE`. + ## `ocamlc` support The following behaviours are added to `ocamlc`. -* Archive creation. `ocamlc -a -o ar.cma -lib-require MYLIB`. The - repeatable and ordered `-lib-require MYLIB` option adds the library - name `MYLIB` to the library dependencies of `ar.cma`. This stores - the information in a new field `lib_requires : string list` of the - `cma` [library descriptor][cma-lib-descriptor]. `MYLIB` does not - need to exist in the current `OCAMLPATH`. -* Compilation phase. `ocamlc -c -lib MYLIB src.ml`. The repeatable and - ordered `-lib MYLIB` option resolves `MYLIB` in `OCAMLPATH` and adds - its library directory to includes. Errors if `MYLIB` does not - resolve to an existing library. Type checking does not need - to access library archives. -* Link phase. `ocamlc -o a.out -lib MYLIB`. The repeatable and ordered - `-lib MYLIB` resolves `MYLIB` in `OCAMLPATH` and adds its `lib.cma` - file to the link sequence. It then consults the `lib_requires` field - of `lib.cma`, resolve these names in `OCAMLPATH` to their `lib.cma` - file and add them to the link sequence and recursively in stable - dependency order. Errors if any library name resolution fails. -* Link phase. Direct `ar.cma` arguments. The `lib_requires` field - is consulted and the names are resolved as in the preceeding point. -* Link phase. `-noautoliblink` can be specified to prevent the linking - of libraries specified in the `lib_requires` of the `.cma` files - considered for linking (either as direct argument or found via `-lib`). -* Link phase. If `-linkall` is specified on the cli, the name of libraries - specified via `-lib` and those recursively resolved is embedded in - the executable (for `Dynlink` API support). +### Archive creation + +A repeatable and *ordered* `-require LIB` option with `LIB` a library name +is added to archive creation mode. For example: +``` +ocamlc -a -o ar.cma -require LIB +``` +This adds the library name `LIB` to the library dependencies of +archive `ar.cma`. The information is stored in a new field +`lib_requires : string list` of the `cma` +[library descriptor][cma-lib-descriptor]. + +`LIB` does not need to exist in the current `OCAMLPATH` when the command +is invoked. [cma-lib-descriptor]: https://github.com/ocaml/ocaml/blob/e18564d8393475acd9e36f02fe3b0927fe8a0f5c/file_formats/cmo_format.mli#L53-L58 +### Compilation phase + +A repeatable and *ordered* `-require LIB` option with `LIB` a library +name is added to compilation phase. For example: + +``` +ocamlc -c -I bla -require LIB src.ml +``` +This resolves `LIB` in the `OCAMLPATH` and adds its library directory +to the includes after `-I bla`. Effectively `-require LIB` is replaced +by `-I /path/to/LIB`. + +Errors if `LIB` does not resolve to an existing library. Note that +type checking does not need to access library archives. + +### Link phase + +A repeateable and *ordered* `-require LIB|PATH.cma` option is +added to the link phase. For example: + +``` +ocamlc -o a.out -require ARG src.ml +``` + +If `ARG` is: + +1. A library name `LIB`. The library is resolved in `OCAMLPATH` and its archive + file `lib.cma` is added to the link sequence. The `lib_requires` of + `lib.cma` is read and these names are resolved in `OCAMLPATH` to their + `lib.cma` archive, added to the link sequence and recursively in + stable dependency order. Errors if any library name resolution fails. +2. A direct path to an archive `PATH/ar.cma` file. The archive is added to + the link sequence. The `lib_requires` of `ar.cma` is read and + the dependencies resolved as is done in the previous point. + +### Dynlink API support on `-linkall` + +Support is provided to record in bytecode executables which libraries are +statically linked as a whole. + +This happens whenever `-linkall` is specified. In this case the name +of any library resolved in `OCAMLPATH` via `-require LIB|PATH.cma` is +embedded in the executable. + +Besides names specified via the new `-assume-require LIB` option are +also added to these names. `LIB` must be a library name and it does not +need to exist in the current `OCAMLPATH` when the command is invoked. + ## `ocamlopt` support The following behaviours are added to `ocamlopt` -* Static archive creation. `ocamlopt -a -o ar.cmxa -lib-require - MYLIB`. The repeatable and ordered `-lib-require MYLIB` option adds - the library name `MYLIB` to the library dependencies of - `ar.cmxa`. This stores the information in a new field - `lib_requires : string list` of the `cmxa` [library - descriptor][cmxa-lib-descriptor]. `MYLIB` does not need to exist in - the current `OCAMLPATH`. -* Shared archive creation. `ocamlopt -shared -a -o ar.cmxs - -lib-require MYLIB`. The repeatable and ordered `-lib-require - MYLIB` option adds the library name `MYLIB` to the library - dependencies of `ar.cmxs`. This store the information in a - new field `dynu_requires : string list` of the `cmxs` - [plugin descriptor][cmxs-plugin-descriptor]. - `MYLIB` does not need to exist in the current `OCAMLPATH`. If - the `cmxs` is produced from a `.cmxa` then we simply transfer - the latter's `lib_requires` field to the `dynu_requires` field - unless `-noautoliblink` is mentioned on the cli. -* Compilation phase. `ocamlopt -c -lib MYLIB src.ml`. The repeatable and - ordered `-lib MYLIB` option resolves `MYLIB` in `OCAMLPATH` and adds - its library directory to includes. Errors if `MYLIB` does not - resolve to an existing library. Type checking does not need - to access library archives. -* Link phase. `ocamlopt -o a.out -lib MYLIB`. The repeatable and - ordered `-lib MYLIB` option resolves `MYLIB` in `OCAMLPATH` and adds its - `lib.cmxa` file to the link sequence. It then consults the - `lib_requires` field of `lib.cmxa`, resolve these names in - `OCAMLPATH` to their `lib.cmxa` file and add them to the link - sequence and recursively in stable dependency order. Errors if any - library name resolution fails. -* Link phase. Direct `ar.cmxa` arguments. The `lib_requires` field - is consulted and the names are resolved as in the preceeding point. -* Link phase. `-noautoliblink` can be specified to prevent the linking - of libraries specified in the `lib_requires` of the `.cmxa` files - considered for linking. -* Link phase. If `-linkall` is specified on the cli, the name of - libraries specified via `-lib` and those recursively resolved is embedded in - the executable (for `Dynlink` API support). - +### Static archive creation + +A repeatable and *ordered* `-require LIB` option with `LIB` a library name +is added to archive creation mode. For example: +``` +ocamlopt -a -o ar.cmxa -require LIB +``` +This adds the library name `LIB` to the library dependencies of +archive `ar.cmxa`. The information is stored in a new field +`lib_requires : string list` of the `cmxa` +[library descriptor][cmxa-lib-descriptor]. + +`LIB` does not need to exist in the current `OCAMLPATH` when the command +is invoked. + [cmxa-lib-descriptor]: https://github.com/ocaml/ocaml/blob/8947a38b61c9de1f95bcd2f5ec8292987211bf4b/file_formats/cmx_format.mli#L53-L56 + +### Shared archive creation (plugins) + +A repeatable and *ordered* `-require LIB|PATH.cmxa` option with `LIB` a library +name is added to shared archive creation mode. For example: +``` +ocamlopt -shared -a -o ar.cmxs -require LIB +``` + +If `ARG` is: + +1. A library name `LIB`, this adds the library name `LIB` to the library + dependencies of the archive `ar.cmxs`. This information is stored in + a new field `dynu_requires : string list` of the `cmxs` + [plugin descriptor][cmxs-plugin-descriptor]. `LIB` does not need to + exist in the current `OCAMLPATH`. +2. A direct path to an archive `PATH/ar.cmxa` file. The archive + is added to the link sequence *and* its `lib_requires` field is + added to to the `dynu_requires` field of the `cmxs` plugin + descriptor. + [cmxs-plugin-descriptor]: https://github.com/ocaml/ocaml/blob/8947a38b61c9de1f95bcd2f5ec8292987211bf4b/file_formats/cmxs_format.mli#L32-L35 +### Compilation phase + +A repeatable and *ordered* `-require LIB` option with `LIB` a library +name is added to compilation phase. For example: + +``` +ocamlopt -c -I bla -require LIB src.ml +``` + +This resolves `LIB` in the `OCAMLPATH` and adds its library directory +to the includes after `-I bla`. Effectively `-require LIB` is replaced +by `-I /path/to/LIB`. + +Errors if `LIB` does not resolve to an existing library. Note that +type checking does not need to access library archives. + +### Link phase + +A repeateable and *ordered* `-require LIB|PATH.cma` option is +added to the link phase. For example: + +``` +ocamlopt -o a.out -require ARG src.ml +``` + +If `ARG` is: + +1. A library name `LIB`. The library is resolved in `OCAMLPATH` and its archive + file `lib.cmxa` is added to the link sequence. The `lib_requires` of + `lib.cmxa` is read and these names are resolved in `OCAMLPATH` to their + `lib.cmxa` archive, added to the link sequence and recursively in + stable dependency order. Errors if any library name resolution fails. +2. A direct path to an archive `PATH/ar.cmxa` file. The archive is added to + the link sequence. The `lib_requires` of `ar.cmxa` is read and + the dependencies resolved as is done in the previous point. + + +### Dynlink API support on `-linkall` + +Support is provided to record in native code executables which +libraries are statically linked as a whole. + +This whenever `-linkall` is specified. In this case the name of any +library resolved in `OCAMLPATH` via `-require LIB|PATH.cmxa` is +embedded in the executable. + +Besides names specified via the new `-assume-require LIB` option +are also added to these names. `LIB` must be a library name and it does +not need to exist in the current `OCAMLPATH` when the command is invoked. + + ## `ocamlobjinfo` support The following behaviours are added to `ocamlobjinfo`. -* The regular `ocamlobjinfo` output on `cmxa` and `cma` is extended - according to the current format to output the new `lib_requires` field. -* ~~`ocamlobjinfo` is made to work on `cmxs` by using the support provided - by `Dynlink` (if available) to output the plugin information rather - than relying on the BDF library (see POC [here][cmxs-read-poc])~~. - That approach is too fragile w.r.t. C bits that may live in the - cmxs see discussion [here](https://github.com/ocaml/ocaml/issues/9306). -* A new `-lib-requires` flag is added that extracts the new `lib_requires` - from `cma` and `cmxa` and `dyn_requires` from `cmxs` files. According - the following line based format: - ``` - File: PATH - LIBREQ0 - LIBREQ1 - LIBREQ2 - ... - ``` - -~~Note this makes `ocamlobjinfo` depend on the C `caml_natdynlink_open` -primitive investigate if the function is available with an error when -natdynlink is not available or if we need some kind of conditional -compilation.~~ - -[cmxs-read-poc]: https://gist.github.com/dbuenzli/0b773f4a4dd9f7a35f30cd9b671e48c5#file-readmeta-ml-L23-L31 +* The regular `ocamlobjinfo` output on `cma`, `cmxa` and `cmxs` (if + BDF library is available) is extended according to the current + format to output the new `lib_requires` and `dynu_requires` field.s ## `ocamldep`, `ocamldebug`, `ocamlmktop` and `ocamlmklib` support The following behaviours are added to `ocamldep` and `ocamldebug` -* `(ocamldep |ocamldebug) -lib MYLIB`. The repeatable and ordered - `-lib MYLIB` option resolves `MYLIB` in `OCAMLPATH` and adds its - library directory to includes. Errors if `MYLIB` does not resolve to +* `(ocamldep |ocamldebug) -require LIB`. The repeatable and ordered + `-require LIB` option resolves `LIB` in `OCAMLPATH` and adds its + library directory to includes. Errors if `LIB` does not resolve to an existing library. The following behaviours are added to `ocamlmktop` -* The repeatable and ordered `-lib MYLIB` option resolves and links - `MYLIB`'s dependencies as per `ocamlc` support. So goes for direct +* The repeatable and ordered `-require LIB` option resolves and links + `LIB`'s dependencies as per `ocamlc` support. So goes for direct `cma` arguments. The following behaviours are added to `ocamlmklib` -* The repeateable and ordered `-lib-require LIB` argument is added +* The repeateable and ordered `-require LIB` argument is added and propogated to the resulting `cma`, `cmxa` and `cmxs` files. ## `ocaml` and `ocamlnat` support The following behaviours are added to `ocaml` and `ocamlnat`. -* The repeatable and ordered `-lib MYLIB` option resolve - `MYLIB` in `OCAMLPATH` adds its directory to includes and - loads the library and its dependencies according to `cma` linking - (for `ocaml`, see `ocamlc` support) or to `cmxs` linking - (for `ocamlnat`, see `ocamlopt` support, though that happens with - the info in `cmxs` files). Note that directories of dependent - libraries are *not* added to the includes. - -* The new `#require "MYLIB"` directive is added to the interactive toplevel. - This directive resolves `MYLIB` in `OCAMLPATH` and adds its library - directory to includes. It then loads the library and its - dependencies as per the preceeding point. Here again directories of - dependent libraries are *not* added to includes. - -* The `#load` directive is extended to load libraries along the lines of - `#require`. - -A nice side effect of the new `#require` is that it doesn't mention -file extensions, this allows to use it in `.ocamlinit` and have it -work both for `ocaml` and `ocamlnat`. +### `-require` option + +The repeatable and ordered `-require LIB|PATH.cm(a|xs)` is added +to `ocaml` and `ocamlnat`. For example: + +``` +ocaml -require ARG +``` + +If `ARG` is: + +1. A library name `LIB`. The library is resolved in `OCAMLPATH`, this + adds its directory to includes and loads the library archive and its + dependencies according to `cma` linking (for `ocaml`, see `ocamlc` support) + or to `cmxa` linking (for `ocamlnat` see `ocamlopt` support). Note that + directories of dependent libraries are *not* added to the includes. +2. A direct path to an archive `PATH/ar.cma` file. `PATH` is added to the + include directories and `ar.cma` and its dependencies are loaded like + in the preceeding point. + +Note that in the above if any of the names resolved in the `OCAMLPATH` is +already embedded in the `ocaml` or `ocamlnat` executable, the names are +not resolved (see `ocamlc` and `ocamlopt` Dynlink API support). + +### `#require` directive + +A new `#require "ARG"` directive is added to the toplevel. If `ARG` is + +1. A library name `LIB`. The library is resolved in `OCAMLPATH`, its + directory added to the includes, its library archive and its + dependencies are loaded according to `cma` linking + (for `ocaml`, see `ocamlc` support) or to `cmxa` linking + (for `ocamlnat`, see `ocamlopt` support)). Note that directories + of dependent libraries are *not* added to the includes. +2. A direct path to an archive `PATH/ar.cma` file. `PATH` is added + to the include di + + +Note that in the above if any of the names resolved in the `OCAMLPTH` is +already embedded in the `ocaml` or `ocamlnat` executable, the names are +not resolved (see `ocamlc` and `ocamlopt` Dynlink API support). + +A nice side effect of the new `#require` is that if used with library +names doesn't mention file extensions, this allows to use it in +`.ocamlinit` and have it work both for `ocaml` and `ocamlnat`. We will +also make sure to make `ocamlnat` use `Dynlink.adapt_filename` when file +paths are specified to `#require` so that a mention of `cma` get turns into +a mention of a `cmxs`. + ## `Dynlink` library support The following entry point is added: ``` -Dynlink.loadlib : ?ocamlpath:string list -> string -> unit +Dynlink.require : ocamlpath:string list -> string -> unit ``` +In `Dynlink.require ~ocamlpath arg` if `arg` is: -In byte code programs if a `cma` is loaded its `lib_requires` field -(see `ocamlc -a` support) is consulted and the library names are -resolved. First we check if those exist in the executable itself -(cf. linking with `-linkall` above), if that happens nothing needs to -be done. Otherwise the library is resolved to a `cma` file to load -according to the directories mentioned in `ocamlpath` (defaults to the -contents of `OCAMLPATH`) and recursively in dependency order. - -In native code the same happens with `cmxs` files via the new -`dynu_requires` field (see `ocamlopt -shared` support). +1. A library name `LIB`, then `LIB` is looked up in the `OCAMLPATH`, its + library archive gets loaded and its recursive dependencies in the correct + order. +2. An archive `PATH/ar.cm(a|xs)`, then the archive and its recursive + dependencies in the correct order. + +In both cases if any resolution requested in the `OCAMLPATH` is a library +name that already exists in the executable (see `ocamlc` and `ocamlopt` Dynlink +API support), the requested name is not looked up and assumed to be already +loaded. Keeping track of library names present in the executable and those -loaded by calls to `Dynlink.loadlib` will be a matter of extending the +loaded by calls to `Dynlink.require` is a matter of extending the existing Dynlink API's [state structure][dynlink-state] which currently keeps track of these things at the compilation unit level. -*Note* it might be possible to meld this into the existing entry -points of the `Dynlink` module rather than add a new entry point but -there are a few things to consider w.r.t. access control. - [dynlink-state]: https://github.com/ocaml/ocaml/blob/03c33f500563f3e12355694f1add98e7bd1096ae/otherlibs/dynlink/dynlink_common.ml#L45-L63 ## OCaml library install support @@ -430,7 +546,7 @@ as if the following `R/foo/META` file exists: ``` package "bar" ( -requires = ... # contents of `ocamlobjinfo -lib-requires R/foo/bar/lib.cma` +requires = ... # contents of `ocamlobjinfo's require of R/foo/bar/lib.cma` directory = "bar" archive(byte) = "lib.cma" archive(native) = "lib.cmxa" @@ -446,8 +562,8 @@ Since `ocamlfind` is in charge to put the recursive archive dependencies on the compiler cli at the link phase and that there may be a mix of archive using `META` files and others using the new mecanism it must take care to resolve the dependencies of the latter -according to the `OCAMLPATH` semantics and specify `-noautoliblink` so -that no double linking occurs. +according to the `OCAMLPATH` semantics and specify them as bare archives +on the command line so that no double linking occurs. ## `dune` support @@ -457,7 +573,7 @@ installs. Once the compiler supports this proposal, Dune will start installing library artifacts using the convention described in this document. It -will also pass the `-lib-require` field when assembling library +will also pass the `-require` field when assembling library archives. This will be a non-breaking change since from the point of view of users the naming of artifacts is a implementation detail of Dune. Dune will only enforce the new convention for the libraries it @@ -515,9 +631,11 @@ Work on the proposal has started thanks to a grant of the ### OCaml status Partial implementation for the compiler support is available -[here](https://github.com/ocaml/ocaml/compare/trunk...dbuenzli:ll). For -interested reviewers, the patches are meant to be read individually and in -sequence. +[here](https://github.com/ocaml/ocaml/compare/trunk...dbuenzli:ll) +for the previous iteration of this RFC. + +For interested reviewers, the patches are meant to be read +individually and in sequence. To create an opam switch with a compiler that has that support in you can use: ``` From bda01ff97cbc6ead4e006ece07053ae9d367fbac Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20B=C3=BCnzli?= Date: Wed, 2 Sep 2020 11:47:23 +0200 Subject: [PATCH 02/16] Add `Dynlink.{assume_library,loaded_libraries,is_library_loaded}`. --- rfcs/ocamlib.md | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/rfcs/ocamlib.md b/rfcs/ocamlib.md index 86f9f29..c3fd7da 100644 --- a/rfcs/ocamlib.md +++ b/rfcs/ocamlib.md @@ -473,7 +473,7 @@ A new `#require "ARG"` directive is added to the toplevel. If `ARG` is (for `ocamlnat`, see `ocamlopt` support)). Note that directories of dependent libraries are *not* added to the includes. 2. A direct path to an archive `PATH/ar.cma` file. `PATH` is added - to the include di + to the include directories. Note that in the above if any of the names resolved in the `OCAMLPTH` is @@ -490,10 +490,11 @@ a mention of a `cmxs`. ## `Dynlink` library support -The following entry point is added: +We add the following entry points: ``` Dynlink.require : ocamlpath:string list -> string -> unit +Dynlink.assume_library : string -> unit ``` In `Dynlink.require ~ocamlpath arg` if `arg` is: @@ -508,6 +509,22 @@ name that already exists in the executable (see `ocamlc` and `ocamlopt` Dynlink API support), the requested name is not looked up and assumed to be already loaded. +A call to `Dynlink.assume_library n` add library name `n` to the set of +loaded libraries names without loading anything. + +We also add the following functions to query the state of the program w.r.t. +to loaded libraries. +``` +Dynlink.loaded_libraries : unit -> string list +(** [Dynlink.loaded_libraries ()] is the + the list of library names loaded in the program + (including libraries already linked in the executable). *) + +Dynlink.is_library_loaded : string -> bool +(** [Dynlink.is_library_loaded l] is + [List.mem l (Dynlink.loaded_libraries ())] *) +``` + Keeping track of library names present in the executable and those loaded by calls to `Dynlink.require` is a matter of extending the existing Dynlink API's [state structure][dynlink-state] which From 1d22c8a0f7810bcbc772ee6bf65fe7e07fa66e6b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20B=C3=BCnzli?= Date: Wed, 2 Sep 2020 12:06:41 +0200 Subject: [PATCH 03/16] s/-assume-require/-assume-library/g --- rfcs/ocamlib.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/rfcs/ocamlib.md b/rfcs/ocamlib.md index c3fd7da..6c0198b 100644 --- a/rfcs/ocamlib.md +++ b/rfcs/ocamlib.md @@ -308,7 +308,7 @@ This happens whenever `-linkall` is specified. In this case the name of any library resolved in `OCAMLPATH` via `-require LIB|PATH.cma` is embedded in the executable. -Besides names specified via the new `-assume-require LIB` option are +Besides names specified via the new `-assume-library LIB` option are also added to these names. `LIB` must be a library name and it does not need to exist in the current `OCAMLPATH` when the command is invoked. @@ -401,7 +401,7 @@ This whenever `-linkall` is specified. In this case the name of any library resolved in `OCAMLPATH` via `-require LIB|PATH.cmxa` is embedded in the executable. -Besides names specified via the new `-assume-require LIB` option +Besides names specified via the new `-assume-library LIB` option are also added to these names. `LIB` must be a library name and it does not need to exist in the current `OCAMLPATH` when the command is invoked. @@ -769,21 +769,21 @@ was proposed above. code executables is done. Respectively by adding a new `LIBS` section in the byte code and a new `caml_imported_libs` symbol in native code with the set of library names that were fully linked - via `-require` and recursively. An `-assume-require LIB` was also + via `-require` and recursively. An `-assume-library LIB` was also added that allows to simply add `LIB` to the set of libraries that are supposed to be embedded in the executable. This is useful for handling `-linkall` with uninstalled libraries or for build systems that perform lookups and put library archives themselves on the cli (cf. the `-noautoliblink` flag). -11. In theory `-assume-require a` (see 10.) and `-require b` could +11. In theory `-assume-library a` (see 10.) and `-require b` could always be used together in particular if library `b` requires `a`, it won't be looked up. In practice however due to point 9. problems will arise at link time since the archive you specify for `a` on the cli will come after `b`'s one and lead to a link error. So at the moment having dependencies from `-require` - to `assume-require` libraries is not really supported but that is + to `assume-library` libraries is not really supported but that is not usually the case. If that happens though you should translate - all `-require` to `-assume-require`, do all the lookups and + all `-require` to `-assume-library`, do all the lookups and sorting yourself and use `-noautoliblink`. 12. Dynlink API: support for library loading was added. Works only in bytecode for now. Both `loadfile` and `loadfile_private` take a new From 37edf5806373cbca7513eb035d5cd6b9bf551bea Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20B=C3=BCnzli?= Date: Wed, 2 Sep 2020 12:19:46 +0200 Subject: [PATCH 04/16] Clarify how library objects should be looked up. --- rfcs/ocamlib.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/rfcs/ocamlib.md b/rfcs/ocamlib.md index 6c0198b..7050d6c 100644 --- a/rfcs/ocamlib.md +++ b/rfcs/ocamlib.md @@ -140,6 +140,12 @@ We get the following map between library directories and library names: /usr/lib/ocaml/ocaml/unix ocaml.unix /usr/lib/ocaml/re/emacs N/A (shadowed) +If you are implementing library object lookups in the `OCAMLPATH`, you +have to stop looking up after finding the first library directory. In +the example above assuming `re.emacs` has no bytecode version of the +library you *MUST NOT* look into `/usr/lib/ocaml/re/emacs` for the +bytecode version. + In what follows we use library names and the relative path it represents interchangeably. When a library name is used to denote a directory or file path we assume the `.` have been substituted with From 88318829aa7d33f9e0401f11d222a43cb74f62ca Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20B=C3=BCnzli?= Date: Wed, 2 Sep 2020 13:48:03 +0200 Subject: [PATCH 05/16] Add an example. --- rfcs/ocamlib.md | 51 +++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 49 insertions(+), 2 deletions(-) diff --git a/rfcs/ocamlib.md b/rfcs/ocamlib.md index 7050d6c..759b90e 100644 --- a/rfcs/ocamlib.md +++ b/rfcs/ocamlib.md @@ -482,12 +482,12 @@ A new `#require "ARG"` directive is added to the toplevel. If `ARG` is to the include directories. -Note that in the above if any of the names resolved in the `OCAMLPTH` is +Note that in the above if any of the names resolved in the `OCAMLPATH` is already embedded in the `ocaml` or `ocamlnat` executable, the names are not resolved (see `ocamlc` and `ocamlopt` Dynlink API support). A nice side effect of the new `#require` is that if used with library -names doesn't mention file extensions, this allows to use it in +names, it doesn't mention file extensions, this allows to use it in `.ocamlinit` and have it work both for `ocaml` and `ocamlnat`. We will also make sure to make `ocamlnat` use `Dynlink.adapt_filename` when file paths are specified to `#require` so that a mention of `cma` get turns into @@ -612,6 +612,53 @@ code base. Starting from Dune 4, Dune will no longer generate `META` files. +## Examples + +### Compiling and using libraries locally in a build + +The following shows how to compile and layout two libraries +to use locally in a build. We have: + +* Library `a` made of source `a.ml` depending on the `ocaml.str` + library. +* Library `b` made of source `b.ml` depending on the library `a`. +* An executable `exec.ml` which depends on `b` + +Here are our sources: + +``` +. +├── a.ml +├── b.ml +└── exec.ml +``` +We create directories for the libraries. The `libs` directory +will be added to the `OCAMLPATH`. + +``` +mkdir -p libs libs/a libs/b +``` + +We compile and layout library `a`: + +``` +ocamlopt -require ocaml.str -c -o libs/a/a.cmx a.ml +ocamlopt -require ocaml.str -a -o libs/a/lib.cmxa libs/a/a.cmx +``` + +We compile and layout library `b`. We extend the OCAMLPATH on the +command line via the `-L` option. + +``` +ocamlopt -L libs -require a -c -o libs/b/b.cmx b.ml +ocamlopt -L libs -require a -a -o libs/b/lib.cmxa libs/b/b.cmx +``` + +We compile our executable: +``` +ocamlopt -L libs -require b exec.ml +``` + ## Unresolved issues ### Compilation phase includes From 7e33fb71659d173ba2cb570ea7f884f0318d7272 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20B=C3=BCnzli?= Date: Mon, 7 Sep 2020 15:44:35 +0200 Subject: [PATCH 06/16] Add a section about OCAMLPATH init. --- rfcs/ocamlib.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/rfcs/ocamlib.md b/rfcs/ocamlib.md index 759b90e..7c00bd9 100644 --- a/rfcs/ocamlib.md +++ b/rfcs/ocamlib.md @@ -661,6 +661,18 @@ ocamlopt -L libs -require b exec.ml ## Unresolved issues +### OCAMLPATH initial value + +At the moment the proposal has no default value for the `OCAMLPATH`. +This means that without specifying any `OCAMLPATH` only the stdlib is +available for compiling and by definition any `-require LIB` will +fail for any library name `LIB`. + +It is expected that system itegrators will setup an appropriate +`OCAMLPATH` when needed (e.g. on `eval $(opam env)`) and according to +the install structure they choose. Whether a more complex mecanism is +needed will be discussed in due time with system integrators. + ### Compilation phase includes At the moment the proposal indicates that during the compilation phase From d99bf76a6a7124aaab88078ac90d609e22473284 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20B=C3=BCnzli?= Date: Tue, 22 Sep 2020 22:11:57 +0200 Subject: [PATCH 07/16] Fix stupid -L specification. The order is inconsistent with both ocaml's `-I` and gcc's `-L`. --- rfcs/ocamlib.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/ocamlib.md b/rfcs/ocamlib.md index 7c00bd9..e8714df 100644 --- a/rfcs/ocamlib.md +++ b/rfcs/ocamlib.md @@ -238,7 +238,7 @@ right. For example with: ocamlc -L dir1 -L dir2 ... ``` -The effective `OCAMLPATH` for the invocation becomes `dir2:dir1:$(OCAMLPATH)`. +The effective `OCAMLPATH` for the invocation becomes `dir1:dir2:$(OCAMLPATH)`. ### Distinction between library names and file paths From 4583b37ec7d05cfc16823c59c9ff1e684225892e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20B=C3=BCnzli?= Date: Fri, 25 Sep 2020 00:11:57 +0200 Subject: [PATCH 08/16] Bytecode -assume-library and -linkall: make more precise. --- rfcs/ocamlib.md | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/rfcs/ocamlib.md b/rfcs/ocamlib.md index e8714df..a4dacd1 100644 --- a/rfcs/ocamlib.md +++ b/rfcs/ocamlib.md @@ -304,19 +304,23 @@ If `ARG` is: 2. A direct path to an archive `PATH/ar.cma` file. The archive is added to the link sequence. The `lib_requires` of `ar.cma` is read and the dependencies resolved as is done in the previous point. + +A repeateable and unordered `-assume-library LIB` option is added to +the link phase. `LIB` must be a library name and it does not need to +exist in the current `OCAMLPATH` when the command is invoked. Using +this flag requires library name `LIB` and assumes it was already +resolved. The user is in charge in providing its functionality on +the cli as bare objects. ### Dynlink API support on `-linkall` -Support is provided to record in bytecode executables which libraries are -statically linked as a whole. +Support is provided to record which libraries are fully statically +linked in bytecode executables. This happens whenever `-linkall` is specified. In this case the name -of any library resolved in `OCAMLPATH` via `-require LIB|PATH.cma` is -embedded in the executable. - -Besides names specified via the new `-assume-library LIB` option are -also added to these names. `LIB` must be a library name and it does not -need to exist in the current `OCAMLPATH` when the command is invoked. +of any library resolved in `OCAMLPATH` via `-require LIB|PATH.cma` and +those specified via `-assume-library LIB` is embedded in the bytecode +executable in a new section called `LIBS`. ## `ocamlopt` support @@ -411,7 +415,6 @@ Besides names specified via the new `-assume-library LIB` option are also added to these names. `LIB` must be a library name and it does not need to exist in the current `OCAMLPATH` when the command is invoked. - ## `ocamlobjinfo` support The following behaviours are added to `ocamlobjinfo`. @@ -419,6 +422,8 @@ The following behaviours are added to `ocamlobjinfo`. * The regular `ocamlobjinfo` output on `cma`, `cmxa` and `cmxs` (if BDF library is available) is extended according to the current format to output the new `lib_requires` and `dynu_requires` field.s +* The output on bytecode executable is extended to report the new + `LIBS` section in an "Imported Libraries" section. ## `ocamldep`, `ocamldebug`, `ocamlmktop` and `ocamlmklib` support From b631c71b48d164f1a297c285bc5e9844dcd0dca3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20B=C3=BCnzli?= Date: Mon, 28 Sep 2020 22:38:08 +0200 Subject: [PATCH 09/16] Native code -assume-library and -linkall: make more precise. --- rfcs/ocamlib.md | 27 +++++++++++++++------------ 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/rfcs/ocamlib.md b/rfcs/ocamlib.md index a4dacd1..7caf7e9 100644 --- a/rfcs/ocamlib.md +++ b/rfcs/ocamlib.md @@ -287,7 +287,7 @@ type checking does not need to access library archives. ### Link phase -A repeateable and *ordered* `-require LIB|PATH.cma` option is +A repeatable and *ordered* `-require LIB|PATH.cma` option is added to the link phase. For example: ``` @@ -317,7 +317,7 @@ the cli as bare objects. Support is provided to record which libraries are fully statically linked in bytecode executables. -This happens whenever `-linkall` is specified. In this case the name +This happens whenever `-linkall` flag is specified. In this case the name of any library resolved in `OCAMLPATH` via `-require LIB|PATH.cma` and those specified via `-assume-library LIB` is embedded in the bytecode executable in a new section called `LIBS`. @@ -400,20 +400,23 @@ If `ARG` is: 2. A direct path to an archive `PATH/ar.cmxa` file. The archive is added to the link sequence. The `lib_requires` of `ar.cmxa` is read and the dependencies resolved as is done in the previous point. - + +A repeateable and unordered `-assume-library LIB` option is added to +the link phase. `LIB` must be a library name and it does not need to +exist in the current `OCAMLPATH` when the command is invoked. Using +this flag requires library name `LIB` and assumes it was already +resolved. The user is in charge in providing its functionality on the +cli as bare objects. ### Dynlink API support on `-linkall` -Support is provided to record in native code executables which -libraries are statically linked as a whole. - -This whenever `-linkall` is specified. In this case the name of any -library resolved in `OCAMLPATH` via `-require LIB|PATH.cmxa` is -embedded in the executable. +Support is provided to record which libraries are fully statically +linked in bytecode executables. -Besides names specified via the new `-assume-library LIB` option -are also added to these names. `LIB` must be a library name and it does -not need to exist in the current `OCAMLPATH` when the command is invoked. +This happens whenever the `-linkall` flag is specified. In this case +the name of any library resolved in `OCAMLPATH` via `-require LIB|PATH.cma` +and those specified via `-assume-library LIB` is embedded in the native +code exectuable in a new `caml_imported_libs` symbol. ## `ocamlobjinfo` support From 97cd1d9dbc9818aa99031458a1bab570503b9c55 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20B=C3=BCnzli?= Date: Mon, 28 Sep 2020 22:29:47 +0200 Subject: [PATCH 10/16] Dynlink state API: change to what was done in the first iteration. The naming is consistent with what is done at the compilation unit level. --- rfcs/ocamlib.md | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/rfcs/ocamlib.md b/rfcs/ocamlib.md index 7caf7e9..7c0fde9 100644 --- a/rfcs/ocamlib.md +++ b/rfcs/ocamlib.md @@ -526,17 +526,24 @@ loaded. A call to `Dynlink.assume_library n` add library name `n` to the set of loaded libraries names without loading anything. -We also add the following functions to query the state of the program w.r.t. -to loaded libraries. +Mirroring the existing support for compilation units, we also add the following +functions to query the state of the program w.r.t. to loaded libraries. + ``` -Dynlink.loaded_libraries : unit -> string list -(** [Dynlink.loaded_libraries ()] is the - the list of library names loaded in the program - (including libraries already linked in the executable). *) +val Dynlink.main_program_libraries : unit -> string list +(** [main_program_libraries ()] is the list of library names statically linked + in the program. *) + +val Dynlink.public_dynamically_loaded_libraries : unit -> string list +(** [public_dynamically_loaded_libraries ()] is the list of library names + that were dynamically loaded via {!require}. *) + +val Dynlink.all_libraries : unit -> string list +(** [all_libraries ()] is the union of {!main_program_libraries} and + {!public_dynamically_loaded_libraries}. *) -Dynlink.is_library_loaded : string -> bool -(** [Dynlink.is_library_loaded l] is - [List.mem l (Dynlink.loaded_libraries ())] *) +val Dynlink.has_library : string -> bool +(** [Dynlink.has_library l] is [List.mem l (Dynlink.all_libraries ())]. *) ``` Keeping track of library names present in the executable and those From 59384a05b8c99ebf3442d23c9c2c81094d4e1d48 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20B=C3=BCnzli?= Date: Tue, 29 Sep 2020 00:27:30 +0200 Subject: [PATCH 11/16] Library directory constraints: make more precise and raise issues. --- rfcs/ocamlib.md | 35 ++++++++++++++++++++--------------- 1 file changed, 20 insertions(+), 15 deletions(-) diff --git a/rfcs/ocamlib.md b/rfcs/ocamlib.md index 7c0fde9..10af2bd 100644 --- a/rfcs/ocamlib.md +++ b/rfcs/ocamlib.md @@ -172,21 +172,26 @@ corresponding DLLs as: The following constraints are assumed to hold on a library directory. -1. If a `M.cmx` has no `cmi` then the module is private to the library. -2. If a `M.cmi` has no `M.cmx` and no implementation in the archives then - it is an `mli`-only module. -3. For any existing `M.cmx` the same `M.cmx` is present in the `.cmxa` archive. -4. All `lib.cma`, `lib.cmxa` and `lib.cmxs` (as available) contain the same - compilation units and the same library dependency specifications - (see below). -5. A library is allowed to contain only `mli`-only modules or be - totally empty. In this case the library objects must still be - present but are empty. They are allowed to have library dependency - specifications (see below). (Empty archives did pose problems - at a certain point but it seem this is mostly fixed, - see [#9011](https://github.com/ocaml/ocaml/pull/9011), - [#6550](https://github.com/ocaml/ocaml/issues/6550) and - [#1094](https://github.com/ocaml/ocaml/pull/1094)). +1. If a `m.cmx` has no `m.cmi` then the module is private to the library. +2. If a `m.cmi` has no `m.cmx` and no implementation in the archives then it + is an `mli`-only module. +3. For any existing `m.cmx` the same `m.cmx` must be present in the `lib.cmxa` + archive. +4. If there is a `m.cmx` file there must be a `lib.cmxa`. +5. If there is a `lib.cmxa` there must be a `lib.a` unless `lib.cmxa` + is empty (since 4.12). +6. FIXME discuss. All `lib.{cma,cmxa,cmxs}` files (as available) contain the + same compilation units. +7. FIXME discuss. All `lib.{cma,cmxa,cmxs}` files (as available) contain the + same dependency specifications. +8. Empty archives are allowed. They can + contain library dependency specifications which are used at link time. +9. A missing library archive means that the library is not available for the + given code backend, failures are reported on usage at link time. This + entails that a library made from `mli`-only modules must install empty + `lib.{cma,cmxa,cmxs}` files in its library directory so that the library + can be used during the compilation and link phase without users needing + to know it has no implementation. ## Library dependency specifications From 4dd989d5434af17439731e635331e0f2b67f3e17 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20B=C3=BCnzli?= Date: Wed, 30 Sep 2020 17:01:38 +0200 Subject: [PATCH 12/16] OCAMLPATH: define a behaviour if undefined. --- rfcs/ocamlib.md | 39 +++++++++++++++++++++++++++------------ 1 file changed, 27 insertions(+), 12 deletions(-) diff --git a/rfcs/ocamlib.md b/rfcs/ocamlib.md index 10af2bd..bb610d6 100644 --- a/rfcs/ocamlib.md +++ b/rfcs/ocamlib.md @@ -232,6 +232,33 @@ those of the executable without double linking. ## Command line interface commonalites +### Support for `OCAMLPATH` default value + +A new configuration variable `DEFAULT_OCAMLPATH` is added to OCaml's +`configure` (defaults to empty). This value is used in case the +`OCAMLPATH` environment variable is undefined. + +For example with: +``` +./configure DEFAULT_OCAMLPATH=/usr/lib/ocaml:/usr/local/lib/ocaml +``` +the default `OCAMLPATH` value, when the variable is undefined, will be: +``` +/usr/lib/ocaml:/usr/local/lib/ocaml +``` +A new option `-ocamlpath` is added to `ocaml{c,opt}` which simply +prints the contents of the `OCAMLPATH` environment variable or the +default value if undefined. + +This allows end user to extend the `OCAMLPATH` with for example: +``` +export OCAMLPATH=~/.local/lib/ocaml:$(ocamlc -ocamlpath) +``` +With this system, reading the value `OCAMLPATH` variable, if defined, always +gives a total picture of what is going on. + +FIXME Ask system packagers if that works for them. + ### Support for `OCAMLPATH` extension Every tool from the toolchain that interprets the `OCAMLPATH` @@ -681,18 +708,6 @@ ocamlopt -L libs -require b exec.ml ## Unresolved issues -### OCAMLPATH initial value - -At the moment the proposal has no default value for the `OCAMLPATH`. -This means that without specifying any `OCAMLPATH` only the stdlib is -available for compiling and by definition any `-require LIB` will -fail for any library name `LIB`. - -It is expected that system itegrators will setup an appropriate -`OCAMLPATH` when needed (e.g. on `eval $(opam env)`) and according to -the install structure they choose. Whether a more complex mecanism is -needed will be discussed in due time with system integrators. - ### Compilation phase includes At the moment the proposal indicates that during the compilation phase From e68f60f4c7d153c360823d93a329fac292dd9e24 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20B=C3=BCnzli?= Date: Thu, 1 Oct 2020 01:24:07 +0200 Subject: [PATCH 13/16] Re-add ocamlobjinfo -requires output. --- rfcs/ocamlib.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/rfcs/ocamlib.md b/rfcs/ocamlib.md index bb610d6..edc8014 100644 --- a/rfcs/ocamlib.md +++ b/rfcs/ocamlib.md @@ -459,6 +459,8 @@ The following behaviours are added to `ocamlobjinfo`. format to output the new `lib_requires` and `dynu_requires` field.s * The output on bytecode executable is extended to report the new `LIBS` section in an "Imported Libraries" section. +* A new option `-requires` is added that only prints libraries required + by `cma`, `cmxa` and `cmxs` objects, one per line. ## `ocamldep`, `ocamldebug`, `ocamlmktop` and `ocamlmklib` support From 73168addaaa166914dc243d750f508009acefec4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20B=C3=BCnzli?= Date: Thu, 1 Oct 2020 11:42:16 +0200 Subject: [PATCH 14/16] Add Dynlink.{default_ocamlpath,ocamlpath_of_string}. --- rfcs/ocamlib.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/rfcs/ocamlib.md b/rfcs/ocamlib.md index edc8014..66d971d 100644 --- a/rfcs/ocamlib.md +++ b/rfcs/ocamlib.md @@ -523,7 +523,6 @@ A new `#require "ARG"` directive is added to the toplevel. If `ARG` is 2. A direct path to an archive `PATH/ar.cma` file. `PATH` is added to the include directories. - Note that in the above if any of the names resolved in the `OCAMLPATH` is already embedded in the `ocaml` or `ocamlnat` executable, the names are not resolved (see `ocamlc` and `ocamlopt` Dynlink API support). @@ -560,6 +559,21 @@ loaded. A call to `Dynlink.assume_library n` add library name `n` to the set of loaded libraries names without loading anything. +We also provide the following two functions to let client handle their +own `OCAMLPATH`-like variable. + +``` +val Dynlink.default_ocamlpath : string list +(** [default_ocamlpath] is the value of the ocamlpath that has been + set at OCaml configuration time. *) + +val Dynlink.ocamlpath_of_string : string -> string list +(** [ocamlpath_of_string s] parses [s] into a list of directories by + following the OS convention for [PATH]-like variables. This means they + are colon ':' (semi-colon ';' if {!Sys.win32}) separated paths. Empty + paths are allowed and discarded. *) +``` + Mirroring the existing support for compilation units, we also add the following functions to query the state of the program w.r.t. to loaded libraries. From 5598864917bebb4dbc47331ba9058a138f3468fe Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20B=C3=BCnzli?= Date: Thu, 1 Oct 2020 01:18:20 +0200 Subject: [PATCH 15/16] OCaml implementation notes v2. --- rfcs/ocamlib.md | 120 +++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 108 insertions(+), 12 deletions(-) diff --git a/rfcs/ocamlib.md b/rfcs/ocamlib.md index 66d971d..693fd67 100644 --- a/rfcs/ocamlib.md +++ b/rfcs/ocamlib.md @@ -755,17 +755,122 @@ problem with 2. is that it breaks the idea that the compilation phase need not be aware of the dependencies of a library which are currently only stored in library archives. - ## Implementation -Work on the proposal has started thanks to a grant of the +Work on the proposal is ongoing thanks to a grant of the [OCaml software foundation](http://ocaml-sf.org/). ### OCaml status +An OCaml implementation of the RFC is available and can be used with: + +``` +opam switch create ocaml-ll --empty +opam pin add ocaml-variants.4.12.0+ll git+https://github.com/dbuenzli/ocaml#ll-v2 +``` + +For interested reviewers, +[the patches](https://github.com/ocaml/ocaml/compare/trunk...dbuenzli:ll-v2) +are meant to be read individually and in sequence. + +A draft for a manual section about the library convention can be found +[here](https://github.com/dbuenzli/ocaml-lltest/blob/master/manual.mdy). + +A bit of testing can be found in [this +repository](https://github.com/dbuenzli/ocaml-lltest/). Feel free to +open issues you may find there. + +The following notes indicate what is *not* implemented according the +RFC, a few known problems and work that remains to be done. + +* RFC clarification. At the moment there are not provisions to dynamically + tweak the OCAMLPATH in the toplevel (like the `#directory` has for `-I`). + Should we have something ? +* RFC clarification. The current implementation adds the directory of any + archive as a `-L` to the C linker (see point 7 of the previous iteration) + in `{Byte,Asm}link` this is not mentioned in the RFC above. It should be, + along with the rationale. +* RFC clarification. Some of the library directory + [constraints](#semantics-and-integrity-constraints) + should be discussed (6 and 7). Namely uniformity of dependencies + and implementation in different backend archives. Maybe we could relax + that, I think that uniformity of *API* (i.e. implementation with a .cmi file) + across backends is important, not implementations and/or dependencies. +* RFC clarification. Support for the + [default `OCAMLPATH` value](#support-for-ocamlpath-default-value) value + should be discussed, notably with system packagers. +* Implementation bug. The current implementation triggers infinite loops in + case of recursively dependent libraries in `Dynlink.requires` and + `#require`. It is not difficult to fix; in contrast to `Asmlink` and + `Bytelink`, we only update the seen libraries after we are sure it + successfully loaded. We need to carry a separate seen libraries for + the current "require" load. +* Implementation enhancement. The current implementation is inconsistant + about using `lib[s]` and `librar{y,ies}` this should be uniformized. For + now public identifiers consistently use the latter, we should decide on one + (e.g. `-assume-lib` or `-assume-library`?). +* Implementation enhancement. Now that we have @nojb's `Binutils`, + `ocamlobjinfo` on native code executables could report the data of + `caml_imported_libs` and `caml_globals_map`. Like `ocamlobjinfo` + does with `LIBS` on bytecode executables (we get a bit less clear + error checking). +* Implementation RFC deviation. Requires and ordering. The RFC above has it + that the relative order + of `-require`, `-I` and positional arguments should be respected + modulo `-require` dependency sorting at link time. No good way was + found to do this in the code base without making it worse than it + already is. In the current implementation `-require` include effects + are put after all `-I` arguments and before positional + arguments. Incidentally this is what `ocamlfind` does. However for + positional arguments it doesn't work for 1) the idea of using + `-require FILE` to request resolution of `FILE` dependencies 2) + for using `-assume-library a implements_a.cma + -require b` for replacing the dependency `a` of `b` since the object + resolved by `-require b` gets placed before `implements_a.cma` and + link fails. Note that this is a cli user interface problem, the actual + linker implementation supports the interleaving correctly. +* Implementation RFC deviation. Archive creation, transfer of + `lib_requires` information when an archive + is created (`-a`) from another one (a cma from cmas, cmxs from cmxas). + This is still a bit unclear in the RFC above. It was not spelled out but + the idea was that you'd only transfer if `--require FILE` was put on + the `-a` invocation (previously we had the `-noautoliblink` to prevent + transfer). But the preceeding point makes that problematic. For now we + unconditionally transfer `lib_requires` of the arguments to the new + archive (because most build system out there only use this workflow + for creating a `.cmxs` matching a `.cmxa`). Depending on what happens + with the preceding point we still may want a mecanism to prevent + the transfer. (e.g. `-assume-library` could be used to remove + specific mentions in the final result). +* Implementation TODO. Manpages must be updated. +* Implementation TODO. Beyond this [new manual section](https://github.com/dbuenzli/ocaml-lltest/blob/master/manual.md), the narrative about libraries should be integrated in manual sections about individual tools. +* Implementation TODO. Test suite. The tests + [here](https://github.com/dbuenzli/ocaml-lltest) should be integrated + into the compiler test suite, as cram tests. (So that e.g. *stable* topo + sorts are tested). +* Implementation TODO. There is no support for `OCAMLPARAM`. Should there + be one ? + +## Old supporting work + +### OCAMLPATH + +There is a [PR](https://github.com/ocaml/ocaml/pull/8946) for making +`OCAMLPATH` meaningful to the OCaml toolchain. For now its only effect +is to redefine the `-I +DIR` notation so that distributions can start +reshuffling their install structure to make it easier to extend a +system OCaml package install prefix with an opam package install +prefix. + + +### OCaml status RFC v1 + +These are the note the for the [previous +iteration](https://github.com/ocaml/RFCs/pull/7) of the RFC. + Partial implementation for the compiler support is available [here](https://github.com/ocaml/ocaml/compare/trunk...dbuenzli:ll) -for the previous iteration of this RFC. + For interested reviewers, the patches are meant to be read individually and in sequence. @@ -939,15 +1044,6 @@ was proposed above. good idea or not. 15. This implementation has no support for `OCAMLPARAM`. -## Old supporting work -### OCAMLPATH - -There is a [PR](https://github.com/ocaml/ocaml/pull/8946) for making -`OCAMLPATH` meaningful to the OCaml toolchain. For now its only effect -is to redefine the `-I +DIR` notation so that distributions can start -reshuffling their install structure to make it easier to extend a -system OCaml package install prefix with an opam package install -prefix. From e5f45ba6e9568c120c58c70de298c3a93704189a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20B=C3=BCnzli?= Date: Thu, 1 Oct 2020 15:55:24 +0200 Subject: [PATCH 16/16] Implementation notes, enumerate the items. --- rfcs/ocamlib.md | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/rfcs/ocamlib.md b/rfcs/ocamlib.md index 693fd67..103c006 100644 --- a/rfcs/ocamlib.md +++ b/rfcs/ocamlib.md @@ -783,38 +783,38 @@ open issues you may find there. The following notes indicate what is *not* implemented according the RFC, a few known problems and work that remains to be done. -* RFC clarification. At the moment there are not provisions to dynamically +1. RFC clarification. At the moment there are not provisions to dynamically tweak the OCAMLPATH in the toplevel (like the `#directory` has for `-I`). Should we have something ? -* RFC clarification. The current implementation adds the directory of any +2. RFC clarification. The current implementation adds the directory of any archive as a `-L` to the C linker (see point 7 of the previous iteration) in `{Byte,Asm}link` this is not mentioned in the RFC above. It should be, along with the rationale. -* RFC clarification. Some of the library directory +3. RFC clarification. Some of the library directory [constraints](#semantics-and-integrity-constraints) should be discussed (6 and 7). Namely uniformity of dependencies and implementation in different backend archives. Maybe we could relax that, I think that uniformity of *API* (i.e. implementation with a .cmi file) across backends is important, not implementations and/or dependencies. -* RFC clarification. Support for the +4. RFC clarification. Support for the [default `OCAMLPATH` value](#support-for-ocamlpath-default-value) value should be discussed, notably with system packagers. -* Implementation bug. The current implementation triggers infinite loops in +5. Implementation bug. The current implementation triggers infinite loops in case of recursively dependent libraries in `Dynlink.requires` and `#require`. It is not difficult to fix; in contrast to `Asmlink` and `Bytelink`, we only update the seen libraries after we are sure it successfully loaded. We need to carry a separate seen libraries for the current "require" load. -* Implementation enhancement. The current implementation is inconsistant +6. Implementation enhancement. The current implementation is inconsistant about using `lib[s]` and `librar{y,ies}` this should be uniformized. For now public identifiers consistently use the latter, we should decide on one (e.g. `-assume-lib` or `-assume-library`?). -* Implementation enhancement. Now that we have @nojb's `Binutils`, +7. Implementation enhancement. Now that we have @nojb's `Binutils`, `ocamlobjinfo` on native code executables could report the data of `caml_imported_libs` and `caml_globals_map`. Like `ocamlobjinfo` does with `LIBS` on bytecode executables (we get a bit less clear error checking). -* Implementation RFC deviation. Requires and ordering. The RFC above has it +8. Implementation RFC deviation. Requires and ordering. The RFC above has it that the relative order of `-require`, `-I` and positional arguments should be respected modulo `-require` dependency sorting at link time. No good way was @@ -829,7 +829,7 @@ RFC, a few known problems and work that remains to be done. resolved by `-require b` gets placed before `implements_a.cma` and link fails. Note that this is a cli user interface problem, the actual linker implementation supports the interleaving correctly. -* Implementation RFC deviation. Archive creation, transfer of +9. Implementation RFC deviation. Archive creation, transfer of `lib_requires` information when an archive is created (`-a`) from another one (a cma from cmas, cmxs from cmxas). This is still a bit unclear in the RFC above. It was not spelled out but @@ -842,14 +842,14 @@ RFC, a few known problems and work that remains to be done. with the preceding point we still may want a mecanism to prevent the transfer. (e.g. `-assume-library` could be used to remove specific mentions in the final result). -* Implementation TODO. Manpages must be updated. -* Implementation TODO. Beyond this [new manual section](https://github.com/dbuenzli/ocaml-lltest/blob/master/manual.md), the narrative about libraries should be integrated in manual sections about individual tools. -* Implementation TODO. Test suite. The tests +10. Implementation TODO. Manpages must be updated. +11. Implementation TODO. Beyond this [new manual section](https://github.com/dbuenzli/ocaml-lltest/blob/master/manual.md), the narrative about libraries should be integrated in manual sections about individual tools. +12. Implementation TODO. Test suite. The tests [here](https://github.com/dbuenzli/ocaml-lltest) should be integrated into the compiler test suite, as cram tests. (So that e.g. *stable* topo sorts are tested). -* Implementation TODO. There is no support for `OCAMLPARAM`. Should there - be one ? +13. Implementation TODO. There is no support for `OCAMLPARAM`. Should there be + one ? ## Old supporting work