diff --git a/rfcs/ocamlib.md b/rfcs/ocamlib.md index ac54740..103c006 100644 --- a/rfcs/ocamlib.md +++ b/rfcs/ocamlib.md @@ -140,6 +140,12 @@ We get the following map between library directories and library names: /usr/lib/ocaml/ocaml/unix ocaml.unix /usr/lib/ocaml/re/emacs N/A (shadowed) +If you are implementing library object lookups in the `OCAMLPATH`, you +have to stop looking up after finding the first library directory. In +the example above assuming `re.emacs` has no bytecode version of the +library you *MUST NOT* look into `/usr/lib/ocaml/re/emacs` for the +bytecode version. + In what follows we use library names and the relative path it represents interchangeably. When a library name is used to denote a directory or file path we assume the `.` have been substituted with @@ -166,21 +172,26 @@ corresponding DLLs as: The following constraints are assumed to hold on a library directory. -1. If a `M.cmx` has no `cmi` then the module is private to the library. -2. If a `M.cmi` has no `M.cmx` and no implementation in the archives then - it is an `mli`-only module. -3. For any existing `M.cmx` the same `M.cmx` is present in the `.cmxa` archive. -4. All `lib.cma`, `lib.cmxa` and `lib.cmxs` (as available) contain the same - compilation units and the same library dependency specifications - (see below). -5. A library is allowed to contain only `mli`-only modules or be - totally empty. In this case the library objects must still be - present but are empty. They are allowed to have library dependency - specifications (see below). (Empty archives did pose problems - at a certain point but it seem this is mostly fixed, - see [#9011](https://github.com/ocaml/ocaml/pull/9011), - [#6550](https://github.com/ocaml/ocaml/issues/6550) and - [#1094](https://github.com/ocaml/ocaml/pull/1094)). +1. If a `m.cmx` has no `m.cmi` then the module is private to the library. +2. If a `m.cmi` has no `m.cmx` and no implementation in the archives then it + is an `mli`-only module. +3. For any existing `m.cmx` the same `m.cmx` must be present in the `lib.cmxa` + archive. +4. If there is a `m.cmx` file there must be a `lib.cmxa`. +5. If there is a `lib.cmxa` there must be a `lib.a` unless `lib.cmxa` + is empty (since 4.12). +6. FIXME discuss. All `lib.{cma,cmxa,cmxs}` files (as available) contain the + same compilation units. +7. FIXME discuss. All `lib.{cma,cmxa,cmxs}` files (as available) contain the + same dependency specifications. +8. Empty archives are allowed. They can + contain library dependency specifications which are used at link time. +9. A missing library archive means that the library is not available for the + given code backend, failures are reported on usage at link time. This + entails that a library made from `mli`-only modules must install empty + `lib.{cma,cmxa,cmxs}` files in its library directory so that the library + can be used during the compilation and link phase without users needing + to know it has no implementation. ## Library dependency specifications @@ -192,7 +203,7 @@ executable that uses `MYLIB`. We store library dependencies in library archives (`cma`, `cmxa` and `cmxs` files) as *library names* to be looked up in `OCAMLPATH` at (dyn)link time. They are specified at library archive creation time -via the repeatable `-lib-require` flag. +via the repeatable `-require` flag. It was decided to push library dependencies in the library archives themselves rather than have yet another compiler managed side-car @@ -202,201 +213,392 @@ file. Here are few points that motivated this decision: no codec for it to be implemented in the compiler. * It is one file less that can be missing or corrupted at the point where an archive has to be used. -* It makes library archives self-contained (e.g. this is nice for `cmxs` - application plugins). +* It makes library archives self-contained and allows bare + archives to declare their library dependencies (e.g. this is nice + for `cmxs` application plugins). * If needed, it makes it easier to change or migrate the information structure internally. The format is allowed to change on each version of the compiler. * Since it's not written as a separate file there's no fight about who gets to write it during parallel `cma`, `cmxa` and `cmxs` archive creation. * Systems that need to get or store the information separately can - extract it with an `ocamlobjinfo -lib-requires` invocation which provides a - simple line based and stable output accross versions of the OCaml compiler. + extract it with an `ocamlobjinfo` which provides a stable output + accross versions of the OCaml compiler. Executables that are produced with `-linkall` flag also get the name of the libraries that were used embedded in them so that the `Dynlink` API can properly load libraries with library dependencies matching those of the executable without double linking. +## Command line interface commonalites + +### Support for `OCAMLPATH` default value + +A new configuration variable `DEFAULT_OCAMLPATH` is added to OCaml's +`configure` (defaults to empty). This value is used in case the +`OCAMLPATH` environment variable is undefined. + +For example with: +``` +./configure DEFAULT_OCAMLPATH=/usr/lib/ocaml:/usr/local/lib/ocaml +``` +the default `OCAMLPATH` value, when the variable is undefined, will be: +``` +/usr/lib/ocaml:/usr/local/lib/ocaml +``` +A new option `-ocamlpath` is added to `ocaml{c,opt}` which simply +prints the contents of the `OCAMLPATH` environment variable or the +default value if undefined. + +This allows end user to extend the `OCAMLPATH` with for example: +``` +export OCAMLPATH=~/.local/lib/ocaml:$(ocamlc -ocamlpath) +``` +With this system, reading the value `OCAMLPATH` variable, if defined, always +gives a total picture of what is going on. + +FIXME Ask system packagers if that works for them. + +### Support for `OCAMLPATH` extension + +Every tool from the toolchain that interprets the `OCAMLPATH` +environment variable gets a new repeateable and and ordered `-L PATH` +option that allows to prepend `PATH` to `OCAMLPATH` from left to +right. For example with: + +``` +ocamlc -L dir1 -L dir2 ... +``` + +The effective `OCAMLPATH` for the invocation becomes `dir1:dir2:$(OCAMLPATH)`. + +### Distinction between library names and file paths + +The `-require ARG` option argument `ARG` allows, in certain cases, to specify +either a library name `LIB` or a file path `PATH` to an object. A +`PATH` is unconditionally detected by the presence of the directory +separator (`/` on Unix, `/` or `\` on Windows), otherwise the argument +is parsed as library name. To specify a file in the current directory +you must thus use `./FILE`. + ## `ocamlc` support The following behaviours are added to `ocamlc`. -* Archive creation. `ocamlc -a -o ar.cma -lib-require MYLIB`. The - repeatable and ordered `-lib-require MYLIB` option adds the library - name `MYLIB` to the library dependencies of `ar.cma`. This stores - the information in a new field `lib_requires : string list` of the - `cma` [library descriptor][cma-lib-descriptor]. `MYLIB` does not - need to exist in the current `OCAMLPATH`. -* Compilation phase. `ocamlc -c -lib MYLIB src.ml`. The repeatable and - ordered `-lib MYLIB` option resolves `MYLIB` in `OCAMLPATH` and adds - its library directory to includes. Errors if `MYLIB` does not - resolve to an existing library. Type checking does not need - to access library archives. -* Link phase. `ocamlc -o a.out -lib MYLIB`. The repeatable and ordered - `-lib MYLIB` resolves `MYLIB` in `OCAMLPATH` and adds its `lib.cma` - file to the link sequence. It then consults the `lib_requires` field - of `lib.cma`, resolve these names in `OCAMLPATH` to their `lib.cma` - file and add them to the link sequence and recursively in stable - dependency order. Errors if any library name resolution fails. -* Link phase. Direct `ar.cma` arguments. The `lib_requires` field - is consulted and the names are resolved as in the preceeding point. -* Link phase. `-noautoliblink` can be specified to prevent the linking - of libraries specified in the `lib_requires` of the `.cma` files - considered for linking (either as direct argument or found via `-lib`). -* Link phase. If `-linkall` is specified on the cli, the name of libraries - specified via `-lib` and those recursively resolved is embedded in - the executable (for `Dynlink` API support). +### Archive creation + +A repeatable and *ordered* `-require LIB` option with `LIB` a library name +is added to archive creation mode. For example: +``` +ocamlc -a -o ar.cma -require LIB +``` +This adds the library name `LIB` to the library dependencies of +archive `ar.cma`. The information is stored in a new field +`lib_requires : string list` of the `cma` +[library descriptor][cma-lib-descriptor]. + +`LIB` does not need to exist in the current `OCAMLPATH` when the command +is invoked. [cma-lib-descriptor]: https://github.com/ocaml/ocaml/blob/e18564d8393475acd9e36f02fe3b0927fe8a0f5c/file_formats/cmo_format.mli#L53-L58 +### Compilation phase + +A repeatable and *ordered* `-require LIB` option with `LIB` a library +name is added to compilation phase. For example: + +``` +ocamlc -c -I bla -require LIB src.ml +``` +This resolves `LIB` in the `OCAMLPATH` and adds its library directory +to the includes after `-I bla`. Effectively `-require LIB` is replaced +by `-I /path/to/LIB`. + +Errors if `LIB` does not resolve to an existing library. Note that +type checking does not need to access library archives. + +### Link phase + +A repeatable and *ordered* `-require LIB|PATH.cma` option is +added to the link phase. For example: + +``` +ocamlc -o a.out -require ARG src.ml +``` + +If `ARG` is: + +1. A library name `LIB`. The library is resolved in `OCAMLPATH` and its archive + file `lib.cma` is added to the link sequence. The `lib_requires` of + `lib.cma` is read and these names are resolved in `OCAMLPATH` to their + `lib.cma` archive, added to the link sequence and recursively in + stable dependency order. Errors if any library name resolution fails. +2. A direct path to an archive `PATH/ar.cma` file. The archive is added to + the link sequence. The `lib_requires` of `ar.cma` is read and + the dependencies resolved as is done in the previous point. + +A repeateable and unordered `-assume-library LIB` option is added to +the link phase. `LIB` must be a library name and it does not need to +exist in the current `OCAMLPATH` when the command is invoked. Using +this flag requires library name `LIB` and assumes it was already +resolved. The user is in charge in providing its functionality on +the cli as bare objects. + +### Dynlink API support on `-linkall` + +Support is provided to record which libraries are fully statically +linked in bytecode executables. + +This happens whenever `-linkall` flag is specified. In this case the name +of any library resolved in `OCAMLPATH` via `-require LIB|PATH.cma` and +those specified via `-assume-library LIB` is embedded in the bytecode +executable in a new section called `LIBS`. + ## `ocamlopt` support The following behaviours are added to `ocamlopt` -* Static archive creation. `ocamlopt -a -o ar.cmxa -lib-require - MYLIB`. The repeatable and ordered `-lib-require MYLIB` option adds - the library name `MYLIB` to the library dependencies of - `ar.cmxa`. This stores the information in a new field - `lib_requires : string list` of the `cmxa` [library - descriptor][cmxa-lib-descriptor]. `MYLIB` does not need to exist in - the current `OCAMLPATH`. -* Shared archive creation. `ocamlopt -shared -a -o ar.cmxs - -lib-require MYLIB`. The repeatable and ordered `-lib-require - MYLIB` option adds the library name `MYLIB` to the library - dependencies of `ar.cmxs`. This store the information in a - new field `dynu_requires : string list` of the `cmxs` - [plugin descriptor][cmxs-plugin-descriptor]. - `MYLIB` does not need to exist in the current `OCAMLPATH`. If - the `cmxs` is produced from a `.cmxa` then we simply transfer - the latter's `lib_requires` field to the `dynu_requires` field - unless `-noautoliblink` is mentioned on the cli. -* Compilation phase. `ocamlopt -c -lib MYLIB src.ml`. The repeatable and - ordered `-lib MYLIB` option resolves `MYLIB` in `OCAMLPATH` and adds - its library directory to includes. Errors if `MYLIB` does not - resolve to an existing library. Type checking does not need - to access library archives. -* Link phase. `ocamlopt -o a.out -lib MYLIB`. The repeatable and - ordered `-lib MYLIB` option resolves `MYLIB` in `OCAMLPATH` and adds its - `lib.cmxa` file to the link sequence. It then consults the - `lib_requires` field of `lib.cmxa`, resolve these names in - `OCAMLPATH` to their `lib.cmxa` file and add them to the link - sequence and recursively in stable dependency order. Errors if any - library name resolution fails. -* Link phase. Direct `ar.cmxa` arguments. The `lib_requires` field - is consulted and the names are resolved as in the preceeding point. -* Link phase. `-noautoliblink` can be specified to prevent the linking - of libraries specified in the `lib_requires` of the `.cmxa` files - considered for linking. -* Link phase. If `-linkall` is specified on the cli, the name of - libraries specified via `-lib` and those recursively resolved is embedded in - the executable (for `Dynlink` API support). - +### Static archive creation + +A repeatable and *ordered* `-require LIB` option with `LIB` a library name +is added to archive creation mode. For example: +``` +ocamlopt -a -o ar.cmxa -require LIB +``` +This adds the library name `LIB` to the library dependencies of +archive `ar.cmxa`. The information is stored in a new field +`lib_requires : string list` of the `cmxa` +[library descriptor][cmxa-lib-descriptor]. + +`LIB` does not need to exist in the current `OCAMLPATH` when the command +is invoked. + [cmxa-lib-descriptor]: https://github.com/ocaml/ocaml/blob/8947a38b61c9de1f95bcd2f5ec8292987211bf4b/file_formats/cmx_format.mli#L53-L56 + +### Shared archive creation (plugins) + +A repeatable and *ordered* `-require LIB|PATH.cmxa` option with `LIB` a library +name is added to shared archive creation mode. For example: +``` +ocamlopt -shared -a -o ar.cmxs -require LIB +``` + +If `ARG` is: + +1. A library name `LIB`, this adds the library name `LIB` to the library + dependencies of the archive `ar.cmxs`. This information is stored in + a new field `dynu_requires : string list` of the `cmxs` + [plugin descriptor][cmxs-plugin-descriptor]. `LIB` does not need to + exist in the current `OCAMLPATH`. +2. A direct path to an archive `PATH/ar.cmxa` file. The archive + is added to the link sequence *and* its `lib_requires` field is + added to to the `dynu_requires` field of the `cmxs` plugin + descriptor. + [cmxs-plugin-descriptor]: https://github.com/ocaml/ocaml/blob/8947a38b61c9de1f95bcd2f5ec8292987211bf4b/file_formats/cmxs_format.mli#L32-L35 +### Compilation phase + +A repeatable and *ordered* `-require LIB` option with `LIB` a library +name is added to compilation phase. For example: + +``` +ocamlopt -c -I bla -require LIB src.ml +``` + +This resolves `LIB` in the `OCAMLPATH` and adds its library directory +to the includes after `-I bla`. Effectively `-require LIB` is replaced +by `-I /path/to/LIB`. + +Errors if `LIB` does not resolve to an existing library. Note that +type checking does not need to access library archives. + +### Link phase + +A repeateable and *ordered* `-require LIB|PATH.cma` option is +added to the link phase. For example: + +``` +ocamlopt -o a.out -require ARG src.ml +``` + +If `ARG` is: + +1. A library name `LIB`. The library is resolved in `OCAMLPATH` and its archive + file `lib.cmxa` is added to the link sequence. The `lib_requires` of + `lib.cmxa` is read and these names are resolved in `OCAMLPATH` to their + `lib.cmxa` archive, added to the link sequence and recursively in + stable dependency order. Errors if any library name resolution fails. +2. A direct path to an archive `PATH/ar.cmxa` file. The archive is added to + the link sequence. The `lib_requires` of `ar.cmxa` is read and + the dependencies resolved as is done in the previous point. + +A repeateable and unordered `-assume-library LIB` option is added to +the link phase. `LIB` must be a library name and it does not need to +exist in the current `OCAMLPATH` when the command is invoked. Using +this flag requires library name `LIB` and assumes it was already +resolved. The user is in charge in providing its functionality on the +cli as bare objects. + +### Dynlink API support on `-linkall` + +Support is provided to record which libraries are fully statically +linked in bytecode executables. + +This happens whenever the `-linkall` flag is specified. In this case +the name of any library resolved in `OCAMLPATH` via `-require LIB|PATH.cma` +and those specified via `-assume-library LIB` is embedded in the native +code exectuable in a new `caml_imported_libs` symbol. + ## `ocamlobjinfo` support The following behaviours are added to `ocamlobjinfo`. -* The regular `ocamlobjinfo` output on `cmxa` and `cma` is extended - according to the current format to output the new `lib_requires` field. -* ~~`ocamlobjinfo` is made to work on `cmxs` by using the support provided - by `Dynlink` (if available) to output the plugin information rather - than relying on the BDF library (see POC [here][cmxs-read-poc])~~. - That approach is too fragile w.r.t. C bits that may live in the - cmxs see discussion [here](https://github.com/ocaml/ocaml/issues/9306). -* A new `-lib-requires` flag is added that extracts the new `lib_requires` - from `cma` and `cmxa` and `dyn_requires` from `cmxs` files. According - the following line based format: - ``` - File: PATH - LIBREQ0 - LIBREQ1 - LIBREQ2 - ... - ``` - -~~Note this makes `ocamlobjinfo` depend on the C `caml_natdynlink_open` -primitive investigate if the function is available with an error when -natdynlink is not available or if we need some kind of conditional -compilation.~~ - -[cmxs-read-poc]: https://gist.github.com/dbuenzli/0b773f4a4dd9f7a35f30cd9b671e48c5#file-readmeta-ml-L23-L31 +* The regular `ocamlobjinfo` output on `cma`, `cmxa` and `cmxs` (if + BDF library is available) is extended according to the current + format to output the new `lib_requires` and `dynu_requires` field.s +* The output on bytecode executable is extended to report the new + `LIBS` section in an "Imported Libraries" section. +* A new option `-requires` is added that only prints libraries required + by `cma`, `cmxa` and `cmxs` objects, one per line. ## `ocamldep`, `ocamldebug`, `ocamlmktop` and `ocamlmklib` support The following behaviours are added to `ocamldep` and `ocamldebug` -* `(ocamldep |ocamldebug) -lib MYLIB`. The repeatable and ordered - `-lib MYLIB` option resolves `MYLIB` in `OCAMLPATH` and adds its - library directory to includes. Errors if `MYLIB` does not resolve to +* `(ocamldep |ocamldebug) -require LIB`. The repeatable and ordered + `-require LIB` option resolves `LIB` in `OCAMLPATH` and adds its + library directory to includes. Errors if `LIB` does not resolve to an existing library. The following behaviours are added to `ocamlmktop` -* The repeatable and ordered `-lib MYLIB` option resolves and links - `MYLIB`'s dependencies as per `ocamlc` support. So goes for direct +* The repeatable and ordered `-require LIB` option resolves and links + `LIB`'s dependencies as per `ocamlc` support. So goes for direct `cma` arguments. The following behaviours are added to `ocamlmklib` -* The repeateable and ordered `-lib-require LIB` argument is added +* The repeateable and ordered `-require LIB` argument is added and propogated to the resulting `cma`, `cmxa` and `cmxs` files. ## `ocaml` and `ocamlnat` support The following behaviours are added to `ocaml` and `ocamlnat`. -* The repeatable and ordered `-lib MYLIB` option resolve - `MYLIB` in `OCAMLPATH` adds its directory to includes and - loads the library and its dependencies according to `cma` linking - (for `ocaml`, see `ocamlc` support) or to `cmxs` linking - (for `ocamlnat`, see `ocamlopt` support, though that happens with - the info in `cmxs` files). Note that directories of dependent - libraries are *not* added to the includes. - -* The new `#require "MYLIB"` directive is added to the interactive toplevel. - This directive resolves `MYLIB` in `OCAMLPATH` and adds its library - directory to includes. It then loads the library and its - dependencies as per the preceeding point. Here again directories of - dependent libraries are *not* added to includes. - -* The `#load` directive is extended to load libraries along the lines of - `#require`. - -A nice side effect of the new `#require` is that it doesn't mention -file extensions, this allows to use it in `.ocamlinit` and have it -work both for `ocaml` and `ocamlnat`. +### `-require` option + +The repeatable and ordered `-require LIB|PATH.cm(a|xs)` is added +to `ocaml` and `ocamlnat`. For example: + +``` +ocaml -require ARG +``` + +If `ARG` is: + +1. A library name `LIB`. The library is resolved in `OCAMLPATH`, this + adds its directory to includes and loads the library archive and its + dependencies according to `cma` linking (for `ocaml`, see `ocamlc` support) + or to `cmxa` linking (for `ocamlnat` see `ocamlopt` support). Note that + directories of dependent libraries are *not* added to the includes. +2. A direct path to an archive `PATH/ar.cma` file. `PATH` is added to the + include directories and `ar.cma` and its dependencies are loaded like + in the preceeding point. + +Note that in the above if any of the names resolved in the `OCAMLPATH` is +already embedded in the `ocaml` or `ocamlnat` executable, the names are +not resolved (see `ocamlc` and `ocamlopt` Dynlink API support). + +### `#require` directive + +A new `#require "ARG"` directive is added to the toplevel. If `ARG` is + +1. A library name `LIB`. The library is resolved in `OCAMLPATH`, its + directory added to the includes, its library archive and its + dependencies are loaded according to `cma` linking + (for `ocaml`, see `ocamlc` support) or to `cmxa` linking + (for `ocamlnat`, see `ocamlopt` support)). Note that directories + of dependent libraries are *not* added to the includes. +2. A direct path to an archive `PATH/ar.cma` file. `PATH` is added + to the include directories. + +Note that in the above if any of the names resolved in the `OCAMLPATH` is +already embedded in the `ocaml` or `ocamlnat` executable, the names are +not resolved (see `ocamlc` and `ocamlopt` Dynlink API support). + +A nice side effect of the new `#require` is that if used with library +names, it doesn't mention file extensions, this allows to use it in +`.ocamlinit` and have it work both for `ocaml` and `ocamlnat`. We will +also make sure to make `ocamlnat` use `Dynlink.adapt_filename` when file +paths are specified to `#require` so that a mention of `cma` get turns into +a mention of a `cmxs`. + ## `Dynlink` library support -The following entry point is added: +We add the following entry points: + +``` +Dynlink.require : ocamlpath:string list -> string -> unit +Dynlink.assume_library : string -> unit +``` +In `Dynlink.require ~ocamlpath arg` if `arg` is: + +1. A library name `LIB`, then `LIB` is looked up in the `OCAMLPATH`, its + library archive gets loaded and its recursive dependencies in the correct + order. +2. An archive `PATH/ar.cm(a|xs)`, then the archive and its recursive + dependencies in the correct order. + +In both cases if any resolution requested in the `OCAMLPATH` is a library +name that already exists in the executable (see `ocamlc` and `ocamlopt` Dynlink +API support), the requested name is not looked up and assumed to be already +loaded. + +A call to `Dynlink.assume_library n` add library name `n` to the set of +loaded libraries names without loading anything. + +We also provide the following two functions to let client handle their +own `OCAMLPATH`-like variable. ``` -Dynlink.loadlib : ?ocamlpath:string list -> string -> unit +val Dynlink.default_ocamlpath : string list +(** [default_ocamlpath] is the value of the ocamlpath that has been + set at OCaml configuration time. *) + +val Dynlink.ocamlpath_of_string : string -> string list +(** [ocamlpath_of_string s] parses [s] into a list of directories by + following the OS convention for [PATH]-like variables. This means they + are colon ':' (semi-colon ';' if {!Sys.win32}) separated paths. Empty + paths are allowed and discarded. *) ``` -In byte code programs if a `cma` is loaded its `lib_requires` field -(see `ocamlc -a` support) is consulted and the library names are -resolved. First we check if those exist in the executable itself -(cf. linking with `-linkall` above), if that happens nothing needs to -be done. Otherwise the library is resolved to a `cma` file to load -according to the directories mentioned in `ocamlpath` (defaults to the -contents of `OCAMLPATH`) and recursively in dependency order. +Mirroring the existing support for compilation units, we also add the following +functions to query the state of the program w.r.t. to loaded libraries. -In native code the same happens with `cmxs` files via the new -`dynu_requires` field (see `ocamlopt -shared` support). +``` +val Dynlink.main_program_libraries : unit -> string list +(** [main_program_libraries ()] is the list of library names statically linked + in the program. *) + +val Dynlink.public_dynamically_loaded_libraries : unit -> string list +(** [public_dynamically_loaded_libraries ()] is the list of library names + that were dynamically loaded via {!require}. *) + +val Dynlink.all_libraries : unit -> string list +(** [all_libraries ()] is the union of {!main_program_libraries} and + {!public_dynamically_loaded_libraries}. *) + +val Dynlink.has_library : string -> bool +(** [Dynlink.has_library l] is [List.mem l (Dynlink.all_libraries ())]. *) +``` Keeping track of library names present in the executable and those -loaded by calls to `Dynlink.loadlib` will be a matter of extending the +loaded by calls to `Dynlink.require` is a matter of extending the existing Dynlink API's [state structure][dynlink-state] which currently keeps track of these things at the compilation unit level. -*Note* it might be possible to meld this into the existing entry -points of the `Dynlink` module rather than add a new entry point but -there are a few things to consider w.r.t. access control. - [dynlink-state]: https://github.com/ocaml/ocaml/blob/03c33f500563f3e12355694f1add98e7bd1096ae/otherlibs/dynlink/dynlink_common.ml#L45-L63 ## OCaml library install support @@ -430,7 +632,7 @@ as if the following `R/foo/META` file exists: ``` package "bar" ( -requires = ... # contents of `ocamlobjinfo -lib-requires R/foo/bar/lib.cma` +requires = ... # contents of `ocamlobjinfo's require of R/foo/bar/lib.cma` directory = "bar" archive(byte) = "lib.cma" archive(native) = "lib.cmxa" @@ -446,8 +648,8 @@ Since `ocamlfind` is in charge to put the recursive archive dependencies on the compiler cli at the link phase and that there may be a mix of archive using `META` files and others using the new mecanism it must take care to resolve the dependencies of the latter -according to the `OCAMLPATH` semantics and specify `-noautoliblink` so -that no double linking occurs. +according to the `OCAMLPATH` semantics and specify them as bare archives +on the command line so that no double linking occurs. ## `dune` support @@ -457,7 +659,7 @@ installs. Once the compiler supports this proposal, Dune will start installing library artifacts using the convention described in this document. It -will also pass the `-lib-require` field when assembling library +will also pass the `-require` field when assembling library archives. This will be a non-breaking change since from the point of view of users the naming of artifacts is a implementation detail of Dune. Dune will only enforce the new convention for the libraries it @@ -473,6 +675,53 @@ code base. Starting from Dune 4, Dune will no longer generate `META` files. +## Examples + +### Compiling and using libraries locally in a build + +The following shows how to compile and layout two libraries +to use locally in a build. We have: + +* Library `a` made of source `a.ml` depending on the `ocaml.str` + library. +* Library `b` made of source `b.ml` depending on the library `a`. +* An executable `exec.ml` which depends on `b` + +Here are our sources: + +``` +. +├── a.ml +├── b.ml +└── exec.ml +``` +We create directories for the libraries. The `libs` directory +will be added to the `OCAMLPATH`. + +``` +mkdir -p libs libs/a libs/b +``` + +We compile and layout library `a`: + +``` +ocamlopt -require ocaml.str -c -o libs/a/a.cmx a.ml +ocamlopt -require ocaml.str -a -o libs/a/lib.cmxa libs/a/a.cmx +``` + +We compile and layout library `b`. We extend the OCAMLPATH on the +command line via the `-L` option. + +``` +ocamlopt -L libs -require a -c -o libs/b/b.cmx b.ml +ocamlopt -L libs -require a -a -o libs/b/lib.cmxa libs/b/b.cmx +``` + +We compile our executable: +``` +ocamlopt -L libs -require b exec.ml +``` + ## Unresolved issues ### Compilation phase includes @@ -506,18 +755,125 @@ problem with 2. is that it breaks the idea that the compilation phase need not be aware of the dependencies of a library which are currently only stored in library archives. - ## Implementation -Work on the proposal has started thanks to a grant of the +Work on the proposal is ongoing thanks to a grant of the [OCaml software foundation](http://ocaml-sf.org/). ### OCaml status +An OCaml implementation of the RFC is available and can be used with: + +``` +opam switch create ocaml-ll --empty +opam pin add ocaml-variants.4.12.0+ll git+https://github.com/dbuenzli/ocaml#ll-v2 +``` + +For interested reviewers, +[the patches](https://github.com/ocaml/ocaml/compare/trunk...dbuenzli:ll-v2) +are meant to be read individually and in sequence. + +A draft for a manual section about the library convention can be found +[here](https://github.com/dbuenzli/ocaml-lltest/blob/master/manual.mdy). + +A bit of testing can be found in [this +repository](https://github.com/dbuenzli/ocaml-lltest/). Feel free to +open issues you may find there. + +The following notes indicate what is *not* implemented according the +RFC, a few known problems and work that remains to be done. + +1. RFC clarification. At the moment there are not provisions to dynamically + tweak the OCAMLPATH in the toplevel (like the `#directory` has for `-I`). + Should we have something ? +2. RFC clarification. The current implementation adds the directory of any + archive as a `-L` to the C linker (see point 7 of the previous iteration) + in `{Byte,Asm}link` this is not mentioned in the RFC above. It should be, + along with the rationale. +3. RFC clarification. Some of the library directory + [constraints](#semantics-and-integrity-constraints) + should be discussed (6 and 7). Namely uniformity of dependencies + and implementation in different backend archives. Maybe we could relax + that, I think that uniformity of *API* (i.e. implementation with a .cmi file) + across backends is important, not implementations and/or dependencies. +4. RFC clarification. Support for the + [default `OCAMLPATH` value](#support-for-ocamlpath-default-value) value + should be discussed, notably with system packagers. +5. Implementation bug. The current implementation triggers infinite loops in + case of recursively dependent libraries in `Dynlink.requires` and + `#require`. It is not difficult to fix; in contrast to `Asmlink` and + `Bytelink`, we only update the seen libraries after we are sure it + successfully loaded. We need to carry a separate seen libraries for + the current "require" load. +6. Implementation enhancement. The current implementation is inconsistant + about using `lib[s]` and `librar{y,ies}` this should be uniformized. For + now public identifiers consistently use the latter, we should decide on one + (e.g. `-assume-lib` or `-assume-library`?). +7. Implementation enhancement. Now that we have @nojb's `Binutils`, + `ocamlobjinfo` on native code executables could report the data of + `caml_imported_libs` and `caml_globals_map`. Like `ocamlobjinfo` + does with `LIBS` on bytecode executables (we get a bit less clear + error checking). +8. Implementation RFC deviation. Requires and ordering. The RFC above has it + that the relative order + of `-require`, `-I` and positional arguments should be respected + modulo `-require` dependency sorting at link time. No good way was + found to do this in the code base without making it worse than it + already is. In the current implementation `-require` include effects + are put after all `-I` arguments and before positional + arguments. Incidentally this is what `ocamlfind` does. However for + positional arguments it doesn't work for 1) the idea of using + `-require FILE` to request resolution of `FILE` dependencies 2) + for using `-assume-library a implements_a.cma + -require b` for replacing the dependency `a` of `b` since the object + resolved by `-require b` gets placed before `implements_a.cma` and + link fails. Note that this is a cli user interface problem, the actual + linker implementation supports the interleaving correctly. +9. Implementation RFC deviation. Archive creation, transfer of + `lib_requires` information when an archive + is created (`-a`) from another one (a cma from cmas, cmxs from cmxas). + This is still a bit unclear in the RFC above. It was not spelled out but + the idea was that you'd only transfer if `--require FILE` was put on + the `-a` invocation (previously we had the `-noautoliblink` to prevent + transfer). But the preceeding point makes that problematic. For now we + unconditionally transfer `lib_requires` of the arguments to the new + archive (because most build system out there only use this workflow + for creating a `.cmxs` matching a `.cmxa`). Depending on what happens + with the preceding point we still may want a mecanism to prevent + the transfer. (e.g. `-assume-library` could be used to remove + specific mentions in the final result). +10. Implementation TODO. Manpages must be updated. +11. Implementation TODO. Beyond this [new manual section](https://github.com/dbuenzli/ocaml-lltest/blob/master/manual.md), the narrative about libraries should be integrated in manual sections about individual tools. +12. Implementation TODO. Test suite. The tests + [here](https://github.com/dbuenzli/ocaml-lltest) should be integrated + into the compiler test suite, as cram tests. (So that e.g. *stable* topo + sorts are tested). +13. Implementation TODO. There is no support for `OCAMLPARAM`. Should there be + one ? + +## Old supporting work + +### OCAMLPATH + +There is a [PR](https://github.com/ocaml/ocaml/pull/8946) for making +`OCAMLPATH` meaningful to the OCaml toolchain. For now its only effect +is to redefine the `-I +DIR` notation so that distributions can start +reshuffling their install structure to make it easier to extend a +system OCaml package install prefix with an opam package install +prefix. + + +### OCaml status RFC v1 + +These are the note the for the [previous +iteration](https://github.com/ocaml/RFCs/pull/7) of the RFC. + Partial implementation for the compiler support is available -[here](https://github.com/ocaml/ocaml/compare/trunk...dbuenzli:ll). For -interested reviewers, the patches are meant to be read individually and in -sequence. +[here](https://github.com/ocaml/ocaml/compare/trunk...dbuenzli:ll) + + +For interested reviewers, the patches are meant to be read +individually and in sequence. To create an opam switch with a compiler that has that support in you can use: ``` @@ -634,21 +990,21 @@ was proposed above. code executables is done. Respectively by adding a new `LIBS` section in the byte code and a new `caml_imported_libs` symbol in native code with the set of library names that were fully linked - via `-require` and recursively. An `-assume-require LIB` was also + via `-require` and recursively. An `-assume-library LIB` was also added that allows to simply add `LIB` to the set of libraries that are supposed to be embedded in the executable. This is useful for handling `-linkall` with uninstalled libraries or for build systems that perform lookups and put library archives themselves on the cli (cf. the `-noautoliblink` flag). -11. In theory `-assume-require a` (see 10.) and `-require b` could +11. In theory `-assume-library a` (see 10.) and `-require b` could always be used together in particular if library `b` requires `a`, it won't be looked up. In practice however due to point 9. problems will arise at link time since the archive you specify for `a` on the cli will come after `b`'s one and lead to a link error. So at the moment having dependencies from `-require` - to `assume-require` libraries is not really supported but that is + to `assume-library` libraries is not really supported but that is not usually the case. If that happens though you should translate - all `-require` to `-assume-require`, do all the lookups and + all `-require` to `-assume-library`, do all the lookups and sorting yourself and use `-noautoliblink`. 12. Dynlink API: support for library loading was added. Works only in bytecode for now. Both `loadfile` and `loadfile_private` take a new @@ -688,15 +1044,6 @@ was proposed above. good idea or not. 15. This implementation has no support for `OCAMLPARAM`. -## Old supporting work - -### OCAMLPATH -There is a [PR](https://github.com/ocaml/ocaml/pull/8946) for making -`OCAMLPATH` meaningful to the OCaml toolchain. For now its only effect -is to redefine the `-I +DIR` notation so that distributions can start -reshuffling their install structure to make it easier to extend a -system OCaml package install prefix with an opam package install -prefix.