opam support for third-party tools tracking per-package data?

[gasche]

In his talk at the OCaml Workshop 2026, @jonludlam argued that packages should install their odoc-generated intermediate files by default, to simplify building the odoc documentation (which cross-references the documentation of other libraries the package depends on). Systematically installing the .odoc files on each package takes space (and a bit of install time), but it avoids forcing odoc to duplicate package-manager logic to (1) locate the dependencies of the package and (2) figuring out how/where to store the documentation files. (In particular, only the package's install script is allowed to write in the install directory of the package, so we cannot easily populate the .odoc files on-demand after installation.)

I had a discussion with @Octachron to understand whether other tools could be improved to make this systematic installation un-necessary (according to my (limited) understanding of the needs here). Our non-expert conclusion is that it would suffice for opam to provide the two following support features:

1. The ability to ask opam, for a package installed in the current switch, how it satisfied its dependency requirements: which dependencies it installed, at which version. (Maybe this is already available?):

   "opam, please show me the actual dependencies you installed for <name> in this switch"

2. The ability to use opam as a key-value store to store metadata for packages installed in the current switch. A package <name>.<version> has metadata keys that can be set and read by arbitrary tools (and anyone can set them, unlike the install files that can only be populated by the package).

• "opam, what is the payload attache to the metadata key odoc for the package <name>.<version>?"
• "opam, please associate to the metadata key odoc for <name>.<version> the string '~/.opam/5.5.0/share/odoc/<name>/<name>.odoc'"

Say you ask odoc for the documentation of containers, it would look at whether the containers.<version> package in the current switch already has metadata attached to the odoc key:

• if yes: this is the path to the .odoc files, stored somewhere in a directory odoc controls);
• if not: odoc queries opam to find the exact dependencies installed in the switch, recursively finds their .odoc files, builds the documentation for containers.<version>, and stores build an .odoc file for containers, stores it somewhere, and sets the odoc key for the containers.<version> package.

Our hope is that this would allow odoc to build (and store) documentation for arbitrary opam packages of the current switch on-demand. When a package is removed, the corresponding metadata is also removed. In particular, after package updates, the odoc metadata will be rebuilt (on-demand on the next request for documentation). (In contrast, without those metadata for each <name>.<version> package -- if odoc simply stored the doc in share/odoc/<name> and looked it up there -- there would be no easy way to tell that there is a version mismatch after an opam update.)

I have the impression that this approach would let odoc track per-installed-package data without coupling with the build-system or package-management layers -- and this approach extends to other tools (Merlin indices, etc.). It supports on-demand construction of per-package information (reducing worries about disk usage impact), and avoids the "bootstrapping" problem mentioned in Jon's talk: if each package builds and installs its documentation, how can the dependencies of odoc do it if the tool is not yet installed?

[dbuenzli]

The problem is that you have now made ocaml package documentation a property of the opam package manager and basically have coupled odoc and opam. This scheme cannot work if you assume that packages are possibly managed by a different package manager (e.g. your system package manager).

[kit-ty-kate]

In his talk at the OCaml Workshop 2026, @jonludlam argued that packages should install their odoc-generated intermediate files by default, to simplify building the odoc documentation (which cross-references the documentation of other libraries the package depends on).

Thanks for pointing it out. I would've seen it but sadly the experience for remote viewers was rather frustrating so i skipped the last talk.

1. The ability to ask opam, for a package installed in the current switch, how it satisfied its dependency requirements: which dependencies it installed, at which version. (Maybe this is already available?):
   "opam, please show me the actual dependencies you installed for <name> in this switch"

yes, this is opam tree or opam list -s --all-versions --installed --recursive --required-by (depends if you want it to be human readable or machine readable)

Systematically installing the .odoc files on each package takes space (and a bit of install time), but it avoids forcing odoc to duplicate package-manager logic to (1) locate the dependencies of the package and (2) figuring out how/where to store the documentation files. (In particular, only the package's install script is allowed to write in the install directory of the package, so we cannot easily populate the .odoc files on-demand after installation.)

2. The ability to use opam as a key-value store to store metadata for packages installed in the current switch. A package <name>.<version> has metadata keys that can be set and read by arbitrary tools (and anyone can set them, unlike the install files that can only be populated by the package).

• "opam, what is the payload attache to the metadata key odoc for the package <name>.<version>?"

• "opam, please associate to the metadata key odoc for <name>.<version> the string '~/.opam/5.5.0/share/odoc/<name>/<name>.odoc'"

In theory you could do that by adding files to the added field of <switch>/.opam-switch/install/<name>.changes, this would remove the file with the rest of package as if it was installed by it in the first place (including reinstalls, which i think is the behaviour that you want). Ideally you'd use the opam-format API to access it and the OpamFile.Changes module is the one dealing with this type of file. On our side however this becomes slightly harder to change the internal format but as long as failures to parse are properly handled on your side so that no corrupted files are installed (atomic writes, maybe grabbing the opam lock, no recovery on parse failures) this should be ok to do.

---

However taking a step back, i concur with @dbuenzli. It might be too much of special-case idea because you'd need a fallback non-opam-garbage-collected local database anyway, so at this point why not do that without the opam special case and improve the side database approach in the first place (e.g. by garbage collecting packages that are not available anymore whenever odoc is called)

[dbuenzli]

To me this looks like trying to reinvent with-doc in ad-hoc fashion that will only bring problems for integrating OCaml packages in other systems.

Here is a simple scheme:

1. By convention packages generate and install .odoc files only if with-doc is specified.
2. If odoc can't file the .odoc for a package it might invoke an external tool to do something. In an opam context that may just be reinstalling the package with --with-doc.

[kit-ty-kate]

I'm not sure. I think --with-doc is ill-suited for this use-case. First of all almost nobody uses it currently and would at least require dune to do anything with the @install @doc target combo (currently this won't install the documentation).
But also --with-doc was designed long ago to have more of an apt install <name> <name>-doc behaviour, which is to say: "install the package i'm requesting currently and its documentation". To be useful for odoc it would require the option to be global (per switch or at least true for the package and its dependencies instead of just for the requested package)

EDIT: after some history digging, in opam >= 0.9 < 2.0 --with-doc was set for all packages installed within one opam call (ca87999), after 2.0 it was changed to what i'm describing above (feaad77)

[dbuenzli]

But also --with-doc was designed long ago to have more of an apt install <name> <name>-doc behaviour, which is to say:

Which is exactly what you should be aiming for rather than invent new schemes that are only supported by one package manager.

How it's used now is irrelevant. New conventions have to be established anyways.

[kit-ty-kate]

I guess we could allow overriding the with-doc value using opam var --global with-doc=true. Currently this doesn't do anything. Although we'd also have to introduce --without-doc and the like (although we can just say "please unset the global variable before installing that package instead"). This would also go in opposite direction to my long term extended package variable feature wish and we'd have to figure out early how global variables and package variables should interact with these special package variables (with-test, with-doc, …).

All this to say: i'm not sure we have the capacity/woman-hour available currently to think about these tricky design decisions on top of the other important things to take care of in opam, and i don't think y'all would want to wait a couple of years for us + for the majority of packages/build-systems in opam-repository to change how they handle with-doc, to be able to take advantage of that type of feature. But if this can wait, then sure. @jonludlam what do you think?

[gasche]

I get the point that assuming opam is not necessarily a good approach for OCaml tooling. (For example maybe they want to work well in a opam-less nix world, or with Dune package management, etc.). @dbuenzli, do you think that your proposal of adding more metadata to archive files could possibly be extended to support the need of odoc-class tools in a way similar to what I sketched above?

(For example: maybe .cma packages could also contain a hash of their dependencies, and then it would be easy for odoc to tell that the package or one of its dependencies changed since the last doc build.)

[dbuenzli]

(For example: maybe .cma packages could also contain a hash of their dependencies, and then it would be easy for odoc to tell that the package or one of its dependencies changed since the last doc build.)

I don't think that really the problem. I mean odig simply hashes compilation objects like any sane build system technology does nowadays and is very fast with modern hashing technology like xxhash.

I didn't see the presentation or slides so I'm not exactly sure what we are talking about. But based on the feedback he asked me for I think it's rather that when you build docs in your local project you end up creating the .odoc files for your dependencies (and so does odig too in $(opam var prefix)/var/cache/odig/odoc). To avoid that (and solve a bunch of other problems I don't remember), we would rather like to have packages themselves do that and install them but then there is the problem of size.

I think it would be better to have a look at the larger picture here. OCaml has had a metadata proliferation and size for quite some time now. With instructions on what should be installed growing organically without any sort of design – suddenly you should also install sources files too. This issue tracks that ocaml/ocaml#14154 upstream

I think the answer here would be to rather sit at a table to clarify and design something sensitive about what an OCaml package install should look like (here is a starting point :-). Do we really need to have .cmti, .cmt, .odoc, .ml, .mli, .cmi files ? In how many of these files are the doc strings of structure items actually replicated ? What's going to be next ? What about consolidating this in less files and apply modern compression technology to it ?

I think the way to inform that design would be to start from this end result and going backwards: at the package install level you just want two install profiles:

1. dev installs all the metadata you can dream of for development.
2. deploy has only the metadata needed for runtime debugging.

The second one keeping your switch smalls for the installs on the machine on which you build your program for deployment (which can coincide with the machine you deploy and may not have TB of space…, has been an issue for me).

The only thing you need for this in opam is a new switch variable that you can specify at switch creation time that indicates the install profile and pass it to the build systems in the package build/install instructions (in the same way vein that we have these per package with-doc, with-test, with-dev-setup variables).

[gasche]

I think the approach you suggest could work but it is not open-ended: each new tool that wants to track per-package information has to start with hacks on the side, and then maybe if they find wide adoption they would end up integrated in a set of blessed install metadata. I wonder if it is possible to let tools own their per-package metadata, in an easy way, so that there is no capability or ease-of-implementation cliff for new tools.

[dbuenzli]

Rather than work on hypotheticals, what about first solving the problems we have now, with the tools we have now in order to provide users with a good experience? I know, it's less fun, I tried for ages to convince people to clean up OCaml's library install and link model (and to be clear: won't anymore) but that's what is needed.

Besides:

each new tool that wants to track per-package information has to start with hacks on the side,

Honestly I don't see that as a big deal, as odig's implementation shows you can just generate your own metadata on demand in $(PREFIX)/var/cache/mytool. In fact odoc, if it wanted, could even use the .odoc files from there (I'm not suggesting that though :–).

A little bit more infra could be provided by a package manager to make it easier to track changes in the package install set to smooth out the experience (mainly to help pruning data of uninstalled packages) but honestly it's not rocket science and if your metadata needs to track dependencies among objects installed by packages you can use a build system library like odig does by using b0.memo.

[jonludlam]

Thanks for starting this discussion, @gasche!

I want to preface this by saying that I think it's really important to figure out the best way to do this, and there's a good chance that we can come up with a better solution than I've proposed in the talk.

Having said that, I'm not sure that what's being discussed here will help. Firstly, it's not so much the package manager logic that we have to reimplement, but the build system. Finding the package dependencies is a fair bit simpler than the 'within package' dependencies. The compiler itself is quite a good example - figuring out which module is in which 'compiler-libs' library and then getting the dependencies right is really tricky - For example, meta.mli (whose implementation is in ocamlcommon.cma) depends upon instruct.mli (whose implementation is in ocamlbytecomp.cma) even though ocamlbytecomp.cma depends upon ocamlcommon.cma. The point is that this sort of knowledge is known by the build system of the package already, so getting it to build the odoc files is a straightforward extension.

The important thing I want is to have a reliable shared source of odoc files for the installed libraries. Today, every tool maintains its own cache of these - odig, dune, odoc_driver, odd_driver - and each one has to rediscover the rules to build them. I think extending the build system that's already building the libraries to also do to the docs is the most reliable way to do this. I quite like @kit-ty-kate 's nod to Debian's -doc packages - building the source package will always build the docs, but they get put into a separate package that you can choose to install or not. We might have to do it slightly differently, but the point is that the docs can be part of the 'normal' build.

I'd prefer to have search indices installed too, as the alternative would have to be to do the indexing on the first attempt to search after each opam install/upgrade. This can be done either by the package itself or by a post-install hook in opam (a bit like how [odd](https://tangled.org/jon.recoil.org/odd) already works).

@dbuenzli suggested installing the odoc files next to the cmis, which I think is an excellent idea that I've happily adopted, as this makes the include flags for odoc identical to standard compilation. The downside is that it leads to the bootstrap problem if opam restricts installing to 'package owned' directories. Possible solutions I can think of would be getting odoc installed in the same package as the ocaml compiler - which could be done either by putting odoc into ocaml/ocaml (https://github.com/jonludlam/ocaml/tree/add-odoc for proof of concept), or making the ocaml-compiler package do more than just build ocaml/ocaml. Does anyone else have any suggestions?

[jonludlam]

this might be a dumb question, but if there is duplication for this, wouldn't it be appropriate to put this in a common library? I imagine the code would be a bit too different for dune, but for the rest my intuition tells me that it should probably be possible to share that code. Thus avoiding much of the issue we're trying to solve(?)

To share the rules? Or the artifacts? I'm proposing to share the artifacts, and I'm not terribly fussed about how we do it. I don't see a particular advantage in sharing the rules only - and I'm not at all sure how you would share them between dune and b0 and make and any others.

[kit-ty-kate]

I don't see a particular advantage in sharing the rules only - and I'm not at all sure how you would share them between dune and b0 and make and any others.

I was mostly thinking about the rules (basically something like simply sharing what odig already does). But i think i'm a bit lost as to what does what and needs what type of information where. (i'm a filthy .mli reader and rarely reach for html documentation, even less so generate it myself ^^")

[Octachron]

Even from your description finding the intra-package dependencies sounds pretty straightforward starting from the build artefacts.
If the the main issue is to have "a reliable shared source of odoc files" across tools, that can be defined by a standard rather than trying to enforce an implicit standard at the build system level.

[jonludlam]

Having a "reliable shared source of odoc files" is definitely the goal! Obviously it's not impossible to do this from what's currently installed - it's how the docs on ocaml.org are created, and I've got a branch of dune that builds the docs for all dependencies.

I think the experience of writing the dune rules is quite interesting. The branch I've got started by throwing out the existing doc rules in favour of making a 'generic' doc data layer that could handle anything that was installed in your switch, modelled after odoc_driver. We then added rules to build the docs of both the dependencies and the local project on top of that. The odd thing about that was that dune ended up building all the cmt/cmti files, then forgetting everything it ever knew about them, then inspecting them to rediscover all that information, then building the docs. In contrast, the doc rules that were there before already knew everything and just build the odoc files and were therefore much simpler. This is the point I'm driving at - the build system where you're building the OCaml artefacts has precisely all the information and capabilities to slot in the build of the odoc files too.

What I've built with odd is something I'd rather not have to have - similarly odig. In both cases, they have to build many many odoc files from the cmtis, and to do it efficiently both end up needing a parallel build system built in - b0 in odig's case, and an inbuilt one based on eio in odd and odoc_driver's cases. The also both have to handle all possible ways people might come up with of installing bits and pieces.

I think the real point of the talk was that if we as an OCaml community value documentation, then we should not treat it as an afterthought. Instead we should make it a core part of 'normal' builds, CI, installs and so forth - and in doing so the marginal cost of docs for new packages becomes small.

[kit-ty-kate]

should that data be added to the cmti files instead for example?

[jonludlam]

I think the answer here would be to rather sit at a table to clarify and design something sensitive about what an OCaml package install should look like (here is a starting point :-). Do we really need to have .cmti, .cmt, .odoc, .ml, .mli, .cmi files ? In how many of these files are the doc strings of structure items actually replicated ? What's going to be next ? What about consolidating this in less files and apply modern compression technology to it ?

On this front, I was hopeful that we might be able to get rid of the cmt/cmti files and replace them with cms/odoc files, but then tools like ocamlgrep turn up needing the cmt files :-)

But yes, in principle, I'm very much in favour of considering this question together, and trying to avoid duplicating quite so much information across all these files.