Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
[Feature Request] Specify exact git commit of a dependency #2762
We are developing some rather fast-moving (as in, constantly breaking backwards compatibility) research projects and would like to sue opam to express and manage dependencies. The trouble is that generally, project B will need a specific git commit of project A. I hoped I could express that by writing in B's
However, that doesn't seem to work, the dependency is just ignored. Of course, for a local package I can simulate that by first pinning manually, but I'd like to have something that can automatically install the right versions of everything, for example for a CI build. This also has to work transitively; project A may have a similarly volatile dependency on project X, so in A's
It would be great if opam could support pin-like notation in the dependency field, so that one can express a dependency on a particular git commit. (As another benefit, this would make it easy to depend on projects that are not available in any repository; one could just give their git URL. There is precedence for doing this e.g. in Rust's cargo.)
Thanks for the suggestion; indeed, managing development dependencies, and in general providing a way to share and reproduce consistent dev environments, is the last major concern that we are triving to solve before releasing opam 2.0. There are still open questions on how exactly to manage it, though, and input is welcome.
One issue with the suggested behaviour is that it supposes an open universe of packages, while opam relies on solving installation problems within a closed universe. What I mean by this is that opam supposes prior knowledge of the metadata of all packages (or at least their relationships, ie conflicts and dependencies) before attempting to solve the installation problem, and returning the best solution. The installability problem is NP-complete, and, AFAIK, package managers with a different policy don't handle conflicts or expressive dependencies.
Since dependencies, in this case, point to a single version, it could be possible to recursively fetch the metadata for all dependencies before the solver is called; this assumes we know for sure the exact initial package that we want to know the dependencies for and install, though: for example, if there are several versions of
On the bright side, this is in part what
There are generally two ways to handle development environment reproductibility: specific dependencies to URLs, as you suggest, and what npm or Cargo call "lock files".
For lock files, we have "export files" (
As for the feature you are wishing for, I think it could fit into pinning with something like a "recursive mode", that would pin a given package, and, recursively, its dependencies.
(see the current syntax)
Do you think such a feature would, in your case, fit the bill ?
Great to hear that this is a concern of the opam developers! I have to say that my first impression of opam was really great, it seemed like finally someone designed a package manager "by developers, for developers" that still is so easy to use that one can tell end-users to use it for installing stuff. I never wrote a single line of OCaml, so I hadn't heard of it before, but now the Coq community is jumping on this train.
One concern I have here is that I don't want to fix all dependencies. To extend my example above, both projectA and projectB are Coq projects, and both work with versions 8.5 and 8.6 of Coq. So I would like to have CI jobs that test this. The dependency of projectB on projectA is on a particular commit, but it should still be possible to have variations in other dependencies, like Coq.
That sounds great! Adding pins in the
In case that is interesting for you, I hacked something together that approaches this behavior in a very crude way (and relying on assumptions I can make because we control all the projects involved in the "tight" part of the dependency graph). The shell script https://gitlab.mpi-sws.org/FP/LambdaRust-coq/blob/master/build/opam-pins.sh expects on stdin an
It will then apply these pins, but also recursively look at the given commits of the given repositories and process their
EDIT: I now found something that works in way less hacky ways: When I set the pins, the version number is changed to include the commit hash. Then opam remembers it has to recompile. The link above has been updated.
It seems to me that what you suggest with the advanced dependency expressions would cover exactly what I try to cover with my hack, just obviously in a more integrated fashion and hopefully without problems related to not working "within a single opam transaction". That's a great sign :)
Not by far the best interface, but note that while there is no way to automatically generate partial "export" files, the current behaviour is to apply them on top of your existing switch state, changing as little as possible. You might be able to achieve what you are currently trying to do with
I don't really consider that a solution, at least not without tools to generate the file, but in the current state of affairs, it might help.
This is actually related to #2763: opam 2 will check for differences in the metadata, and determine if it has changed in a meaningful way (a different upstream URL or hash being one). If so, it will prompt to reinstall at the next opportunity, so I believe the next version will have the desired behaviour. (pinned packages may require reinstall without changes to their metadata, so the "reinstall" file is still there but only used to handle this case).
After some thinking, I don't think that using the same field for repository and dev opam files, but with a slightly different format, is a good idea. I still think this would be much nicer to specify without having to add an extra file, so the best option might be a new field, that is only meaningful when pinning the package (ignored — or better, disallowed — on the repository).
It could be something like:
This would be read at pinning time, and whenever projectB is updated, causing recursive pins or pin updates. Note that we probably woulnd't unpin if the constraint disappears, though.
I am glad that seems like an adequate solution to your problem.
I don't have a strong opinion on whether it's a new field or a part of the dependency. Making it a new file is obviously a band-aid. ;-) (I tried re-using the
What is the reason this cannot work in a repository? I am thinking of "dev" versions in a repository here, of course, not of stable releases. (Such versions seem to be common at least in the Coq community, see https://github.com/coq/opam-coq-archive/tree/master/extra-dev/packages).
I was already wondering whether the special treatment of OCaml in opam is more than just a history accident. Glad to hear the design scales.
For that, we have added "extension fields": any field in the opam file starting with
So if we want to be able to handle this with repositories, we need a different solution. There is, for example, an idea hanging around about one-command generation of an ad-hoc repositories to share this kind of setups.
Note that my wording wasn't clear: the opam tool should ignore the field, except when pinning, while the repository rules and checking tools should forbid it from appearing, but it's at a different scope.
The possibility to create the coq switches in a single command, by defining "switchable" coq packages, and to have coq and all dependencies considered "base" packages for the switch, so that they won't be changed, removed or affected by upgrades by default. It would even be possible to define a switch relying on a system installation of coq, like is done for opam's
My impression for
That does indeed sound useful, though pins seem to serve the same purpose really well.
While this is probably too much for the current in-process release, I think it could fit quite well as a plugin for now, letting us further experiment with it:
The limitation of using a plugin is that you would have to run
EDIT: note, naming the field an the plugin the same would probably be better.
referenced this issue
Mar 3, 2017
There is now a
I am curious how well this addresses your use-cases, and if it could be adapted to fit them better, or if recursive handling of a precise dependency tree with commit-hashes really seems to be the best (or only ?) solution for you.
Not having recursive pinning would certainly be a problem. However, opam-lock looks like it could provide the right tooling to overcome that.
If I compare this with Rust, there Cargo does not just bring lock files (emulated by opam-lock), but also direct dependencies on git commits:
Of course, these also work recursively and in repositories (if the repository permits that; the "main" repository at https://crates.io insists on all dependencies being version numbers of packages in the same repository). I think that's what I originally was asking for here. And indeed we now also want to support people installing the latest version of our project from a repository, so
Anyway, we have since then changed our approach to handling dependencies: We now have tooling that automatically gives each commit we push a version number and puts it into an opam repository. That lets us use normal opam dependencies throughout. The one thing we still have to hack around is the lack of a command saying "please install the build-dependencies of this package (given by a path to the opam file) and satisfy all pins and upgrade everything"; from my understanding based on #2764, opam 2.0 will bring such a feature. Right now, we have some hilarious hacks to make things work.