How to get a local opam switch in a consistent state?

[MdeLv]

On different machines (same distribution, debian buster), as soon as installation problems are encountered and solved on one machine, doing opam install foo bar baz is often not enough to get exactly the same switch state on another machine (i.e. the same set of installed packages). It's like opam silently keeps track of some previously computed constraints, that don't allow to reach a new switch state with opam install commands.
So, once a set of installed opam packages is locally validated by the developer, instead of sharing a set of opam install commands, one reliable solution is to share a switch state by doing opam switch export from the validated switch on first machine, then by doing opam switch import from an opam switch on another machine.

When package installation is broken, doing opam pin add some-patched-repo can temporarily solve the problem but it often adds some confusion to the state of the opam switch.

Doing opam switch export from two machines gives a comparison of their switch states, but is not enough to explain why there are not in the same state despite the same opam { install | reinstall | pin add } commands are executed.

Does opam keep track of change history of a switch?

What is the recommended way/tool for inspecting accurately what's beneath the state of an opam switch? (i.e. to get more than what opam switch export gives).

How to cleanly and securely fix/reset a switch to get exactly the same switch state as if build from scratch with a sequence of opam install commands?

Thanks.

[rjbou]

Opam relies on a solver to compute the actions to do (install, upgrade, downgrade, remove), and the result is not deterministic. The available packages in the repository can also have an effect on the solver's result.
Opam doesn't track its own history, and the switch export file is the switch state. The main difference is that export file contains opam files of pinned packages.
To have the exact same switch on an already installed one, you can retrieve from the export file the exact version of packages and install them. If you target an precise package or set of packages dependencies, you can use opam lock.

Why pinning packages add confusion to the switch state ? If you want to keep a specific version from being upgraded, you can use a version pinning. Otherwise, you can create an opam file containing all needed packages with their exact version and install that virtual opam file, opam-otopop can maybe help with that.

[MdeLv]

Opam relies on a solver to compute the actions to do (install, upgrade, downgrade, remove), and the result is not deterministic.

I feel like there is too much freedom left for the specification of a package's dependencies.
I tried to express a way of solving that (see point 2).
Is there a way to make the result deterministic, for example with adding specific constraints to an opam package?

The available packages in the repository can also have an effect on the solver's result.

Sure. According to my experience, even the release of a thorougly tested package by opam-repository can break the installation of an already published dependent package.
See: How to get the opam repository in a consistent state?#16309 .
See also related yesterday's issues and PRs: [BUG] ocsigen-toolkit.2.5.0 fails to compile after upgrade to js_of_ocaml.3.6.0 #1005 .

Why pinning packages add confusion to the switch state ?

It's unclear and based on my experience.
First, pinning add complexity.
Second, I think that it can trigger the installation of various packages having a doubtful compatibility and effects (see previous point). This also increases what I call the noise within the switch, that I associate with the non deterministic property you mentioned.

If you want to keep a specific version from being upgraded, you can use a version pinning.

For a complex set of packages, I do opam switch export (very easy and reliable).
For a single package, I do opam pin add.
Anyway, opam update && opam upgrade (or opam update -u) is today a powerful and dangerous command that must be used with care if one wants to stay up and running (installation issues are frequently discussed in various forum, or in discuss.ocaml.org).

[rjbou]

Is there a way to make the result deterministic, for example with adding specific constraints to an opam package?

If you lock all dependencies to a version, you'll get a deterministic package set to install. But the set of actions will remain dependent from your switch state (if a package is already installed in one and not in the other, the set of actions will differ).

On pinning, when you version pin a package, it ensures that the package won't be upgraded on update, so it reduces nondeterminism. Others kind of pin have an effect on the opam file, so yes, it can make the installation less consistent regarding opam repository.

[MdeLv]

Summary:

• lock all dependencies to a version in the opam file:
  determinism is affected by the switch state before the specified installation actions.
  Solution 1: a fresh switch -> does it guarantee determinism?
  Solution 2: an "old" switch -> is there a way to give priority to a package and its dependencies in that switch? (that would be a strict rule for the solver).

• define a set of pinned packages and dependencies:
  it only guarantees that once a package and its dependencies are installed, it will stay in that situation. That's a good and recommended practice to stay safe, and decide to upgrade with care. But It's just a lock and it's orthogonal to the discussed determinism.

• produce a configuration file with opam switch export:
  when a complex switch is defined, it's deterministic because opam will build exactly what has already been defined. This could even become a standard service for package provider along with the opam file (today, it's only a rescue technique when people ask help in forums)

Thinking about these opam files, I had the following idea:
instead of writing weird constraints on dependencies, that nothing enforces to be consistent (can be not seen by opam-repository CI and successfully installed with luck), and that will sooner or later crash someone's local opam repository, a package provider could deliver :

• an opam file for each version of the ocaml compiler, which is a key dependency (people reason mainly about it) ; with strict/univalent version for all dependencies of the considered package.
• a new version of the opam file for each new set of dependencies as they evolve, and once they have been validated by the package provider (this would guarantee that the installation of each new package version has been thoroughly tested)

Looking at https://opam.ocaml.org/packages/foo/ one will find:
Dependencies
drop-down menu with
ocaml.4.07.1
ocaml.4.08.0
ocaml.4.08.1
ocaml.4.09.0
ocaml.4.09.1
ocaml.4.10.0 < selected (last)

and for each compiler, the set of dependencies.
It could also be one opam file with sections for all compiler-related dependencies, but that would be a very long file.
The name could stay simple (opam) or hard code what it is about:
foo.ocaml.4.10.0.opam
or
foo.2.12.ocaml.4.10.0.opam
or with opam as a central point:
foo.2.12.opam.ocaml.4.10.0
ocaml.4.10.0.opam.foo.2.12
ocaml.4.10.0_opam_foo.2.12

That can create conflicts between sets of packages. But in fact this would reveal and isolate a compatibility issue instead of letting him appear on someone local opam repo.

It may look weird/unusual to many people because they are used to think about their package. But in reality, the OCaml compiler is the leader of the ecosystem, and any package provider should align to it.

What do you think about that proposal?

PS: in, say https://opam.ocaml.org/packages/ocaml-base-compiler/ , the drop-down menu is sorted in increasing order. What about sorting in decreasing order, so that the more recent are immediately available?

[dra27]

Ping @rjbou for a decision on this one