Spec for OCaml compiler invariants

Note: this one is a bit special in that this is more a change of implementation choice. However, it happens in the user-visible space, so I'll try to detail the consequences in a user-centric perspective.

Summary

An opam switch is basically a container/sandbox for a set of packages ; it helps to keep some constraints on it however, and the way to do that evolved over time:

• in opam 1, a switch was necessarily created with a specific compiler, that was then immutable for the life of the switch
• opam 2.0 added flexibility, in that you defined a set of packages to lock into place. Options allow to create empty switches, or change the locked packages in an existing switch. This allows workarounds but proved to often not be the most convenient interface for the task at hand.
• opam 2.1 generalises further, by replacing the lock on a set of passage to a generic dependency formula. This is much more flexible, and can still perfectly express the previous case.

Status

Implemented and integrated; being tested and might be adjusted in some areas.

Regression: at the moment, the conflict messages can be much worse than they used to be when compiler versions are involved. This should be fixed before the release, and I expect the fix to improve conflict message verbosity overall.

Bug (?): the user is not asked for confirmation on switch creation, like for normal installs. Here some choice might be left to the solver, so maybe it should be kept in non-obvious cases.

Usecases

Switch creation

Any opam switch create command that worked on 2.0 is expected to work on 2.1.

Example:

opam switch create 4.09.0

this will still scan for existing packages with flag compiler having the given version, and set them as switch invariant of a new switch named 4.09.0. A difference will happen in case your system compiler is 4.09.0: opam 2.0 would then fail because it couldn't choose between the two available implementations. Instead, 2.1 will keep the disjunction in the invariant (without needing to check the system): ["ocaml-base-compiler" {= "4.09.0"} | "ocaml-system" {= "4.09.0"}]. Then choosing an implementation will be left to the solver as normal. Supposing you were on ocaml-system and your system compiler got updated, opam will automatically switch back to ocaml-base-compiler and recompile all the dependents.

If the command is more explicit, e.g. ocaml-base-compiler.4.10.0, that will be set as invariant, leading to a single "locked" package, like with opam 1 or 2.0.

It is also possible to choose a package, without version constraint, or with an open one:

opam switch create test 'ocaml>=4.05.0'

The latest version will be chosen, with an implementation up to the solver, so normally ocaml-base-compiler.

Creating a switch without specifying an invariant is now supported, and will use the default (as opam init already did). The new default is set to the more open ocaml>=4.05.0.

Switch upgrade

One of the advantages is to leave more freedom to the solver on upgrades, if desired. The solver follows its normal behaviour: if the invariant is strict, nothing will change compared to opam 2.0; if the invariant is less explicit, the solver will be free to change the solver version within the constraints:

Example: invariant = ocaml-base-compiler

in this case, opam upgrade will always optimise to the latest versions of the compiler, given the other installed packages are compatible

an opam install foo might prompt to downgrade the compiler instead of failing, if no better solution exists to install foo.

Example: invariant = ocaml.4.10.0

as hinted above, it becomes possible to switch between system or non-system compilers, typically when the availability status of the former changes. Keep in mind that the solver always minimises changes, so it won't jump back and forth without reason.

New commands and options

Uses the same format for the formula as opam files do:

opam switch create foo --formula=<dependency-formula>

Prints the current invariant

opam switch invariant

Updates the invariant. It can also be automatically inferred (using installed packages with the "compiler" flag)

opam switch set-invariant [--force]

Similar to 2.0's --unlock-base, will relax and automatically update the invariant.

opam install --update-invariant

Windows and toolchain setups

(todo)

Playing with pinned compilers

(todo)

Implementation

The interface with the solver uses a virtual package with the "invariant" as dependency formula, that is marked as required. Care is taken to extract it before the universe is returned (dose uses the same trick already, so it's quite safe).

Care is taken to infer correct invariants when upgrading from 2.0.

Overall, removing the specific treatment for base packages is a simplification. In particular, opam switch create DIR had special code to guess a compatible compiler in advance, while this can now be delayed to the solver. Many parts needed to be adapted though (e.g. detecting the packages in conflict with the invariant).