opam 2.1.3 wants to recompile a lot of packages upon removal of an unused package

[brandenburg]

When uninstalling a package, opam proceeds to recompile massive parts of the current switch, which is very counterintuitive. This suddenly started happening to us in mathcomp Docker images used for CI, which brought the CI infrastructure to a crawl and ultimately down.

How to reproduce:

docker run --rm mathcomp/mathcomp:1.16.0-coq-8.16 bash -c "opam update; opam remove -y coq-mathcomp-character"

I would expect this to just remove the library. Instead, we get:

                                                                                                                       
<><> Updating package repositories ><><><><><><><><><><><><><><><><><><><><><><>                                       
[coq-released] synchronised from https://coq.inria.fr/opam/released                                                    
[default] synchronised from https://opam.ocaml.org                                                                     
Now run 'opam upgrade' to apply any package updates.                                                                   
The following actions will be performed:                                                                               
  - recompile dune                   3.6.1*        [upstream or system changes]                                        
  - remove    coq-mathcomp-character 1.16.0*                                                                           
  - recompile stdlib-shims           0.3.0         [uses dune]                                                         
  - recompile sexplib0               v0.15.1       [uses dune]                                                         
  - recompile result                 1.5           [uses dune]                                                         
  - recompile ppx_derivers           1.2.1         [uses dune]                                                         
  - recompile ocaml-compiler-libs    v0.12.4       [uses dune]                                                         
  - recompile csexp                  1.5.1         [uses dune]                                                         
  - recompile cppo                   1.6.9         [uses dune]                                                         
  - recompile coq                    8.16.1*       [uses dune]                                                         
  - recompile ppxlib                 0.28.0        [uses dune]                                                         
  - recompile dune-configurator      3.6.1         [uses dune]                                                         
  - recompile yojson                 2.0.2         [uses dune]                                                         
  - recompile coq-mathcomp-ssreflect 1.16.0*       [uses coq]                                                          
  - recompile coq-bignums            8.16.0        [uses coq]                                                          
  - recompile ppx_import             1.10.0        [uses dune]                                                         
  - recompile ppx_deriving           5.2.1         [uses dune]                                                         
  - recompile base                   v0.15.1       [uses dune]                                                         
  - recompile coq-mathcomp-fingroup  1.16.0*       [uses coq-mathcomp-ssreflect]                                       
  - recompile ppx_deriving_yojson    3.7.0         [uses dune]                                                         
  - recompile ppx_sexp_conv          v0.15.1       [uses dune]                                                         
  - recompile ppx_compare            v0.15.0       [uses dune]                                                         
  - recompile parsexp                v0.15.0       [uses dune]                                                         
  - recompile coq-mathcomp-algebra   1.16.0*       [uses coq-mathcomp-fingroup]                                        
  - recompile ppx_hash               v0.15.0       [uses dune]                                                         
  - recompile sexplib                v0.15.1       [uses dune]                                                         
  - recompile coq-mathcomp-solvable  1.16.0*       [uses coq-mathcomp-algebra]                                         
  - recompile coq-serapi             8.16.0+0.16.2 [uses dune]                                                         
  - recompile coq-mathcomp-field     1.16.0*       [uses coq-mathcomp-character]                                       
===== 28 to recompile | 1 to remove =====                                                                              

(Note: we need the opam update because we also want to install a couple of packages afterwards.)

I guess this is triggered by the dune package somehow?

Note this happens despite dune being pinned to the current version:

$ docker run --rm mathcomp/mathcomp:1.16.0-coq-8.16 opam pin list
coq.8.16.1                       version  8.16.1
coq-mathcomp-algebra.1.16.0      version  1.16.0
coq-mathcomp-character.1.16.0    version  1.16.0
coq-mathcomp-field.1.16.0        version  1.16.0
coq-mathcomp-fingroup.1.16.0     version  1.16.0
coq-mathcomp-solvable.1.16.0     version  1.16.0
coq-mathcomp-ssreflect.1.16.0    version  1.16.0
dune.3.6.1                       version  3.6.1
num.1.4                          version  1.4
ocamlfind.1.9.1                  version  1.9.1
zarith.1.12                      version  1.12

Config report, as requested:

$ docker run --rm mathcomp/mathcomp:1.16.0-coq-8.16 opam config report
# opam config report
# opam-version         2.1.3 
# self-upgrade         no
# system               arch=x86_64 os=linux os-distribution=debian os-version=11
# solver               builtin-mccs+glpk
# install-criteria     -removed,-count[avoid-version,changed],-count[version-lag,request],-count[version-lag,changed],-count[missing-depexts,changed],-changed
# upgrade-criteria     -removed,-count[avoid-version,changed],-count[version-lag,solution],-count[missing-depexts,changed],-new
# jobs                 111
# repositories         2 (http) (default repo at d85efebe)
# pinned               11 (version)
# current-switch       4.13.1+flambda
# ocaml:native         true
# ocaml:native-tools   true
# ocaml:native-dynlink true
# ocaml:stubsdir       /home/coq/.opam/4.13.1+flambda/lib/ocaml/stublibs:/home/coq/.opam/4.13.1+flambda/lib/ocaml
# ocaml:preinstalled   false
# ocaml:compiler       4.13.1+options+flambda

I need opam to not recompile everything in mathcomp Docker images despite everything being pinned. How can I stop it from triggering the recompilation?

[rjbou]

For the moment, it is not possible to not trigger recompile on upstream changes. In your case, dune.3.6.1 changed its opam file (depends constraints), and it triggers recompile. It is pinned, but version pinned, so opam takes opam file from repo. Version pin only ensure that version won't change (upgrade/downgrade).
If you want to be sure to have your packages not recompiled, you need to pin them manually to their archive (or url), for ex opam pin dune.3.6.1 https://github.com/ocaml/dune/releases/download/3.6.1/dune-3.6.1.tbz. you can automatically retrieve url with opam show
related #4928 & revision field.

[kit-ty-kate]

That's sadly expected with opam < 2.2 (fixed in #5118).
This happened because we've fixed some metadata in ocaml/opam-repository#23574

opam 2.2 is hopefully coming in the next few weeks (see https://github.com/ocaml/opam/projects/2#column-19255146) but if you want to try it now you can do it with the following commands:

git clone https://github.com/ocaml/opam
make -C opam cold
sudo install ./opam/opam /usr/local/bin/opam
opam init --reinit -ni

However for your docker setup I'm not sure there is anything easy to use at the moment aside from using archive pins as @rjbou suggested.

[brandenburg]

Thanks a lot for your answers! It's very good to see that a proper solution is on the horizon.

To get our CI working again, I needed a workaround now, so I ended up creating a custom Docker image that bakes everything in. The "rebuild everything" behavior still happens, but only once during the image build. The downside is that we now need to maintain yet another image (and that the image is needlessly large due to recompiling everything).

Once OPAM 2.2 has made its way into the Debian base images, maybe we can reconsider.

Thanks for your work on opam! :-)