opam2 selects out-of-date packages more than opam1

[avsm]

Discussed with @AltGr and is due to a very different package universe in opam2, but the problem is quite badly manifesting from a user perspective.

Two Dockerfiles that are reproducible for a mirage installation:

opam2:

$ opam install mirage
The following actions will be performed:
  - install conf-which      1                           [required by camlp4]
  - install conf-pkg-config 1.0                         [required by conf-ncurses]
  - install conf-m4         1                           [required by ocamlfind]
  - install camlp4          4.02+6                      [required by cstruct, lwt]
  - install conf-ncurses    1                           [required by lwt]
  - install ocamlfind       1.7.1                       [required by mirage]
  - install lwt             2.4.5                       [required by mirage]
  - install cstruct         0.4.0                       [required by mirage]
  - install mirage          0.4.0
===== 9 to install =====

with opam1:

The following actions will be simulated:
  - install conf-m4          1                                  [required by ocamlfind]
  - install result           1.2                                [required by logs]
  - install ocamlfind        1.7.1                              [required by mirage-types, ipaddr, astring, etc.]
  - install topkg            0.8.1                              [required by ipaddr, astring, logs]
  - install ppx_tools        5.0+4.02.0                         [required by lwt, ppx_sexp_conv, cstruct]
  - install ocamlgraph       1.8.7                              [required by functoria]
  - install js-build-tools   113.33.04                          [required by ppx_sexp_conv]
  - install base-bytes       base                               [required by ipaddr]
  - install rresult          0.5.0                              [required by functoria]
  - install fmt              0.8.0                              [required by functoria, logs]
  - install astring          0.8.3                              [required by mirage]
  - install sexplib          113.33.03                          [required by ipaddr]
  - install ppx_core         113.33.03                          [required by ppx_sexp_conv]
  - install lwt              2.6.0                              [required by mirage-types, logs, mirage-types-lwt]
  - install cppo             1.4.1                              [required by ocplib-endian]
  - install functoria        1.1.1                              [required by mirage]
  - install ppx_optcomp      113.33.03                          [required by ppx_driver]
  - install logs             0.6.2                              [required by mirage]
  - install ppx_deriving     4.1                                [required by ppx_type_conv]
  - install ocplib-endian    1.0                                [required by cstruct]
  - install ppx_driver       113.33.03                          [required by ppx_type_conv]
  - install cstruct          2.3.1                              [required by mirage-types, mirage-types-lwt]
  - install ppx_type_conv    113.33.03                          [required by ppx_sexp_conv]
  - install io-page          1.6.1                              [required by mirage-types, mirage-types-lwt]
  - install ppx_sexp_conv    113.33.03                          [required by ipaddr]
  - install ipaddr           2.7.1                              [required by mirage]
  - install mirage-types     2.8.0                              [required by mirage]
  - install mirage-types-lwt 2.8.0                              [required by mirage]
  - install mirage           2.9.1     

The two dockerfiles are:

opam2:

FROM ocaml/opam-dev:alpine-3.5_ocaml-4.02.3
WORKDIR /home/opam/opam-repository
RUN git checkout master
RUN git fetch origin master:cibranch
RUN git branch -D v2
RUN git checkout -b v2 c96ef118c49ff9722654bbd4e31c863b6b24b3f7
RUN opam admin upgrade-format
RUN git add .
RUN git commit -a -m 'upgrade format to opam2'
RUN git clone git://github.com/mirage/mirage-dev /home/opam/src
WORKDIR /home/opam/src
RUN git fetch origin heads/master:cibranch
RUN git checkout cd73241ae0a2701bfbb6029dca6ecf299409a7a3
RUN opam admin upgrade-format
RUN opam remote add local /home/opam/src
RUN opam update -uy
RUN opam install mirage

and for opam1

FROM ocaml/opam:alpine-3.5_ocaml-4.02.3
WORKDIR /home/opam/opam-repository
RUN git fetch origin master:cibranch
RUN git checkout c96ef118c49ff9722654bbd4e31c863b6b24b3f7
RUN git clone git://github.com/mirage/mirage-dev /home/opam/src
WORKDIR /home/opam/src
RUN git fetch origin heads/master:cibranch
RUN git checkout cd73241ae0a2701bfbb6029dca6ecf299409a7a3
RUN opam remote add local /home/opam/src
RUN opam update -uy
RUN opam install mirage

[ghost]

Hi @avsm, it seems it is more a bug related to the docker image. I noticed the bug recently (when fighting with ocaml/opam-repository#8048) and shamefully postponed the report ! Sorry about that. Here it is.

If you run opam config report in the two docker images, you will see that opam does not use the same criteria. That's because the result of aspucd -v is not the same.

When opam is not able to detect that aspcud is recent enough, it uses rather simple criteria (too simple ? I don't know @AltGr maybe has an answer ?).

In the image with opam1, you use a binary aspcud, which opam detects correctly as version 1.9. In the image with opam2, you use a "remote" aspcud through a shell script that does not handle -v. We had the same bug a long time ago with the remote solver from Irill. See: http://cudf-solvers.irill.org/cudf_remote_proxy

Maybe aspcud-1.9 is old enough for us to change the defaut criteria in opam2 (while using the simpler when it detects aspcud < 1.9 ?)

[ghost]

Another fix would be to set OPAMCRITERIA in the docker image, but for some encoding reason, it seems to fail the remote_aspcud script.

[AltGr]

Ah! Thanks @OCamlPro-Henry for spotting this!

(while using the simpler when it detects aspcud < 1.9 ?)
Not easily doable, older versions don't properly handle -v... Also, these criteria have the advantage to work with any other solver too, while the more advanced ones are more aspcud-specific.

Note that mccs (any version, AFAIK) supports handling the version-lag feature, too.
EDIT: (but requires specific criteria)

[avsm]

Many thanks for the expert detective work @OCamlPro-Henry! There were a couple of bugs here that are now fixed:

• Alpine 3.5 now includes OPAM 1.2.2 and Aspcud (WIP: OCaml ports alpinelinux/aports#26) and so I updated the remote solver proxy to now run aspcud 1.9 from an Alpine 3.5 base (avsm/opam-solver-proxy@722e644), and pushed live to solver.ocaml.io (upgrade aspcud in solver.ocaml.io infrastructure#10)
• for distributions without aspcud, the wrapper script now responds to -v and does better escaping, as you suggested.
• The opam2 Alpine 3.5 containers were incorrectly using the cloud solver despite aspcud being present, so the Dockerfile generator has been fixed (ocurrent/ocaml-dockerfile@1c4e3a4)

So once all the rebuilds are done, we should have Aspcud 1.9 criteria available consistently on both OPAM 1.2 and OPAM 2.0 Dockerfile containers. Please let me know if you see any more issues with this, and I'll close this for now.