[opam 2.1~alpha4] "unavailable system package" failures are difficult to analyze in an automated way in CI

[kit-ty-kate]

ocurrent/opam-repo-ci#62 brought support for an automated failures classification system in which the CI can ignore failures on some platform based on the x-ci-accept-failures custom field.
Here is the way this works:

1. Try to install the required package
2. If this is a failure (exit code = 31), list the package that failed using ls $(opam var prefix)/.opam-switch/build and see if the current platform is specified in the x-ci-accept-failures field.
3. Otherwise exit as usual.

This works fine, however this method does not cover all the cases. In some cases the depext for some platform will be defined but does not exist – e.g. the user is expected to use an external repository or the package is not there yet (LLVM). In this case the failure will be more direct:

# on debian 10
$ opam-2.1 install conf-llvm.10.0.0
[ERROR] Package conf-llvm.10.0.0 depends on the unavailable system package 'llvm-10-dev'. You can use `--no-depexts' to attempt installation anyway.

vs

$ opam-2.1 install llvm.10.0.0
[ERROR] Package conflict!
  * Missing dependency:
    - llvm >= 10.0.0 -> conf-llvm >= 10.0.0
    depends on the unavailable system package 'llvm-10-dev'. Use `--assume-depexts' to attempt installation anyway.

No solution found, exiting

I'm not seeing a way to somewhat accurately detect which package fails apart from parsing the two types of stdout opam gives us, which doesn't seem right.

Maybe, would it maybe be possible to create an empty directory for each package from which the depexts are coming from under .opam-switch/build like it's done during the compilation process but when installing the depexts? Does anyone see another solution?

[AltGr]

Maybe a good solution could be to add more information to opam's json output ?

[dra27]

Update from discussions this morning: augmenting the JSON output seems to be the way to go. This issue arises because depexts are solved in two places - the first case is the "availability" check on atoms on the command line which takes place before the solver is invoked.

This should be fixed at some point, but we need to think carefully about versioning of the output of --json, as it's not that widely used at the moment.