cmo files have different digest on different platforms and ocaml 5.1.1 variant option

[arbipher]

This issue comes from this Discuss post and encouraged by @dra27 .

We used to deliver students some reference implementation using compiled cmo files for years. They're built on one platform and can run on all platforms (macOS, Ubuntu, wsl2).

We just found this approach doesn't work for OCaml 5.1.1. When utop loads those pre-compiled cmo files, it reports errors like Corrupted compiled interface (for cmi) and disagree over interface Stdlib (for cmo). It's related to they have computed different digest values for the standard library therefore never compatible anymore. Here are some of my results to see the digest:

OCaml 5.1.1 on ubuntu on a real linux machine

$ ocamlobjinfo ~/.opam/5.1.1/lib/ocaml/stdlib__Uchar.cmi

File /home/arb/.opam/5.1.1/lib/ocaml/stdlib__Uchar.cmi
Unit name: Stdlib__Uchar
Interfaces imported:
        a8733118f9a4891e68bd3430f8176bb5        Stdlib__Uchar
        f4b8105ebd6e6055083019140b5caac1        Stdlib
        19b059febe2d4c7c479d13b2c961fa93        CamlinternalFormatBasics

OCaml 5.1.1 on ubuntu on windows wsl2

$ ocamlobjinfo ~/.opam/5.1.1/lib/ocaml/stdlib__Uchar.cmi
File /home/arb/.opam/5.1.1/lib/ocaml/stdlib__Uchar.cmi
Unit name: Stdlib__Uchar
Interfaces imported:
        a8733118f9a4891e68bd3430f8176bb5        Stdlib__Uchar
        f4b8105ebd6e6055083019140b5caac1        Stdlib
        19b059febe2d4c7c479d13b2c961fa93        CamlinternalFormatBasics

OCaml 5.1.1 (ocaml-option-afl.1,ocaml-option-flambda.1,ocaml-variants.5.1.0+options) on ubuntu on windows wsl2

$ ocamlobjinfo ~/.opam/5.1.1/lib/ocaml/stdlib__Uchar.cmi

File /home/ex/.opam/5.1.1/lib/ocaml/stdlib__Uchar.cmi
Unit name: Stdlib__Uchar
Interfaces imported:
        ff5f408a256915d7eef65919f866b521        Stdlib__Uchar
        421e9148dff60cb01a86feb5372ce6a3        Stdlib
        fb72a4260e4cc6c0d222a5d680c83278        CamlinternalFormatBasics

OCaml 5.1.1 on (probably) macOS

File /home/ex/code/fbdk-all/fbdk-web/dist/reference/Fb/libs/debugutils.cmi
Unit name: Debugutils
Interfaces imported:
	6f3d3ef3752407502bace4f4b66297d7	Debugutils
	bd0ab58d166b896b51c953631e3a39c5	Stdlib__Uchar

[gasche]

Thanks for the report.

From the discuss discussion, it sounds like @dra27 successfully diagnosed the problem as follows: .cmi digests are different depending on whether zstd compression is enabled or not. His words:

The fix I expect is simple, if a little unfortunate (I expect we have either to disable compression for .cmi files completely, marshal .cmi files twice when compression is enabled, or generalise the marshaller further to allow simultaneous compression and computation of a checksum.)

The last suggestion ("generalise the marshaller further to allow simultaneous compression and computation of a checksum") is the most satisfying to me, but it requires some extra work. I don't know who would be available to give it a try.

In the meantime, the easiest advice we can give to affected users is to configure their system without zstd compression, that is, adding the ocaml-option-no-compression option to opam ocaml option packages. (Both the producer and the consumer who expect reproducible digests should enable this.)

[gasche]

(cc @MisterDA : a juicy change in the marshaller could be of interest to you)