feat(pkg): interpret %{pkg-self:build-id} variable

[Alizter]

According to [1] and [2], build-id is a SHA256 hash identifying the precise package version with all its dependencies.

Consumers of this variable, such as the process.sh script in [3], use this to identify specific packages in existing opam switches. In [3] this is used to find an already-built relocatable OCaml compiler.

In dune we wish to re-use [3] for choosing the relocatable compiler, therefore we need to substitute the value with something that will not be picked up as a hash from an existing opam valuation of build-id.

For now we choose a magic number that is easily recognizable in case anything needs to be debugged.

[1] https://github.com/ocaml/opam/blob/master/src/state/opamPackageVar.ml
[2] https://github.com/ocaml/opam/wiki/Spec-for-Extended-package-specific-variables
[3] https://github.com/dra27/opam-repository/tree/relocatable

Addressing relevant concern in:

• Relocatable compiler in dune #13229

[Alizter]

I will address this point in a follow-up PR.

[Leonidas-from-XIV]

To me it seems odd that we fake a build-id which is not a correct implementation of the OPAM semantics to circumvent a hack in a third party OPAM repo. From what I see the relocatable compiler version that's merged in the upstream OCaml 5.5 does not need this, so maybe the better solution is to patch it out of the third-party repo instead of implementing a hack on the Dune side?

The issue is that if there's another user of build-id that relies on the ID to be an identifier, we won't be able to cater to both the hardcoded build-id as well as a proper build-id.

[Alizter]

To me it seems odd that we fake a build-id which is not a correct implementation of the OPAM semantics to circumvent a hack in a third party OPAM repo. From what I see the relocatable compiler version that's merged in the upstream OCaml 5.5 does not need this, so maybe the better solution is to patch it out of the third-party repo instead of implementing a hack on the Dune side?

The issue is that if there's another user of build-id that relies on the ID to be an identifier, we won't be able to cater to both the hardcoded build-id as well as a proper build-id.

Packages can put build-id but it is ultimately opam that decides what to do with this information. The aim here is not to give a well-rounded semantics meaning to the variable but to pick something that doesn't confuse the build scripts when they get it. A bad value would be a collision with a existing opam switch for instance.

Patching the other repo is also not an option. We are consuming the repo and modifying it to meet our needs simply creates more work for both sides. Picking a dummy value, with the knowledge that we didn't pick anything before and rejected it allows us to build more packages than before.

The hack in the third party repo is simply as a demo. Opam will intend to use something similar. The exact details of the implementation haven't settled yet, but its clear for now that build-id will need to be interpreted.

[rgrinberg]

To me it seems odd that we fake a build-id which is not a correct implementation of the OPAM semantics

The problem is that a faithful implementation of this variable would break dune's caching. Dune relies on digesting the contents of packages and their build instructions to avoid recompilation. Any sort of "unique" build identifier will break that.