bad checksum for ocaml-migrate-parsetree

[aryx]

I use opam in a docker container to install some packages but I recently got this error in my CI:

[topkg.1.0.1] downloaded from http://erratique.ch/software/topkg/releases/topkg-1.0.1.tbz
[ERROR] The sources of the following couldn't be obtained, aborting:
- ocaml-migrate-parsetree.1.4.0: Bad checksum

Anyway I can fix that?

Here is the line in the Dockerfile:
Step 6/18 : RUN git pull && opam update && opam switch 4.07 && opam install dune yaml && opam install pfff pfff.0.39.3

I run this in the docker image FROM ocaml/opam2:alpine

[aryx]

Here is the relevant part of the Dockerfile:
FROM ocaml/opam2:alpine as build
USER root
RUN apk add --no-cache perl m4
USER opam
WORKDIR /home/opam/opam-repository
RUN git pull && opam update && opam switch 4.07 &&
opam install dune yaml &&
opam install pfff pfff.0.39.4
COPY --chown=opam src /build
WORKDIR /build/
RUN eval $(opam env); make

[ygrek]

$ opam info ocaml-migrate-parsetree
...
url.src:      "https://github.com/ocaml-ppx/ocaml-migrate-parsetree/releases/download/v1.4.0/ocaml-migrate-parsetree-v1.4.0.tbz"
url.checksum: "sha256=231fbdc205187b3ee266b535d9cfe44b599067b2f6e97883c782ea7bb577d3b8"
              "sha512=61ee91d2d146cc2d2ff2d5dc4ef5dea4dc4d3c8dbd8b4c9586d64b6ad7302327ab35547aa0a5b0103c3f07b66b13d416a1bee6d4d117293cd3cabe44113ec6d4"
...
$ curl -L https://github.com/ocaml-ppx/ocaml-migrate-parsetree/releases/download/v1.4.0/ocaml-migrate-parsetree-v1.4.0.tbz | sha256sum 
dcd77cd7090ce181a87df08a910ab2935da466f37db8e674e5a966d49743de64 -

[ygrek]

looks like upstream tarball was updated. cc @diml
https://github.com/ocaml-ppx/ocaml-migrate-parsetree/releases/tag/v1.4.0

v1.4.0
@diml diml released this 6 hours ago

[ygrek]

ref #15280 (comment)

[mseri]

This may also be an issue with github for which we will need to either upload the old source tarball (should be available in the opam online cache) or change the checksum in the opam file. It turns out that if one doesn’t statically upload a source for the release, github sometimes rebuild the tarballs changing the checksums. Either way we should fix it asap

[mseri]

Looks like it happened here: #15279

[ygrek]

It turns out that if one doesn’t statically upload a source for the release, github sometimes rebuild the tarballs changing the checksums.

Do you know for sure that it may happen? I have heard it happens but never experienced myself. Curious for myself - if I should keep uploading tarballs for releases or use github tags.

[aryx]

This is annoying, that means I can't just simply git pull from my docker container the opam-repository
and hope that opam install a specific version of a package would simply work.
Now I need both to install a specific version of a package AND checkout a specific commit of opam-repository. This should have been caught by the CI of opam-repository.

[aryx]

So what is the best way to fix it right now? How can I force to install an ocaml-migrate-parsetree that will not generate the error?

[mseri]

The CI caught it, but only after the file had changed: see #15279 and #15280 (comment)

[mseri]

@aryx, I think you can temporarily opam install omp.1.3.1 before installing pfff. Hopefully it will be resolved soon

[mseri]

Do you know for sure that it may happen? I have heard it happens but never experienced myself. Curious for myself - if I should keep uploading tarballs for releases or use github tags.

@ygrek It happened to multiple projects changing owner or with repos changing name but I think to remember also some PR when no change had happened and the tarballs had changed. I think both dune-release and opam-publish upload the binary tarballs before releasing the package for this reason.

[aryx]

@mseri I get [ERROR] No package named omp found.

[aryx]

Also is there a way from the command-line to disable the checksum check during opam install?

[aryx]

Ok got it, ... ocaml-migrate-parsetree for omp :)

[mseri]

Yes, sorry, I went too fast there and used the wrong name. I think you can disable the checks in some way, but in this case the tarball may have been changed by mistake. We are working on it

[samoht]

Can someone submit a PR to fix the checksum? (I don't have a Git cli nearby ;p)

[samoht]

and yes, sometimes GitHub rebuilds the tag archive which is changing the checksum. dune-release is creating (and uploading) its own archive to avoid this issue.

[ivg]

For the context: libgit2/libgit2#4343 (comment)
Basically, the archives in the <pkg>/archive/<tagname> are generated on the fly and are subject to change when any program involved in the pipeline changes, e.g., git changes its layout, order of files is changed, compressing options and so on.

The manually uploaded artifacts, i.e., that have path <pkg>/releases/download/<tag>/<file> are persistent as they are stored on their servers, so those artifacts will never change their md5sum.

It is advised to manually upload a tarball with the source code to the releases folder of github (or use dune-release, when applicable). It is also a good idea to mirror your packages, as github.com is not always available, banned in some countries, and prone to various rate-limiting errors.

Ideally, the opam linter should emit a warning, when it detects an unstable tarball url.

Also, the issues like this could be prevented with mirroring, for example, if opam had a mirror which mirrors all (or the most important) packages, we won't be having this discussion :) So far, very few packages are using mirrors in the opam universe. The last time I checked it was only bap packages.

[hannesm]

@ivg the global cache at http://opam.ocaml.org/cache/ does not count as mirror? which seems to be used by default by opam!?

[ivg]

@hannesm, will opam try the cache to retrieve a package that has a mismatching checksum?

[kit-ty-kate]

@hannesm, will opam try the cache to retrieve a package that has a mismatching checksum?

the cache shouldn't have changed so the checksum should be the right one. Only people who attempt to get the archive without the cache will experience the problem.

[hannesm]

@ivg I'm not sure I understand your question, sorry. AFAIU, opam takes the hash (hash_algo being md5, sha256, sha512) and attempts to download http://opam.ocaml.org/cache/$hash_algo/$hash[0..1]/$hash -- if successful this is the archive used for unpacking etc. if unavailable, (maybe? I'm not sure) move to the next hash_algo, and/or download from the URL specified in the opam file.

[ygrek]

So with opam cache in place there should be no upstream url downloads in perfect scenario and this problem should not have surfaced. So it means currently opam cache is not working well? Maybe more mirrors for cache needed?

[kit-ty-kate]

So with opam cache in place there should be no upstream url downloads in perfect scenario and this problem should not have surfaced. So it means currently opam cache is not working well? Maybe more mirrors for cache needed?

The cache is working perfectly well. The problem arises only for people using the docker images/opam-repository's git repository directly (for CI for instance). In those cases the cache is not used, for normal users it is used as usual.

[ygrek]

ah, so when opam remote url is not default - cache is not used, this makes sense.

corollary: mirroring opam-repository (to have stable snapshots of universe state) disables cache, ie removes protection from unstable upstreams, which is sad.

[ivg]

The cache is working perfectly well. The problem arises only for people using the docker images/opam-repository's git repository directly (for CI for instance). In those cases the cache is not used, for normal users it is used as usual.

We had a different experience this morning, as we were unable to install any package that is dependent on omp on our regular machines, that were using regular opam-repository, for example

Update: that was actually a docker container from opam-ci that was failing for us :)

[crmn111]

I encountered this issue when following instruction to install Dune from sources, using the documentation. When i try to install the newest version 'opam install ocaml-migrate-parsetree.1.4.1' it says '[Error] Package ocaml-migrate-parsetree has no version 1.4.1'. Basically the documentation on installing and running a Dune node is broken now.

[mseri]

@crmn111 Once #15280 is fixed we can merge it and it will fix your issue. Version 1.4.1 is not yet officially released