SHA256 mismatch on homebrew micromamba installation

[b4rlw]

SHA256 mismatch when installing micromamba 0.27.0-2 on a fresh installation of Monterey via homebrew. Everything else brew related seems to be working correctly. Successfully installed a few days ago so not sure what's changed.

[wolfv]

thanks for the note. The SHA256 mismatch is thrown by brew? I am not sure what could've caused this :/

[wolfv]

Ah, OK, I think I know whats going on (just tried it locally):

==> Downloading https://micro.mamba.pm/api/micromamba/osx-arm64/0.27.0
==> Downloading from https://binstar-cio-packages-prod.s3.amazonaws.com/5ef2fca5fe6b2a019b74a98b/635a7c044c56ddbcd99c1c7c?response-content-disposition=attachment%3B%20filename%3D%22micromamba-0.27.0-2.tar
######################################################################## 100.0%
Error: SHA256 mismatch
Expected: 4514ca08786692a7b4619fa7c215830e5b6cc6b7077b1c463f4a54b5c82b04e4
  Actual: cefcbab3c6e5b1cf2157211af64a9c054bbf842e632f48cfd14889b11c3f09c3
    File: /Users/wolfvollprecht/Library/Caches/Homebrew/downloads/b19784b866abe6f656eaab08fcb0bb33cf6b653a5f94ce2b618f72214bacf1b4--micromamba-0.27.0-2.tar.bz2

this seems to always fetch the latest version but it doesn't take into account the build number. So when we bump the build number (which we just did), the SHA256 goes out of sync :/

I don't have a good solution for this right now. Maybe the brew maintainers of micromamba can help us out?

[pavelzw]

Maybe the brew maintainers of micromamba can help us out?

This would be @YYYasin19 and me.

The version bumping for micromamba in brew can be done with brew bump-cask-pr micromamba --version 0.27.0, this should update the SHA.

[wolfv]

@pavelzw i think maybe in brew you could also speak directly to the Anaconda API (instead of micro.mamba.pm) and get the SHA sum from there? I unfortunately don't know too much about brew

[pavelzw]

If iI'm not mistaken, you need to provide the SHA always directly in the cask code, see here. I don't think there is a way to provide an alternative SHA source other than the cask code.

[pavelzw]

Homebrew/homebrew-cask #134447 should fix this issue.

[YYYasin19]

I'm not too sure either, I think bumping the cask for every build instead of every version is the most straight forward solution.
Maybe we can automate this over CI, running an action on every build that calls brew bump-cask-pr?

[pavelzw]

The build-nr increases happen in conda-forge/micromamba-feedstock. I don't think we should touch the CI there...

[YYYasin19]

Jep, you're right, but maybe we can use something like the brew formula github action which has a livecheck mode, i.e. it detects changes based on the regex in the formula, which in turn could also capture the build number.
That means a new build would be detected as a new version as well -- 0.27.0-0 vs. 0.27.0-1 for example.
But that's also not a super clean solution

[YYYasin19]

I'm not too familiar with the conda-forge process, how often does a build number increase occur?

[pavelzw]

Not that often... Usually for build fixes or when a new architecture is added.

[pavelzw]

We could do a scheduled action in this repo that updates brew...

I would leave the version as-is since it makes more sense semantically imo.

[p-linnane]

Hi. Homebrew maintainer here .I was going to approve the PR in homebrew-cask but wanted to pop in here first. We can adjust the livecheck/version to include the timestamp so that brew will register it as a new version if that makes sense to you all.

[YYYasin19]

include the timestamp so that brew will register it as a new version

Does livecheck extract the version or just verify that it is, in fact, different?
Otherwise we could just update the regex to also check for the build number, or do you think timestamp is easier?
Thanks for jumping in, btw!

[p-linnane]

It's pretty versatile. We can have it pull any text we can reliably regex. So if you have somewhere we can pull version & build from we can use that.

[pavelzw]

The current livecheck url already has version and build nr in it. The version is here 0.27.0 and the build nr is 2.

Details

[YYYasin19]

Does something like this (in cleaner, maybe) work?
How does livecheck know which group to use? Or does it use the whole string?

[p-linnane]

Please take a look at the commit I just pushed. It should take care of this issue.

[pavelzw]

Looks good, thanks!

[p-linnane]

PR is merged. Thanks all.

[pavelzw]

@beatslikeahelix can you now try again after updating the taps? If it works you can close the issue.

[b4rlw]

@pavelzw Successfully installed!

Thanks so much everyone, this was resolved super fast.

[giordano]

This broke also MicroMamba.jl. Do we have guarantees that published tarballs won't ever be changed in-place again? That makes things non-reproducible.

[jonashaag]

There are no changes to published tarballs. You're just not guaranteed that micro.mamba.pm/api/micromamba/<ver> will always return the same tarball (it actually redirects to the latest build I guess).

[giordano]

Any "stable" URL downstream packages can use?

[jonashaag]

Well you'll always want to use the latest build. I don't know if micro.mamba.pm supports specifying a build number. Although that would open another problem of knowing what's the latest build version. Maybe can you use micromamba/conda-forge to determine the latest build and also download the files?

[wolfv]

@giordano sorry about that. I can Post the steps that the micromamba server does (use the anaconda api to find the latest version and then redirect there). The problem is that our link doesn't take into account the build number. The previous version tarballs still exist with unchanged sha, they have a different build number though.

[staticfloat]

Hello there, I'm one of the Julia developers that is impacted by this, and I thought I should explain a bit why we want stable links.

The way Julia's package manager works is to record a manifest with hashes on all downloaded resources, so that when you come back later you can "instantiate" your project manifest and get bit-for-bit identical versions of all of your software for maximum reproducibility. This reproducibility extends not only to Julia packages, but also to the binary dependencies that we serve. We really try hard to provide reproducibility up to the extreme of shipping things like CUDA userspace libraries, Cairo/GTK stacks, etc... While a big initial lift, this has worked out quite well for us, and has resulted in the community making wide use of tooling that rejects tarball URLs that change their contents.

An example of how this looks in practice is this Artifacts.toml file, which expects to, at any time in the future, be able to download that URL and verify the hashes, to ensure perfect reproducibility.

I should note that we are definitely not the only people to perform this kind of hash checking, most distribution packages (Debian, Ubuntu, Homebrew, etc...) will encode some form of hash verification on upstream releases to prevent malicious actors from spreading compromised source tarballs through their distribution channels.

Is there a URL that we can pull an exact version down from, that is guaranteed to never change? Something like https://micro.mamba.pm/api/micromamba/linux-64/0.19.0-0?

[wolfv]

I completely get the need for stable URLs and checking hashes. I need to check if the Anaconda API provides something that would allow us to provide more stable URLs. If not, we can change the server to also provide URLs derived from the repodata.json which we can make more stable (proper link to the tarball).

You could also capture the redirect from the current URL that you are using (it will redirect to the unique tarball that is going to be stable).

[jonashaag]

How are you folks keeping up to date with the latest versions? Just asking because you should also be looking for new builds of the same version, not just new versions. Does the Julia package ecosystem have the notion of build numbers?

[staticfloat]

You could also capture the redirect from the current URL that you are using (it will redirect to the unique tarball that is going to be stable).

Is that redirect URL going to be stable? That is the simplest and most expedient method for us.

How are you folks keeping up to date with the latest versions? Just asking because you should also be looking for new builds of the same version, not just new versions.

Updating versions of dependencies is always an explicit step. Julia itself uses more-or-less semantic versioning for its own packages, but in this case we're not using a Julia package, we're just treating the mamba release as a binary artifact that gets downloaded for use within a Julia package. So it's up to the Julia package author to update to a new mamba tarball when they decide there is a bug/missing feature. If they don't update, we will keep on installing the same bits.

Does the Julia package ecosystem have the notion of build numbers?

Officially, no. Packages are only just major.minor.patch, and if a package wants to make a change to a package that contains no new features nor API changes, they should bump the patch version. We bump versions and release software quite often, which you can see clearly by looking at the version history of the registry (our map of version numbers to treehashes) for an example package such as DiffEqBase.jl. But as I said before, there's a bit of a difference between versioning of Julia packages, where we have compatibility constraints, automated testing of downstream breakage, and other tools to help manage the complexity of hundreds of packages making multiple software releases a month, versus grabbing non-Julia dependencies for use in the ecosystem.

[giordano]

I'd also point out that MicroMamba.jl uses its own version numbering, doesn't follow micromamba version number, so the fact that micromamba uses build numbers is slightly irrelevant for the problem at hand. The only need the package has is to have a somewhat reliable URL. So for example that'd https://api.anaconda.org/download/conda-forge/micromamba/0.27.0/linux-64/micromamba-0.27.0-0.tar.bz2 (for x86_64 linux)?

[jonashaag]

I think that one should be the correct one but in this particular case I suggest to use the latest build.

Those files are immutable.

[wolfv]

@giordano the link you posted looks like it should work. There is also the link that mamba would use to download a package, which is https://conda.anaconda.org/conda-forge/linux-64/micromamba-0.27.0-0.tar.bz2. That link should also be stable. Now, the latest version would be https://conda.anaconda.org/conda-forge/linux-64/micromamba-0.27.0-2.tar.bz2.

The micro.mamba.pm server first calls the anaconda api: https://api.anaconda.org/release/conda-forge/micromamba/

Where version is either latest or the version number you specified (0.27.0). This API seems to return multiple entries per build number, and the server just returns the latest of them.

The server just redirects with a 301 to the selected version.

[wolfv]

Sorry guys, just a heads-up, I merged another PR that increments the build number :(

[jonashaag]

Hi everyone, are there any open problems on the Julia packaging side that we can help fix?

[giordano]

I think the URLs shared above by @wolfv should be good for us, as long as they're persistent 🙂 Thanks!