Checksums that have changed #18044

ilovezfs · 2017-09-13T19:41:16Z

I have only checked the stable specs so far, but the checksums for the following tarballs have changed, and I expect more to change over the coming days:

CC @DomT4 this makes the patch ordeal look like nothing.

ilovezfs · 2017-09-13T19:55:05Z

What we know so far: libgit2/libgit2#4343 (comment)

fxcoudert · 2017-09-13T20:01:22Z

Either we hope that these sums don't change too often, bite the bullet this time and wait for the next. Or we disable SHA sums for all github tarballs, and start moving all formulas for which there exists an alternative hosting to that.

We could also in theory start mirroring every github tarball, but that seems like a no-no to me (cf: Sisyphus).

ilovezfs · 2017-09-13T20:20:12Z

We could also in theory start mirroring every github tarball, but that seems like a no-no to me (cf: Sisyphus).

That will not actually help, unless we made the mirror the primary, because we don't fall through to a mirror on a checksum mismatch.

DomT4 · 2017-09-13T20:26:48Z

Oh god. This is going to be actual hell.

Or we disable SHA sums for all github tarballs, and start moving all formulas for which there exists an alternative hosting to that.

Tossing any form of integrity verification out of the window seems like a drastic first step to take. Ideally, if anyone is extremely bored, projects need to get nagged to start doing actual release tarballs & have explained to them why this is important.

ilovezfs · 2017-09-13T20:49:26Z

Tossing any form of integrity verification out of the window seems like a drastic first step to take.

I'm just going to assume @fxcoudert was being facetious. 🙉

DomT4 · 2017-09-13T20:55:49Z

😆 In my defence, with how busy this week is proving to be, if you tell me that you're actually a 🐐 I'd probably take it at face value for at least an hour or so before thinking "Wait a second".

ilovezfs · 2017-09-13T20:57:10Z

I am actually a 🐐.

DomT4 · 2017-09-13T20:59:37Z

The team used to joke Xu was very possibly a non-malicious botnet because of how quickly high-quality solutions & ideas were churned out and implemented, so, rule nothing out I guess 🙈.

peff · 2017-09-13T21:26:11Z

Either we hope that these sums don't change too often, bite the bullet this time and wait for the next.

Historically, they don't change that often, and certainly we (GitHub) don't plan to make gratuitous changes. We're discussing the possibility of providing byte-stable tarballs automatically (by cementing the archives, bugs and all, at the time a tag is uploaded).

In the meantime, what can we do to help resolve the pain of this particular change? If you provided a list of archive URLs, would it be helpful to get back a list of the updated hash for each case? That would allow a mass-update of the homebrew formulas.

fxcoudert · 2017-09-13T21:39:01Z

We have 1498 URLs matching github.com/.*/archive/ in homebrew-core and homebrew-science), so I guess that's the extent of our potential problem.

But from where I'm sitting, not all of them are returning hashes that differ from the stored values. For example, https://github.com/01org/hyperscan/archive/v4.5.2.tar.gz still gives the value of 1f8fa44e94b642e54edc6a74cb8117d01984c0e661a15cad5a785e3ba28d18f5 we have on record, while https://github.com/airspy/host/archive/v1.0.9.tar.gz currently yields (for me) 967ef256596d4527b81f007f77b91caec3e9f5ab148a8fec436a703db85234cc which is different from our expectation of 358fea19f90bde13babc57ee7fdefeff3d8d8f5d629b0891734c5d4e811e8e6b.

To do a mass update we need things to have stabilised.

ilovezfs · 2017-09-13T21:40:21Z

@peff See #18048

In the meantime, what can we do to help resolve the pain of this particular change?

Unfortunately, the major concern is whether the underlying files have changed, which we can only know

a) if we have access to the original tarballs to compare them to the new tarballs
b) if we have some way to reconstruct the old checksum from the current archives

In the meantime, what can we do to help resolve the pain of this particular change? If you provided a list of archive URLs, would it be helpful to get back a list of the updated hash for each case? That would allow a mass-update of the homebrew formulas.

The list is not fixed since I assume not all have been converted to the new format yet.

ilovezfs · 2017-09-13T21:48:41Z

We're discussing the possibility of providing byte-stable tarballs automatically (by cementing the archives, bugs and all, at the time a tag is uploaded).

@peff please, please, please.... That would be wonderful ❤️

peff · 2017-09-13T22:20:13Z

Not all hashes will change. Most of the changes are due to the bugfix in git/git@22f0dcd, so only tarballs with filenames greater than 100 characters are affected.

You're right that there may also be some caching in effect. We can flush those caches, which give us a stable state.

a) if we have access to the original tarballs to compare them to the new tarballs
b) if we have some way to reconstruct the old checksum from the current archives

You can verify the change by running a version of Git both with and without that patch (i.e., git revert 22f0dcd9634a818a0c83f23ea1a48f2d620c0546 and build the result). For instance:

$ git clone https://github.com/smira/aptly
$ cd aptly
$ /path/to/reverted/build/bin-wrappers/git archive --format=tar.gz --prefix=aptly-1.1.1/ v1.1.1 | sha256sum
92aa5caa12d756cb7469fa5772a03d7631b73d655b7329408a4d597ee8fb0ba4  -
$ /path/to/modern/git/bin-wrappers/git archive --format=tar.gz --prefix=aptly-1.1.1/ v1.1.1 | sha256sum
f7bc97e46cbff2f194af2c09db099d252f290a2ea90c251adea115e6d66cf31d  -

which matches what's in #18048. A few caveats, though:

the prefix is obviously important, and comes from the canonical repo name. So if a repo is renamed, GitHub will issue a redirect from the old name, but the resulting tarball will have the new prefix in it. airspy/host is an example of this, and probably has been broken since it was renamed in March.
note that the v is stripped when producing the prefix name from the tag name
this is calling gzip -cn under the hood. Which is pretty stable, but it's possible your local gzip may produce slightly different output, wrecking the hash

ilovezfs · 2017-09-13T22:51:36Z

It looks like git will use brew's version of gzip if it happens to be installed, and that produces the correct checksums.

iMac-TMP:aptly joe$ /usr/local/opt/git-reverted/bin/git archive --format=tar.gz --prefix=aptly-1.1.1/ v1.1.1 | shasum -a256
92aa5caa12d756cb7469fa5772a03d7631b73d655b7329408a4d597ee8fb0ba4  -
iMac-TMP:aptly joe$ /usr/local/opt/git/bin/git archive --format=tar.gz --prefix=aptly-1.1.1/ v1.1.1 | shasum -a256
f7bc97e46cbff2f194af2c09db099d252f290a2ea90c251adea115e6d66cf31d  -

So this looks promising.

ilovezfs · 2017-09-14T06:53:04Z

For stable, I have verified all of the original checksums could be reconstructed except for

actual retags:

hh: dvorka/hstr#231
ndpi: ntop/nDPI#446
openvdb: https://github.com/dreamworksanimation/openvdb/issues/172
ssdb: ideawu/ssdb#1139

So far ssdb is the only upstream to acknowledge the retag.

kops, which is likely somehow related to the problem described in Was 1.6.0 retagged? kubernetes/kops#2630 and Github source code archives are not immutable (?) kubernetes/kubernetes#46443. I have informed upstream here: Was 1.6.0 retagged? kubernetes/kops#2630 (comment)
kubernetes-cli@1.3, which was already known to be wrong due to those same two issues (Was 1.6.0 retagged? kubernetes/kops#2630 and Github source code archives are not immutable (?) kubernetes/kubernetes#46443) as noted here: Github source code archives are not immutable (?) kubernetes/kubernetes#46443 (comment)

So I have merged #18048 and reverted (in #18053) the updates for hh, ndpi, and openvdb pending responses from upstream, and will not revert the others since they're accounted for.

On to devel next …

ilovezfs · 2017-09-14T07:32:48Z

The devel specs for freerdp and hub (irony, much?) needed to be fixed:
#18055

I was able to reconstruct the original checksum for both.

The remaining mismatches will likely be in resource blocks.

ilovezfs · 2017-09-14T10:09:31Z

Two checksums that were fine yesterday have gone bad today. I'm fixing them in #18072.

povray: https://github.com/POV-Ray/povray/archive/v3.7.0.3.tar.gz
teleport: https://github.com/gravitational/teleport/archive/v2.2.7.tar.gz

peff · 2017-09-14T11:26:53Z

@ilovezfs I did a complete run over all of the GitHub archive URLs I could find in homebrew-core, and picked up a few stragglers. The results are in #18073, along with the scripts I used (they clone each repo, so unless you're running them on a fast connection, it will take a while).

peff · 2017-09-14T11:28:24Z

I did a complete run

Note that this just looked for changes due to the Git patch. I didn't look for retags, renames, etc, that would cause the formula hashes to be invalid.

sjackman · 2017-09-14T18:47:33Z

You lot are all amazing. Thanks for your fast and hard work to resolve this issue so quickly.

ilyavaiser · 2017-09-19T12:00:57Z

brew install hh

Error: SHA256 mismatch
Expected: c4995e7041dc66e2118f83bd4c6c7f4cff5b4c493ca28bd7e4aef76edeff71ba
Actual: 384fee04e4c80a1964dcf443131c1da4a20dd474fb48132a51d3de0a946ba996

ilovezfs · 2017-09-19T12:02:27Z

@ilyaskorik yes that is an actual retag, which I've reported upstream, as noted here: #18044 (comment)

ilyavaiser · 2017-09-19T12:08:45Z

@ilovezfs But I can not install this program

ilovezfs · 2017-09-19T12:12:55Z

@ilyaskorik yes, you'll need to wait for upstream to respond to dvorka/hstr#231 or risk changing the checksum yourself.

boegel · 2017-09-21T07:56:44Z

@peff Do you mind clarifying something?

Not all hashes will change. Most of the changes are due to the bugfix in git/git@22f0dcd, so only tarballs with filenames greater than 100 characters are affected.

With that in mind, why is the tagged 'release' https://github.com/airspy/host/archive/v1.0.9.tar.gz affected, the filename there is way shorter than 100 characters... Am I overlooking something?

In the long run, what is the best alternative here to avoid being hit from possible future changes in the generated tarballs?
Do a git clone (or download & unpack) of the tagged version and create a tarball yourself (via tar cfvz)? I guess that wouldn't be reliable either to generate a tarball that has the same checksum on different systems, for pretty much the same reasons that cause this issue in the first place...

peff · 2017-09-21T16:01:41Z

With that in mind, why is the tagged 'release' https://github.com/airspy/host/archive/v1.0.9.tar.gz affected, the filename there is way shorter than 100 characters... Am I overlooking something?

@boegel The airspy/host hash was not affected by any change to the tarball-generating code. Its contents changed because the repository was renamed, and we use the canonical repository name as the tarball prefix. We provide an HTTP redirect from the old name, though you probably would want to update to the new URL along with the new hash. E.g.:

$ curl -w '%{redirect_url}' -o /dev/null https://github.com/airspy/host/archive/v1.0.9.tar.gz 
https://github.com/airspy/airspyone_host/archive/v1.0.9.tar.gz

In the long run, what is the best alternative here to avoid being hit from possible future changes in the generated tarballs?

Right now, I think there are basically two options:

Projects can create their own tarball using git archive (or tar, or whatever) and then upload them as an artifact to the Releases page. These are stored byte-for-byte indefinitely. Many projects already do this because they run autoconf or similar to generate their release tarballs. The big downsides are:

The UI on the Releases page still lists the auto-generated tarball (albeit at the bottom), making it potentially confusing for visitors.
This is an extra step that is borne by the project maintainer. But it's the packager who deals with the fallout from changing hashes. So the incentives are ,isaligned. And my understanding is that in general it's hard for packagers like homebrew to convince project maintainers to follow specific workflows.

Verification doesn't necessarily have to be over the byte-for-byte tarball contents. It could be over some canonicalized representation (e.g., concatenating the filenames and contents in order and taking the sha256sum of that). The problems there are:

A convenient tool doesn't already exist (though it would not be too hard to come up with one).
It increases the attack surface for impersonating an official tarball, since an attacker has more leeway to change the bytes in a way that keeps their tarball matching the canonical hash, but which may behave differently in practice.
Likewise, it increases the attack surface of the overall system. Right now nothing gets fed to tar until its hash is checked. If your canonical-hash-checker uses tar under the hood, then an attacker may target bugs in tar even before the hash is checked. You can say the same for existing hash checkers, of course, but it's a lot more likely to have bugs in something as complicated as tar versus sha256sum.

I don't know if those are compelling downsides or not.

That's a bit of a brain dump on the subject. There may be other options that I haven't thought of, too.

ilovezfs · 2017-09-21T16:07:03Z

We're discussing the possibility of providing byte-stable tarballs automatically (by cementing the archives, bugs and all, at the time a tag is uploaded).

@peff is this still in the cards as a possibility?

peff · 2017-09-21T16:30:03Z

@ilovezfs Sorry, I don't have an update. It's not something I'd be working on personally, but my understanding is it's still being explored.

ilovezfs · 2017-09-21T16:31:12Z

@peff OK, thanks :)

boegel · 2017-09-21T17:35:40Z

@peff Thanks for the extensive reply, much appreciated!

tgamblin · 2017-09-21T19:44:08Z

@peff: Thanks from me for the detailed response, as well.

I think ensuring byte-stable tarballs, either by saving them or using something like Debian's special tar arguments, is the way to go here. We were hit with this in Spack as well, and I'm having trouble deciding exactly how to resolve this, since it's ambiguous whether there are going to be any guarantees on tarballs in /archive/ links going forward.

If you do not end up guaranteeing anything about /archive/ links, I'll probably convert our 7-800 affected packages to use /release where possible, and use git clone for specific commits where there is no stable release. I may also try to convince some package maintainers to post actual releases, but I agree strongly with all the downsides to that which you mentioned, particularly:

my understanding is that in general it's hard for packagers like homebrew to convince project maintainers to follow specific workflows.

It would be good if you put a warning on the release page about the /archive/ links so that other folks aren't encouraged to rely on their hashes.

ilovezfs mentioned this issue Sep 13, 2017

SHA256 checksum changed for v0.26.0 libgit2/libgit2#4343

Open

ilovezfs mentioned this issue Sep 14, 2017

Was 1.6.0 retagged? kubernetes/kops#2630

Closed

ilovezfs mentioned this issue Sep 14, 2017

Fix devel checksums #18055

Merged

4 tasks

ilovezfs mentioned this issue Sep 14, 2017

Fix stable checksums 2 #18072

Merged

4 tasks

peff mentioned this issue Sep 14, 2017

update GitHub tarball sha256 hashes #18073

Merged

fschopp mentioned this issue Sep 18, 2017

SHA-256 of 2017_U5.tar.gz has changed oneapi-src/oneTBB#33

Closed

boegel mentioned this issue Sep 21, 2017

checksums for source tarballs downloaded from github.com/.../.../archive can change over time easybuilders/easybuild-easyconfigs#5151

Open

tgamblin mentioned this issue Sep 21, 2017

Github Archive URL checksums have changed spack/spack#5411

Closed

mislav mentioned this issue Sep 22, 2017

SHA256 mismatch when brew installing --devel mislav/hub#1568

Closed

deu mentioned this issue Sep 28, 2017

SRC_URI instead of git fetch deu/palemoon-overlay#53

Open

ilovezfs closed this as completed Sep 28, 2017

kentsommer mentioned this issue Oct 2, 2017

Libsnark version fix jl777/komodo#467

Closed

ilovezfs mentioned this issue Nov 14, 2017

azure-cli 2.0.21 #20654

Closed

4 tasks

jmillikin-stripe mentioned this issue Nov 15, 2017

Please upload non-language-specific tarballs for releases protocolbuffers/protobuf#3894

Closed

alyssais mentioned this issue Dec 15, 2017

SHA256 mismatch in bitcoin #21774

Closed

5 tasks

whot mentioned this issue Mar 8, 2018

flatpak: use cached versions from pypi instead of git tuhiproject/tuhi#106

Closed

Quuxplusone mentioned this issue Apr 15, 2018

Provide a release tarball troydhanson/uthash#152

Closed

Homebrew locked and limited conversation to collaborators May 4, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Checksums that have changed #18044

Checksums that have changed #18044

ilovezfs commented Sep 13, 2017

ilovezfs commented Sep 13, 2017

fxcoudert commented Sep 13, 2017

ilovezfs commented Sep 13, 2017

DomT4 commented Sep 13, 2017

ilovezfs commented Sep 13, 2017

DomT4 commented Sep 13, 2017 •

edited

ilovezfs commented Sep 13, 2017

DomT4 commented Sep 13, 2017

peff commented Sep 13, 2017

fxcoudert commented Sep 13, 2017 •

edited

ilovezfs commented Sep 13, 2017

ilovezfs commented Sep 13, 2017

peff commented Sep 13, 2017 •

edited

ilovezfs commented Sep 13, 2017

ilovezfs commented Sep 14, 2017

ilovezfs commented Sep 14, 2017

ilovezfs commented Sep 14, 2017

peff commented Sep 14, 2017

peff commented Sep 14, 2017

sjackman commented Sep 14, 2017

ilyavaiser commented Sep 19, 2017

ilovezfs commented Sep 19, 2017

ilyavaiser commented Sep 19, 2017

ilovezfs commented Sep 19, 2017

boegel commented Sep 21, 2017

peff commented Sep 21, 2017

ilovezfs commented Sep 21, 2017

peff commented Sep 21, 2017

ilovezfs commented Sep 21, 2017

boegel commented Sep 21, 2017

tgamblin commented Sep 21, 2017 •

edited

Checksums that have changed #18044

Checksums that have changed #18044

Comments

ilovezfs commented Sep 13, 2017

ilovezfs commented Sep 13, 2017

fxcoudert commented Sep 13, 2017

ilovezfs commented Sep 13, 2017

DomT4 commented Sep 13, 2017

ilovezfs commented Sep 13, 2017

DomT4 commented Sep 13, 2017 • edited

ilovezfs commented Sep 13, 2017

DomT4 commented Sep 13, 2017

peff commented Sep 13, 2017

fxcoudert commented Sep 13, 2017 • edited

ilovezfs commented Sep 13, 2017

ilovezfs commented Sep 13, 2017

peff commented Sep 13, 2017 • edited

ilovezfs commented Sep 13, 2017

ilovezfs commented Sep 14, 2017

ilovezfs commented Sep 14, 2017

ilovezfs commented Sep 14, 2017

peff commented Sep 14, 2017

peff commented Sep 14, 2017

sjackman commented Sep 14, 2017

ilyavaiser commented Sep 19, 2017

ilovezfs commented Sep 19, 2017

ilyavaiser commented Sep 19, 2017

ilovezfs commented Sep 19, 2017

boegel commented Sep 21, 2017

peff commented Sep 21, 2017

ilovezfs commented Sep 21, 2017

peff commented Sep 21, 2017

ilovezfs commented Sep 21, 2017

boegel commented Sep 21, 2017

tgamblin commented Sep 21, 2017 • edited

DomT4 commented Sep 13, 2017 •

edited

fxcoudert commented Sep 13, 2017 •

edited

peff commented Sep 13, 2017 •

edited

tgamblin commented Sep 21, 2017 •

edited