sha1collisiondetection: init -> unstable-2017-02-21

[leenaars]

Motivation for this change

This creates a drop-in replacement for sha1sum, which can be used to detect collision attacks on SHA1.

(see:
http://www.cwi.nl/news/2017/cwi-and-google-announce-first-collision-industry-security-standard-sha-1)

There are currently over 8000 sha1 hashes still in use in nixpkgs itself.

Things done

• Tested using sandboxing
  (nix.useSandbox on NixOS,
  or option build-use-sandbox in nix.conf
  on non-NixOS)
• Built on platform(s)
  • NixOS
  • macOS
  • Linux
• Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Fits CONTRIBUTING.md.

---

[rycee]

According to the package naming guidelines this package should have

version = "2017-02-21";
name = "${pname}-unstable-${version}";

[leenaars]

Done.

[leenaars]

I've made that change. Thanks for reviewing.

[leenaars]

@grahamc : would the Security Team consider the above proposal to replace the old sha1sum with the new one that is able to detect collision attacks, and deprecate the 'shattered' one?

[leenaars]

The idea is that it should generate exactly the same hashes for the 'real' artefacts, but will actually generate another one for manipulated artefacts. The possibility of false positives can be neglected as the probability is smaller than 2^-90.

[rycee]

@leenaars As far as I can see the version = "git-20170221"; is still there. It should be version = "2017-02-21"; also note that versions in Nix always start with a digit. So at the moment the package name, according to Nix, is "sha1collisiondetection-unstable-git" and the version is "20170221".

[leenaars]

rycee: sorry, I'm used to the abbreviated ISO 8601 date notation. Fixed.

[leenaars]

@joachifm : sha1sum is broken, because it implements a hashing algorithm that is now known to be broken. This tool implements exactly that same algorithm, but as it calculates the hash it also does a thorough check on whether the object for which the hash is requested has not been manipulated as described in the paper by Stevens et al. In the latter case, it will detect this, and the result is another SHA1 - meaning the user is alarmed and the attack has failed.

My argument is thus, that there is no reason to keep the sha1sum as it is in NixOS - its only function can be dealt with by this tool better (the new version is twice as fast). And the new tool has the advantage of protection against a core weakness in that only function of the tool, namely using hashes to protect the integrity of objects downloaded from untrusted sources like the internet.

So I'd rename sha1sum into sha1sum.legacy and symlink the new tool to sha1sum. This is partially orthogonal to addressing the possibility of having collisions in NixOS in the first place (all other uses of sha1sum by users of NixOS should benefit as well), if it weren't for the fact that it actually does mitigate the risk of having 8000+ hashes still in place. These should be safe against this collision attack. And did I mention that it is over twice as fast?

[joachifm]

sha1 has been known to be suspect since 2005. It's difficult to see how the recent announcement changes anything, as concerns using sha1 by itself for anything other than detecting accidental corruption.

[leenaars]

Well, the second part of the announcement is that there is now code from the science community that everyone can use to counter all known negative issues with the known weakness - and avoid all suspicion. So this is an actual fix and improvement - by a full and independent reimplementation. For all intents and purposes, this is the 'new' sha1sum and the old one should be removed. It just has a different name, because someone else wrote it. The old sha1sum is broken, the new one is not (and in fact is pretty damage proof). When using the new sha1collisiondetection tool, you are safe against this whole class of attacks. That is new.

[edolstra]

What does this mean? What are "collision attacks present in each file"?

[leenaars]

@edolstra: The description is from the author (Marc Stevens from CWI) himself. The cool trick the tools perform is counter-cryptanalytics. Imagine some tarball with source code which is crafted specifically to perform an attack, and injected through a man in the middle attack. How nix plans to deal with the source file once it has passed the test (say: configure, make, make install) is fully transparent, and (if the MitM works) it would be possible to have attacker controlled code to end up in a running system.

Manipulating the sha1 hash value of the fake source object is after all now quite realistic (as per the paper) and would result in its sha1 hash value to be the same as the original object (a "cryptanalytic collision attack"). However, and fortunate for us, that procedure leaves traces. With these tools those traces show up like on an x-ray: each file is checked for cryptanalytic collision attacks after the regular hash is computed. But instead of making it a separate tool, this does something even smarter: it replaces the original tool with a 'secure' version. So it does exactly what the original sha1sum does, but then proceeds to a second stage checking the countercryptanalytic indicators. If an attack (i.e. a manipulated sha1 hash) is found, rather than crying foul it will just return an alternative hash - making the attacked process reject the manipulated file. Which is why the hash was calculated, making it behave as users would expect.

[joachifm]

From the descriptions above, it sounds as if any archive can be attacked, but that's not what has been demonstrated, is it? Wouldn't the MITM need to generate both the good and bad archives before even thinking about how to mount an attack? (That is, existing sources using sha1 in Nixpkgs are not "vulnerable", though it is somewhat feasible that one or more of them were created by a state actor who also has a corresponding bad archive, to be disseminated at their discretion.)

[leenaars]

Why run any risk, if the solution is superior in every respect other than its age?

You are right that this probably needs a significant effort, but this scientific scoop has been in the works for years and with collaborators outside of science. Like the Kaminsky DNS hack, it now comes down to ingenuity and time - and with high stakes, both are available to quite some shady organisations. File archives are within direct scope:

Our work shows that it is now practical to find collisions for SHA1 and that thus it is not secure to use for digital signatures, file integrity, and file identification purposes.

The author said this in an interview:

As an example of the insecurity of SHA1 [the identical-prefix collision] is very meaningful. Chosen-prefix collisions are more powerful and relieve an attacker from some constraints, but identical-prefix collisions can be just as dangerous with a very creative attacker. It is hard to say which things you can’t do with an identical-prefix collision.

But we do know quite some demonstrated threats using identical-prefix collisions: certificates with identical names and different pubkeys, PDF files, PostScript files, TIFF files, JPEG files, Word files, file archives, signed software, Email message/attachment PGP/GPG signatures.

I don't know what Nix users use sha1 hashes for, but from my perspective there seem to be no reasons to hang on to the broken version as the new one is:

a) more future-proof against known and suspected attacks
b) faster (so it would speed up Nix)
c) otherwise functionally identical

(the first thing I would do would be to look for near-collision blocks around all the popular open source license texts - that would open up a lot of hurt)

[joachifm]

So long as we proceed with a realistic mindset and with a view to how this concerns us as downstream packagers.

[joachifm]

Merged at b5bb83b with some fixups.