How to deal with package signing?

[joepie91]

I just read #75, which seems to suggest that binary cache 'packages' can be signed... however, this does not seem to cover packages that are installed from source. The Nix packages do seem to have hashes of external downloads, but those wouldn't be useful for verification unless the .nix file itself were signed somehow.

How would one publish a Nix expression(?) that can be cryptographically verified to be unmodified?

[vcunat]

That seems quite an overkill. We don't even sign the git commits (yet). You get the sources through https typically, so that's some security at least.

[joepie91]

For it to be useful in production (eg. for an end-user distro), this would be critical functionality, though.

HTTPS only provides transport security - ie. it ensures that your connection to the distribution server is not intercepted or modified, but places its trust in the distribution server.

Package signing serves a different purpose - it ensures that a package was not modified from the moment and place it was created by a maintainer, even if the distribution server is untrusted (eg. compromised).

A distribution server that distributes unsigned packages would be a high-value target for a watering hole attack (whether targeted or not), as it becomes trivial to compromise a large amount of machines through the compromise of a single server.

I understand that Nix might not be at that point of development yet, but it would certainly be a pre-requisite for production use outside of tightly controlled internal networks (which typically have their own distribution servers anyway).

[vcunat]

We hand out commit access relatively easily – if someone wanted to attack nixpkgs, it's enough to send many useful PRs, then ask for push access after a month or so and sneak in a different package hash or something...

[joepie91]

Right. I don't just mean nixpkgs, though - I mean having support for it in Nix as a package manager, from an implementation point of view. That's why I filed it here rather than on nixpkgs :)

I'd imagine that eg. other distributions might want to use their own channels, rather than nixpkgs, with more stringent package/maintainer control.

[edolstra]

What's the threat model here? And what would signing a source package look like? What does it mean for a package not to be modified? Signing a package is kind of pointless if you don't also sign all the dependencies, in such a way that a change to a dependency invalidates the signature on the package. Otherwise, you can compromise a package by compromising any dependency. E.g., updating the Nix expression for GCC should invalidate the signature on the Nix expression for Firefox. But that would be pretty unworkable...

[paragonie-scott]

Digital signature algorithms, such as Ed25519, can allow you to verify that the package you are receiving is authentic even if the server you got it from has been compromised. HTTPS doesn't protect data at rest, it only protects it in transit.

Without digitally signing packages, you have to do the following to attack a user:

• 0wn the server.

With digitally signing packages:

• 0wn the server
• Get the signing key too, which if stored offline will be a pain

The idea is to protect your users from your infrastructure. It won't protect your users from you, should you turn to the dark side.

[vcunat]

@paragonie-scott: IMO noone was questioning those points. Also note that the transport of packages is completely secure already – binary cache packages are signed now (after being built) and verified by default (when downloaded).

[paragonie-scott]

Right. I was trying to clarify the threat model a bit.

[vcunat]

Next step I can see is the question of trusting github and/or nixpkgs repo transport, i.e. each contributor signing his/her nixpkgs commits. That probably wouldn't be costly (by itself), but I don't feel it would bring significant security improvement (by itself).

I don't feel github as a significant threat, as they have a lot to lose if betraying the trust, compared to e.g. random person with push access into nixpkgs. Heck, we don't even do any thorough security review of most PRs coming from complete strangers. (I guess I'm repeating myself, so let me stop here.)

[joepie91]

@vcunat It's not just about GitHub itself, though - if their systems get compromised (happens), then everybody is still at risk.

Regardless, this isn't just about nixpkgs - an individual or organization (eg. a distro maintainer) may want to operate their own Nix channel with package signing, as I mentioned before. This would have to be at the very least a feature available or implementable in Nix, in some manner.

[vcunat]

Anyone can start signing his/her own tags into a nixpkgs tree fork (e.g. after a security review). The infrastructure is all there (git's --verify-signatures). Channels can't do that automatically AFAIK, but frankly, that's only a tiny inconvenience, and you can do the same with a few lines of scripts through git.

[diimpp]

As a random gentoo dude, I might say, that this feature is a must. 👍

[copumpkin]

I'm still struggling to understand what additional things need signing.
I've already asked for signed channels but I see those as more useful for
private organizations. What types of signatures would add meaningfully to
the security of the public Nix infrastructure?
On Thu, Feb 25, 2016 at 23:27 Dmitri Perunov notifications@github.com
wrote:

As a random gentoo dude, I might say, that this feature is a must. [image:
👍]

—
Reply to this email directly or view it on GitHub
#613 (comment).

[diimpp]

@copumpkin .nix files by itself, attached patches. All files in packages dir.

For example gentoo has Manifest (https://devmanual.gentoo.org/general-concepts/manifest/) file, which lists all files for package, and which should be signed with gpg.

Example Manifest file for portage package manager

 AUX README.RESCUE 137 SHA256 69bea0c755eab59b6b4d90838b8551744f0b0c8eeac170b1cd13a71ee83f9ac4 SHA512 dba0f0d6d4cd140e636cc70dfb7bdb298e2f2fb040bc07b0874cac994bdbabd26bfd9361d85a0f8b9a92a185 281e4901606b9e4f563afdc8867341bbe6af1cb0 WHIRLPOOL 713fcc790442999d56e13efaf536fec9954dadeda680b3642d0358311332096ddedf8b55697bd1ca620d84f0dc7d025ff7931e086c74c1f5032be72dca2ff8f5
 DIST portage-2.2.20.1.tar.bz2 950298 SHA256 84dd3ed1d9fd0a3336e956d057590c75834a906ae62a450411232fe5a842475c SHA512 5f7373da20d448059e8fdc8a3593eaca780be3184f0494bac66f5eea7c33d00dd61b9e74c be9a52413f08cef29d426ed8751e20e16be09178468c18d77a6ae2e WHIRLPOOL 99643ce0ab74c5df64fb5239948c6102bc39847ff88eaf210c069c0c9587c1e8bdd1a032a7e61c1225b6113645387861d439e9c4f103df4d0f042b73e1a fc328
 DIST portage-2.2.24.tar.bz2 972610 SHA256 c2dbd53dd474c6fe455ffa5920514a6afe467abf02959c33ddb7a7e363939550 SHA512 9dc00410de604db66843bdf62cf0ee7e93ff2c4d562390172c4f0c55ab98b2712078b0eb02c 9f35d52633b0fff0b024736e25a1c66cf40ad61a6bf99a7cee4a6 WHIRLPOOL 3227c61662acf69abd843344d0fb1b7cc2cfa91c1bae48b08e1da8fdf42cda15834994852d42dfc089f189e318c657e12fd6b1c19885279e0bb5cfd234923 5f8
 DIST portage-2.2.26.tar.bz2 977277 SHA256 7f57aa704c58ea47f031f177203dc7b335e01e7ec1e809437ea9e46e3f9263e5 SHA512 fd1a5e5d028d52ba82bbeb7f87bf331d0c1d8c7f5a3bd450c668dd742e99185b6d2b21944f5 d7788fe941c99736744b08157ca30145f0964ea60a7c2c435d601 WHIRLPOOL 0469f4c5609146c57eae4baeb8488b95c04f336a5511460ee44d15746339fd43a7c2d61b58528845d77a6a0b21e46c2afad19a9f05e94ac8dc6180032d71f 298
 DIST portage-2.2.27.tar.bz2 977125 SHA256 3126b59456d2f0a1c9b2c03df15154b2fb29e462abbbe1b8898fa7bc291421af SHA512 8f16a7c9ce33bd34d60f0b06d3e4e0bd4e2c27e6bdc0756e6d3b59108b5438e92f5ca9e981b bfe4ebe9b64f1afe9442d72c52c02fe456e0d36daffae6f77d694 WHIRLPOOL e467c6d9e4e1c177667389a4ce134fabc8676ff5baf9403d73de241086c7d5bab4d11bfe84be749cf9f9781cadc1dd8d173e03e9fba4c926e24131c261a28 d1e
 DIST portage-2.2.8.tar.bz2 874672 SHA256 c1074e01173df5384e003598dbc5f64f09e92f83fd26faab2f88f9f0bc64aa7e SHA512 acc47d94ff8c08e5a7bf2d4c4e600863430dd1e7d0bb8b4254d6477a498933cae663db84debf 9487d2243c2cba689629ef8e8b6cf108e10d40e89e8e8d7e1257 WHIRLPOOL d54290ede8d3d85e5211cda7e86ed740178121ef2f47cf0858c27941d8679d29c11445befff962986d40a47ded50d0d4654de19e7d4aef3fdf2546655a0d6f e2
 EBUILD portage-2.2.20.1.ebuild 11680 SHA256 6c51e764f0bc015275bf050b83df8f661a21cb620ccb280044d021c1c6929733 SHA512 700e1733583e7462abe24a00f9f74a205c3faefb011f61a02eb517764301cbad0fec1e32b aded4c436ecf1a0435deed96e8eec8586bfcdba6b0699375626794d WHIRLPOOL 4298c287eadbdc1510616f155a64edcc033887dd1aa673d824295a03ce4f8c073ff82074cdf9e55ca559a8e7b5cf4b476dc1df5ed177917f248b3b1843a ec20f
 EBUILD portage-2.2.24.ebuild 11655 SHA256 4e792a3dacd232defcf8d9c498b63ccf311518f1c7846a69d7696b0c1ceb051a SHA512 89fb3b6ca2dcbad68c2e8315b5cd511c01bc47f845a4505d2607df6e06a737d670822c9f0e6 637ad01a78ada619c9ca93adccd3a86aeb40bb15033261a10587e WHIRLPOOL 3415f90db101bb8e72207871ee580cd4fd806ec6190659afbcf6843494e6c4bdd282f7716fad9c4e99f362f59522adb8c6021a658a8455912b35223f83b95 2d4
 EBUILD portage-2.2.26.ebuild 11729 SHA256 840aa1305c93f155f8a019c978dffed9e32d95e371fb5acc211b2adb77aa62b8 SHA512 6fe47e1522f84a72ff995d65b63555c6aa8faeea0ebaaca242b57ec38ba0034fc3f26bef39c a048aa30cd7bbd366de2205c7988d16e8bff28440deda60365454 WHIRLPOOL 8fd8618f353f04dc2bbf0c4416e05596d11b662bd2f341e3ab471fb2edd846c5dd416441f994e43ac41b4bbd12c34d3c496e4eb7cd7184ab074a7da60cee0 763
 EBUILD portage-2.2.27.ebuild 11739 SHA256 5f8ee0af706a8c2f2606c1169587338eae914a2fe058ece5cb441e380639f4be SHA512 acb8516b139972585063e3d64ba07296910316ccb000076036cbfeaf3b71548d27b3f96b434 bcaeb498a8baea8b4ad70aba5af81d44637a9d07bc2855ae981a5 WHIRLPOOL 70f710d84648cb17fd928db1092d1f86a6138b6fc59c7493a7d5515b336c0a6bee4bc16accbaa5f30ad9c8adec82f3763c89e3454f5a8eac0d78ba84c1d88 5de
 EBUILD portage-2.2.8-r2.ebuild 19136 SHA256 dc5f7640568365db34c7309f2348601adfbdd17d80e9f244e8b0c49a944f5381 SHA512 d4314de9503a591ce90ba288e5265daf9325e061a8bdfd7aa4cab4d3a336c78841b709264 a7efac555f9adcfc9c084e26080c958a4ace82c348a8efddd7bbb97 WHIRLPOOL 6ff7fa99e17f9979a3097e6dc563e96934a8eb2b82d3525a10f568372da477ae7cd8d6d53470a60a63cda022de593bfcbe726f65586df6470d2277b9aea e1c01
 EBUILD portage-9999.ebuild 8319 SHA256 75d0c8a9cfef948f652310bb0291189da9405fe105e800c2fea30cd965a18309 SHA512 a8fddb73aca79da3fa1a878cb15cfcb7ca1b979b956e10326adad27f2bbc13434d1152f3590906 40290119e38787a91dc572cc7180294e5522afb6e44247a02f WHIRLPOOL 6d66092cde9159228355501861ed45f28db9c0b06234a233378cea0f610b57d477678ec4088e9372fbc4f3e69e20bc26ad02b8cecc6acda27bfcb5b908fc11f8
 MISC ChangeLog 9412 SHA256 995f81f78d09c55b727698522ab6ac39556b86c0e77c6aa5a3db696d6904e59f SHA512 c2a455d47abdaf358cd1db852716a6805b286e803403eeab435d4aa3ec159232ee0a47c715c121e961c63981e9 1f8056dc0d51f520c63f605d6591a45e0ade9b WHIRLPOOL f8df74198e17922ddcaee0a6a64bae2b8a3b2f6d49fd11488b24610578939839ec2ea75312ad1758142b5a28a94f3af0a0b4943ad362ff869f94f516df5d6071
 MISC ChangeLog-2012 133873 SHA256 157a7eb13d4cf2e2c94bfcb7e91f218af1ea88f5ff20c24a50546950a9839c94 SHA512 1241f4ccd7b31bdd340649417257b0e95202992feb66e0cd4d41fc2d35bf0baeac19680f34a831d2fef ce0c0fec03aa6e7a191f783e59cf9b93aaf266f010a3f WHIRLPOOL b027bdba0f5ccc1b3a2dc18df92d82b473af5e55536816b7177071c1ae1a89e4d50d42c2093fa1bbe28b92a42014fbefbfc9e7daa4e6426ebecbd5bffc3d479f
 MISC ChangeLog-2015 66573 SHA256 79f5ff4870713087f1a71d78a608f7c63bd2ce52e9a9642ad23d88e4ea30c807 SHA512 6df0a78314635f8fc0f721a4acda85accfaaa4eb3cefffd90e1fadeec6a3290f813780adbe572e5cd228 1be4d742627f893a88ded06413d88fb8f0c41b01890b WHIRLPOOL 7b0060b46d0f5637189f7e6f915576e2a51977f11ddfdb37b0aa0cf1dac21f869ba2ec80ff0a4e633b547668261284acbaaaf52a3015f4f676e860cf8d48280e
 MISC metadata.xml 1017 SHA256 78bf0d7e87b917ab6eb9c43835c9c688dc3c5ab0a1043608448e12e6bad715bc SHA512 40315f8375409a6e8b86b2d11dc33b28a019911348d66f05fdd1194a4f0859f3f2ebf9c6922b795194669d9 92901c1fdda4cd147100979a7f661915c4094fd35 WHIRLPOOL c2817f5c0005da6f1bf0ecb8d1bde9a6b9379e3022b9e456ee5e932974493a0454166e0113092aae7de2725b0718b3cd10e05ba25012b321644403ab0677c166

Some abstracts on the topic
https://wiki.gentoo.org/wiki/GLEP:57
https://wiki.gentoo.org/wiki/GLEP:58
https://wiki.gentoo.org/wiki/GLEP:59

[copumpkin]

@diimpp how is that different from signing a git hash/channel, given that the git commit is basically the root of a Merkle tree of the things you want signed?

More specifically, assuming we had per-file signatures and that git wasn't sufficient, who would be signing these files (pretty sure Eelco isn't prepared to put his stamp of approval on everything in nixpkgs) and what would you be checking when you checked them?

[diimpp]

@copumpkin, git signing is signing of the means, not packages itself. Technically it does the job, while packages are installed via git.

For example I would create my private repo with hg and packages tree becomes untrusted.

It's not about securing it right away, but building means to do so.
I'm actually surprised this topic doesn't interest community much, as nix pm is exist to have a means of painless package management, which should include securing packages as well.

On keys singing, here is how it's done at gentoo https://www.gentoo.org/downloads/signatures/
I've actually just enabled it on my gentoo instance, so it happened like this:

• install app-crypt/gentoo-keys
• compare via gpg installed keys with https://www.gentoo.org/downloads/signatures/
• enabled portage to check signature of packages.

I'm not sure how big community and user-base of the nixos is, so maybe it's more convenient to keep it on git level, but I don't think it can be denied, that nix pm should support this feature.

So I'm just a passer-by and can't offer you more insight into it, but you can check for more details at gentoo wiki.

[copumpkin]

@diimpp but packages in Nix are installed via git, or via a frozen git tree (called a channel).

So let's say we had signed git tags and signed channels. What would be missing then? Nix package definitions are source files. To draw a parallel, what would it mean to sign pwd.c in the glibc source code? It depends on several other source files, so signing it in isolation doesn't make much sense. We can sign the entire source tree, suggesting that the signer has no reason to believe anything in it particularly compromised, but given that it's a giant multi-person project, it's hard for any one person to put their stamp of approval on the entire source tree.

So the question boils down to what would such a signature mean to you? If we have git trees, they're already content-addressable and independently verifiable against a common source of truth. So if I say I'm operating off of NixOS/nixpkgs@2bcec42, I (can) know for sure that nobody's modified my package set in flight just as a fundamental property of git. With binary cache signing, I can furthermore know that @edolstra is willing to put his stamp of approval on the process that transformed that git revision into a set of binary artifacts so you don't have to recompile the world. Presumably he's willing to sign that because he carefully and securely administers the build farm and is confident in the build isolation machinery he built into Nix.

The only additional layer of verification outside of private channel signatures I could imagine is something like @domenkozar as NixOS release manager saying "yes, this is indeed the set of packages I was intending to put into release 15.09", with the understanding that it's in no way a security guarantee. All such a signature would tell you is that our release is actually what he thought it was. Is that a meaningful diff in security stance?

[diimpp]

@copumpkin,
It does, but it's not set in stone, security shouldn't be hardcoded to the one of the possible ways to deliver packages.

Yes, this example was used before. It's glibc job to provide tar with signature, it's nix pm job to ensure this correct tar would be used. I don't see why it means that whole packages tree should be signed as whole. There would glibc signed package and whatever_signed package, that uses glibc. Each of them signed separately.

I'm not saying, that "in repository package signing" does introduce any more security for nixos as is,
I'm saying it's a non-opinionated way to solve package security, while using git is a opinionated way to do it.

That's all. :)

[code-ape]

I know this issue appears untouched for some time now, but as a personal user of Nix and NixOS I wanted to post that I find it frightening there is no aspect of trust throughout the lifecycle from code creation to installation. I am happy to provide knowledge and recommendations on the matter if that would be something the community is interested in pursuing. But if not then, sadly, I will leave Nix and go to Arch or Gentoo for this reason.

[copumpkin]

@code-ape what specifically would you like signed that isn't signed today? All Nix packages built by Hydra are cryptographically signed and all Nix fixed-output derivations in the system have a sha256 that is verified by construction, and the entire system is deterministic unlike most other packaging systems. Beyond that, as I explain two messages above, other notions of signing have far more nebulous semantics.

[code-ape]

@copumpkin As far as I can tell this still places the Github repo as the source of trust, cryptographically. So when I decide to install an application, I ultimately am trusting Github and the repo maintainers that it is secure and the build system that the build hasn’t been tampered with. As it’s been said, it is “unlikely” that Github would be compromised, and I have no doubt the repo maintainers work very hard to ensure that PRs for nixpkgs are not a risk to users. Likewise, the build system is trusted for turning those into binaries / packages.

However, this means a blanket level of trust is assumed. My biggest concern is that, as far as I can tell, is backs the system into a corner if any of these are compromised? This leads to questions like:

1. What is the response to malicious code being added to the repo? Do you simply reject the entire repo then?
2. How does the build system know the code it pulled was written by trusted sources?
3. Do users have no option but to trust the nixpkgs repo entirely or not at all? It’s content is added by a plethora of entities. If it comes out that one of them has been hacked, how might I reject trust of anything published by such an entity?

I know this may sound extreme to some, but some users and organizations need satisfying answers to these questions so to ensure security. I don’t have all the answer, and don’t even expect there to be immediate answers to their questions (trust in security is difficult).

EDIT: As a final thought, if builds are deterministic, then perhaps a system could be created where a signed hash of the build it added to packages. Fully placing trust between the code publisher and user while meaning any tampering would break this trust.

[nejucomo]

Hello, new nix (atop debian) user here. I'd also like to know that software I install with nix is covered by a sensible signature scheme.

The sensible scheme I'd like to see is that every channel update (or revision) is signed and the local configuration for nix-channel has an associated public key(s) for each channel, and refuses to apply an update if it is not signed by that key. (I'm assuming channels specify packages, which specify all dependencies with cryptographic hashes. I don't yet understand how binary cache signatures or hashes fit into nix.)

Sometimes I find discussions about package signing get muddled about "who is trusted" and "preventing compromise". I think it helps to flip things around and imagine that a backdoor is discovered that impacts one or more nix users. Those users or other investigators need to know how the backdoor was inserted.

If channels were updated over http with no signatures, the answer is very broad. If channels are updated over https and come from github using git, with no signatures, the answer is still quite broad: anyone who compromised any of many numerous Certificate Authorities (not uncommon), OR anyone who compromised github, OR anyone who was able to push to the git repo, OR any of the downstream source code producers that fed into the nix package definitions.

If we require channels to be signed, the answer becomes more narrow and we can now exclude all Certificate Authorities, github, and people with git push access who don't control the signature key.

Now here's where we can improve things even more. If there are dozens of people with the authority to publish channel updates because nixpkgs is an open project with many collaborators, let's require each of those people to publish an individual key, and nix-channel can use any of their keys. This is where the "prevention thinking" falls down: it wouldn't matter if those dozens of people shared a single channel signing key or not in terms of preventing a backdoor, but if they each have individual keys and a backdoor is discovered only one of them is now suspect.

Two issues I don't have a proposal for yet because I don't understand nix well enough:

1. How are binary builds authenticated? I advocate that a channel transitively refers to binary builds by their cryptographic hash. Probably the most natural way would be for a package to express both the build-from-source nix expression and also name a specific binary hash. No signatures are necessary for the binary, because instead the signer of the channel is attesting that a given binary is indeed the correct one for a package.

2. If every authorized channel updater has an individual key, how does nix-channel learn about new public keys when new collaborators show up, or remove public keys when old collaborators leave? A relatively easy to reason about scheme would have a second layer public key which has the authority to update the roster of acceptable keys, and those updates would need to be transmitted to nix-channel during the course of typical channel updates.

[7c6f434c]

You need CA + routing compromise (quite feasible, but a bit harder). Also, in the current model everyone assumes that everything on master has been put there by some committer (on their own or via review). Careful enough GitHub compromise is gameover however you validate channel bumps (there are too many commit to verify by a single human). I am not sure what will happen to GitHub once a SHA1 compromise (out there, $100 000/instance or less) gets applied to Git formats…

Given that the project bottleneck is pull request reviewing, and it is sometimes faster to do on GitHub (line comments, for example)… oops.

Also, Hydra compromise — or a generic AWS compromise, as Hydra uses AWS instances — is near-certain gameover, because you will succeed to inject code to a nontrivial fraction of committers' machines. Whatever you do with binary output authentication, until you have bit-perfect builds (we don't yet) and verify everything on builders with different control…

[edolstra]

Closing, see also NixOS/rfcs#34.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/any-interest-in-checkings-signatures-while-building-packages/8918/1