New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC 0005] Nix encryption #5

Open
wants to merge 6 commits into
base: master
from

Conversation

@edolstra
Member

edolstra commented Mar 28, 2017

A proposal for supporting secrets in the Nix store.

Rendered: https://github.com/edolstra/rfcs/blob/nix-encryption/rfcs/0005-nix-encryption.md

edolstra added some commits Mar 28, 2017

@zimbatm

Well written and thought out first draft 👍

* A command `nix decrypt` to decrypt files containing encrypted
strings produced by `encryptString`. This command searches for
`<{|nixcrypt:...|}>` fragments and decrypts them using a decryption

This comment has been minimized.

@zimbatm

zimbatm Mar 28, 2017

Member

Has <{|nixcrypt: been selected to make it easier to do find-and-replace at the decryption phase or are there other considerations? I'm wondering if nixcrypt::<base64-salt> could be enough. It's a bit of a detail though.

@zimbatm

zimbatm Mar 28, 2017

Member

Has <{|nixcrypt: been selected to make it easier to do find-and-replace at the decryption phase or are there other considerations? I'm wondering if nixcrypt::<base64-salt> could be enough. It's a bit of a detail though.

passing something like `--config /nix/store/.../foo.conf` to the
daemon. Instead, we have to decrypt the configuration file to some
suitably secure temporary location in a pre-start script, and then
pass that location (e.g. `--config /tmp/.../foo.conf`).

This comment has been minimized.

@zimbatm

zimbatm Mar 28, 2017

Member

The biggest drawback here is if another file depends on that file.

{
nginxConf = writeFile "nginx.conf" "include ${vserverConf};";
vserverConfg = writeFile "vserver.conf" " my secret is here ";
}

Would it be possible to taint the derivation output with an "encrypted" attribute and then have a mechanism that would prevent referencing it? (unless if it's to decrypt it) The tainting could also be used as a signal to not push it to public stores.

@zimbatm

zimbatm Mar 28, 2017

Member

The biggest drawback here is if another file depends on that file.

{
nginxConf = writeFile "nginx.conf" "include ${vserverConf};";
vserverConfg = writeFile "vserver.conf" " my secret is here ";
}

Would it be possible to taint the derivation output with an "encrypted" attribute and then have a mechanism that would prevent referencing it? (unless if it's to decrypt it) The tainting could also be used as a signal to not push it to public stores.

This comment has been minimized.

@teh

teh Mar 31, 2017

I agree. We often have chains of config files where changing a leaf bubbles up correctly all the way to the top thanks to nix. I'm not sure how many of our files contain secrets though. So this might actually be an OK price to pay in practice.

@teh

teh Mar 31, 2017

I agree. We often have chains of config files where changing a leaf bubbles up correctly all the way to the top thanks to nix. I'm not sure how many of our files contain secrets though. So this might actually be an OK price to pay in practice.

# Decrypt the configuration file.
${config.nix.package}/bin/nix-store \
--decrypt ${toString <nixos-store-key>} \
${configFile} > /run/secret/foo.conf

This comment has been minimized.

@zimbatm

zimbatm Mar 28, 2017

Member

Since the configuration has to be generated at runtime, what difference does it make to get the secret from the nix store or another source like hashicorp vault, /run/keys/* or an environment variable?

@zimbatm

zimbatm Mar 28, 2017

Member

Since the configuration has to be generated at runtime, what difference does it make to get the secret from the nix store or another source like hashicorp vault, /run/keys/* or an environment variable?

This comment has been minimized.

@arianvp

arianvp Mar 29, 2017

Exactly my thought. Currently, nixops supports delivering keys out of band a well and saves them in /run/keys . I think the actual hard part is not solved by this RFC as it gives the same level of flexibility as what nixops currently already delivers! The real hard part is that we need to rewrite dozens of NixOS modules that directly store secrets such that they can retrieve their secrets from disk. And we should discourage or maybe totally deprecate any kind of other usage ...

Perhaps a larger part of the proposal isn't actually the encryption builtin, but actually rewriting modules in such way that they support this paradigm (and the nixops / vault paradigm).

Actually, I am in favor to make the nixops feature more easy to use instead of introducing an encryption primitive. I think it clashes too much with reproducibility to store nonced encrypted blobs in the nix store.

@arianvp

arianvp Mar 29, 2017

Exactly my thought. Currently, nixops supports delivering keys out of band a well and saves them in /run/keys . I think the actual hard part is not solved by this RFC as it gives the same level of flexibility as what nixops currently already delivers! The real hard part is that we need to rewrite dozens of NixOS modules that directly store secrets such that they can retrieve their secrets from disk. And we should discourage or maybe totally deprecate any kind of other usage ...

Perhaps a larger part of the proposal isn't actually the encryption builtin, but actually rewriting modules in such way that they support this paradigm (and the nixops / vault paradigm).

Actually, I am in favor to make the nixops feature more easy to use instead of introducing an encryption primitive. I think it clashes too much with reproducibility to store nonced encrypted blobs in the nix store.

This comment has been minimized.

@zimbatm

zimbatm Mar 29, 2017

Member

There are two things here I think: (1) an interface to declare that the secret should exist (or other kind of states actually). Potentially it could also have an initialization script if the secret can be dynamically generated. (2) a way to compose existing nix derivations with the state.

For (2) it would be cool if it integrated with the language. Potentially we could have a subset of nix that can be evaluated at runtime.

@zimbatm

zimbatm Mar 29, 2017

Member

There are two things here I think: (1) an interface to declare that the secret should exist (or other kind of states actually). Potentially it could also have an initialization script if the secret can be dynamically generated. (2) a way to compose existing nix derivations with the state.

For (2) it would be cool if it integrated with the language. Potentially we could have a subset of nix that can be evaluated at runtime.

This comment has been minimized.

@danbst

danbst Apr 1, 2017

Will FUSE wrapper on encrypted files solve this problem of requiring rewrite of modules?

@danbst

danbst Apr 1, 2017

Will FUSE wrapper on encrypted files solve this problem of requiring rewrite of modules?

This comment has been minimized.

@Gabriel439

Gabriel439 Jun 6, 2017

@zimbatm @arianvp: My understanding is that the main benefit of this approach is that some programs might not be equipped to read secrets from a file (for example, a configuration file that only accepts a secret as an inline string field). I don't know any such programs off the top of my head that are like this, though, but I guess it could happen. Are there any concrete examples where this is the case?

However, I that most programs can accept secrets from protected files outside the /nix/store and we could easily fix most NixOS services to use this approach today without any changes to Nix.

@Gabriel439

Gabriel439 Jun 6, 2017

@zimbatm @arianvp: My understanding is that the main benefit of this approach is that some programs might not be equipped to read secrets from a file (for example, a configuration file that only accepts a secret as an inline string field). I don't know any such programs off the top of my head that are like this, though, but I guess it could happen. Are there any concrete examples where this is the case?

However, I that most programs can accept secrets from protected files outside the /nix/store and we could easily fix most NixOS services to use this approach today without any changes to Nix.

The assumption is that `<nixos-store-key>` resolves to something like
`/var/lib/nixos/store-key`, which should contain a key generated by
`nix generate-key` and should obviously be readable by root only.

This comment has been minimized.

@zimbatm

zimbatm Mar 28, 2017

Member

Let's say I have 3 different servers with different configs and a Hydra that pushes their config to a binary cache. Wouldn't one compromised server be able to read all the server's config from the binary cache?

@zimbatm

zimbatm Mar 28, 2017

Member

Let's say I have 3 different servers with different configs and a Hydra that pushes their config to a binary cache. Wouldn't one compromised server be able to read all the server's config from the binary cache?

This comment has been minimized.

@Gabriel439

Gabriel439 Jun 6, 2017

@zimbatm: I believe this is safe in the presence of both binary caches and distributed builds.

For binary caches, the actual cached configuration only stores encrypted secrets, so even if you mirror the configuration to other /nix/stores they won't be able to decrypt these secrets unless they have access the same key.

For distributed builds, files on the NIX_PATH are not copied to distributed build slaves, so if the <nixos-store-key> path is missing on the build slave machine(s) then the build will just fail.

@Gabriel439

Gabriel439 Jun 6, 2017

@zimbatm: I believe this is safe in the presence of both binary caches and distributed builds.

For binary caches, the actual cached configuration only stores encrypted secrets, so even if you mirror the configuration to other /nix/stores they won't be able to decrypt these secrets unless they have access the same key.

For distributed builds, files on the NIX_PATH are not copied to distributed build slaves, so if the <nixos-store-key> path is missing on the build slave machine(s) then the build will just fail.

Show outdated Hide outdated rfcs/0005-nix-encryption.md Outdated
@domenkozar

This comment has been minimized.

Show comment
Hide comment
@domenkozar

domenkozar Apr 3, 2017

Member

I really like the proposal, it doesn't fight Nix design but rather embraces it.

My biggest concern is usability. I'd like for us to have more conventions over configurations where possible.

So encryptString first argument would accept the name of the key to be used, for example:

builtins.encryptString "nixpkgs" cfg.password

Then in nixpkgs we would have encryptString = builtins.encryptString "nixpkgs" to use one key for all encrypted strings.

At runtime we could do:

nix generate-key <key-name> where key-name is required
and nix decrypt <key-name> where `key-name* is optional and if not given it would decrypt all encrypted keys in the store.

The keys would get stored in /nix/var/nix/keys/<key-name>.

I agree with @teh as with impure derivations, nothing would be able to reference the encrypted store path.

Member

domenkozar commented Apr 3, 2017

I really like the proposal, it doesn't fight Nix design but rather embraces it.

My biggest concern is usability. I'd like for us to have more conventions over configurations where possible.

So encryptString first argument would accept the name of the key to be used, for example:

builtins.encryptString "nixpkgs" cfg.password

Then in nixpkgs we would have encryptString = builtins.encryptString "nixpkgs" to use one key for all encrypted strings.

At runtime we could do:

nix generate-key <key-name> where key-name is required
and nix decrypt <key-name> where `key-name* is optional and if not given it would decrypt all encrypted keys in the store.

The keys would get stored in /nix/var/nix/keys/<key-name>.

I agree with @teh as with impure derivations, nothing would be able to reference the encrypted store path.

We could replace the nonce with a deterministic value, but it's not
entirely clear what the cryptographic implications are. At the very
least, it allows attackers to obverse that a secret has changed, or

This comment has been minimized.

@taktoa

taktoa Apr 3, 2017

typo: s/obverse/observe/g

@taktoa

taktoa Apr 3, 2017

typo: s/obverse/observe/g

@zimbatm zimbatm changed the title from Nix encryption to [RFC 005] Nix encryption Apr 4, 2017

@edolstra edolstra changed the title from [RFC 005] Nix encryption to [RFC 0005] Nix encryption Apr 10, 2017

configFile = pkgs.writeText "foo.conf"
''
# Store the password of the foo service in encrypted form.
password=${builtins.encryptString <nixos-store-key> cfg.password}

This comment has been minimized.

@steveeJ

steveeJ Apr 12, 2017

My understanding is that cfg.password is cleartext, but it's not clear to me where it is defined and/or how the user will provide this.

@steveeJ

steveeJ Apr 12, 2017

My understanding is that cfg.password is cleartext, but it's not clear to me where it is defined and/or how the user will provide this.

This comment has been minimized.

@Gabriel439

Gabriel439 Jun 6, 2017

I'm aware of two solutions to providing the password:

  • Approach 1: The user does not supply a secret at all and the derivation auto-generates the secret at startup. The secret is generated so that only root can access it and the user has to ssh into the machine as root to obtain the credential if they need it to log in.
  • Approach 2: The user logs into the machine after deploy and deposits the secret in a preconfigured location by the user after deploy and the program/service waits or fails until the secret is available.
@Gabriel439

Gabriel439 Jun 6, 2017

I'm aware of two solutions to providing the password:

  • Approach 1: The user does not supply a secret at all and the derivation auto-generates the secret at startup. The secret is generated so that only root can access it and the user has to ssh into the machine as root to obtain the credential if they need it to log in.
  • Approach 2: The user logs into the machine after deploy and deposits the secret in a preconfigured location by the user after deploy and the program/service waits or fails until the secret is available.
@taktoa

This comment has been minimized.

Show comment
Hide comment
@taktoa

taktoa Apr 12, 2017

I think it might be a good idea to have an optional libsecret backend for secret storage, since it seems to be the current state of the art library for secret storage across various desktop environments. Of course, since glib is in the dependencies of libsecret, we would not want Nix to necessarily depend on it, but there are a few ways to avoid this issue:

  1. Nix could delegate secret retrieval to a subprocess, the same way git credential currently works, so you could plug in any backend you want. One point in favor of this idea is that this is probably also just good hygiene since processes have isolated memory spaces (and there could be bugs in Nix).
  2. We could use conditional compilation to avoid the dependency on libsecret whenever we are bootstrapping nix, but enable the libsecret backend in all other cases, so that e.g.: there would be a nixBootstrap attribute and a nixFull attribute.

Having libsecret support would make it impossible to successfully evaluate a Nix expression that uses it until the keyring is unlocked. This may or may not be desirable.

taktoa commented Apr 12, 2017

I think it might be a good idea to have an optional libsecret backend for secret storage, since it seems to be the current state of the art library for secret storage across various desktop environments. Of course, since glib is in the dependencies of libsecret, we would not want Nix to necessarily depend on it, but there are a few ways to avoid this issue:

  1. Nix could delegate secret retrieval to a subprocess, the same way git credential currently works, so you could plug in any backend you want. One point in favor of this idea is that this is probably also just good hygiene since processes have isolated memory spaces (and there could be bugs in Nix).
  2. We could use conditional compilation to avoid the dependency on libsecret whenever we are bootstrapping nix, but enable the libsecret backend in all other cases, so that e.g.: there would be a nixBootstrap attribute and a nixFull attribute.

Having libsecret support would make it impossible to successfully evaluate a Nix expression that uses it until the keyring is unlocked. This may or may not be desirable.

@domenkozar

This comment has been minimized.

Show comment
Hide comment
@domenkozar

domenkozar Apr 14, 2017

Member

How would NixOps integration look like? Run nix decrypt before activation, using supplied password from local machine?

Member

domenkozar commented Apr 14, 2017

How would NixOps integration look like? Run nix decrypt before activation, using supplied password from local machine?

''
# Decrypt the configuration file.
${config.nix.package}/bin/nix-store \
--decrypt ${toString <nixos-store-key>} \

This comment has been minimized.

@michalrus

michalrus May 2, 2017

If somebody has access to the file system (before this solution), they also have access to /proc and can read this argv, stealing the key…

@michalrus

michalrus May 2, 2017

If somebody has access to the file system (before this solution), they also have access to /proc and can read this argv, stealing the key…

This comment has been minimized.

@edolstra

edolstra May 2, 2017

Member

No, because this passes the path to the key, not the key itself.

@edolstra

edolstra May 2, 2017

Member

No, because this passes the path to the key, not the key itself.

carefully audit that nothing in the daemon can leak the contents of
store paths.
* Users should be allowed to export closures that they built, even

This comment has been minimized.

@FRidh

FRidh May 6, 2017

Member

Perhaps instead of checking the ACL's of the store paths, use the evaluator to determine whether the user can copy the closure?

@FRidh

FRidh May 6, 2017

Member

Perhaps instead of checking the ACL's of the store paths, use the evaluator to determine whether the user can copy the closure?

This comment has been minimized.

@FRidh

FRidh May 6, 2017

Member

Which would require tracking the original expressions or .drv's which we do not do.

@FRidh

FRidh May 6, 2017

Member

Which would require tracking the original expressions or .drv's which we do not do.

@eternaleye

This comment has been minimized.

Show comment
Hide comment
@eternaleye

eternaleye May 6, 2017

So, this was brought up in the #rust-crypto IRC channel (on moznet), and I figured I'd reproduce what I said here.

<eternaleye> If [you] want to do reproducible encryption, then you need a way of either making it so the nonce being repeated doesn't matter (MRAE), or you need a way to ensure that the nonce, while deterministic, is never the same for distinct AD/plaintext tuples (which one does by way of an SIV-like construct, which... is an MRAE.)
<eternaleye> So I'd suggest looking into AES-SIV and HS1-SIV
<eternaleye> Neither of which, sadly, is provided by libsodium
<eternaleye> You may be able to build HS1-SIV using what's in libsodium? But I'd be wary of doing so.

The issue here is that crypto_secretbox (like most other AEADs) fails very very badly if a nonce is ever repeated with a different AD and/or message. AEADs that do not fail in this manner (and instead meet the optimal security of deterministic encryption[1]) are called "Misuse-Resistant", or MRAE.

This definition can be found in (and is expanded upon by) several papers, most involving Rogaway. These include the initial paper on SIV, the paper that introduced the AEZ algorithm, a paper describing strong online-encryption constructions built upon MRAE, and a paper devoted entirely to the definition and its properties.

By comparison, crypto_secretbox is based on a stream cipher using the nonce directly as an IV; as a result, there is a loss of confidentiality if a nonce is reused - specifically, the XOR of the plaintexts of the two messages leaks.

Similarly, it uses Poly1305 as its authenticator, which relies on nonce uniqueness. As a result, integrity also suffers if nonces are reused.

HS1-SIV is an AEAD that meets the MRAE security definition, and is built from ChaCha20 and Poly1305, so if you trust DJB's cryptographic primitives, it's a reasonable option. It was submitted to the CAESAR competition; a detailed description can be found here.

Alternately, AES-SIV relies on nothing except the AES block cipher itself. As a result, rather than relying on two primitives, one can rely on only a single primitive. In some cases, this might be a better tradeoff. A detailed description can be found in RFC 5297; it uses CMAC internally, which is specified in RFC 4493.

In short, by using an MRAE, Nix can achieve both the goal of secure, encrypted secret data in the store and bitwise-deterministic outputs.

[1] Optimally, there is no loss of integrity regardless of any properties of the (nonce, AD, message) tuple, there is no loss of confidentiality so long as the same (nonce, AD, message) tuple is not encrypted multiple times, and the only loss of confidentiality in that case is the one-bit information leak "these two messages were identical."

eternaleye commented May 6, 2017

So, this was brought up in the #rust-crypto IRC channel (on moznet), and I figured I'd reproduce what I said here.

<eternaleye> If [you] want to do reproducible encryption, then you need a way of either making it so the nonce being repeated doesn't matter (MRAE), or you need a way to ensure that the nonce, while deterministic, is never the same for distinct AD/plaintext tuples (which one does by way of an SIV-like construct, which... is an MRAE.)
<eternaleye> So I'd suggest looking into AES-SIV and HS1-SIV
<eternaleye> Neither of which, sadly, is provided by libsodium
<eternaleye> You may be able to build HS1-SIV using what's in libsodium? But I'd be wary of doing so.

The issue here is that crypto_secretbox (like most other AEADs) fails very very badly if a nonce is ever repeated with a different AD and/or message. AEADs that do not fail in this manner (and instead meet the optimal security of deterministic encryption[1]) are called "Misuse-Resistant", or MRAE.

This definition can be found in (and is expanded upon by) several papers, most involving Rogaway. These include the initial paper on SIV, the paper that introduced the AEZ algorithm, a paper describing strong online-encryption constructions built upon MRAE, and a paper devoted entirely to the definition and its properties.

By comparison, crypto_secretbox is based on a stream cipher using the nonce directly as an IV; as a result, there is a loss of confidentiality if a nonce is reused - specifically, the XOR of the plaintexts of the two messages leaks.

Similarly, it uses Poly1305 as its authenticator, which relies on nonce uniqueness. As a result, integrity also suffers if nonces are reused.

HS1-SIV is an AEAD that meets the MRAE security definition, and is built from ChaCha20 and Poly1305, so if you trust DJB's cryptographic primitives, it's a reasonable option. It was submitted to the CAESAR competition; a detailed description can be found here.

Alternately, AES-SIV relies on nothing except the AES block cipher itself. As a result, rather than relying on two primitives, one can rely on only a single primitive. In some cases, this might be a better tradeoff. A detailed description can be found in RFC 5297; it uses CMAC internally, which is specified in RFC 4493.

In short, by using an MRAE, Nix can achieve both the goal of secure, encrypted secret data in the store and bitwise-deterministic outputs.

[1] Optimally, there is no loss of integrity regardless of any properties of the (nonce, AD, message) tuple, there is no loss of confidentiality so long as the same (nonce, AD, message) tuple is not encrypted multiple times, and the only loss of confidentiality in that case is the one-bit information leak "these two messages were identical."

@Ekleog

This comment has been minimized.

Show comment
Hide comment
@Ekleog

Ekleog May 7, 2017

Ok, so... First, disclaimer: I'm not a cryptographer.

That said, after looking a bit further into SIV via http://csrc.nist.gov/groups/ST/toolkit/BCM/documents/proposedmodes/siv/siv.pdf , it looks like it's possible to implement (a variant of) it in a simple way by using the fact that we need no associated cleartext data along with the encrypted data: in pseudocode,

encrypt(key1, key2, value) =
    let nonce = MAC(key1, value) in
    nonce || CIPHER(key2, value, nonce)
decrypt(key1, key2, value) =
    let nonce || cipher = value in
    let clear = DECIPHER(key2, cipher, nonce) in
    if MAC(key1, value) != nonce then
        FAIL()
    else
        clear

(note that using this is not optimal from a performance point of view, as CIPHER will likely store the nonce someplace we could reuse it; real SIV uses it only as an IV that can be checked then but this would require internal knowledge of the encryption algorithms, thing that libsodium doesn't provide us with)

The thing to note is that there is no real violation of (DE)CIPHER's requirement for a nonce is not reused twice: it will be reused multiple times, but always with the same keys and values, which means that it will necessarily return the same result, leaking no information about the cleartext.

Now, the issue remaining is what the paper I linked uses as MAC, CIPHER and DECIPHER operations: they use MAC(k, v) = AES-CMAC(k, C_k ^ v) (with v padded with 10* if need be and C_k a constant depending on k), and (DE)CIPHER that are AES-CTR.

Here is the leap of faith: I think implementing a SIV mode of operation using MAC(k, v) = Truncate-192(HMAC-SHA256(k, v)) and (DE)CIPHER that are XSalsa20 (that is, the primitives libsodium offers) should not be an issue, as the overall idea of the algorithm is preserved, only the MAC and (DE)CIPHER primitives are changed.
The reason why I am saying this is a leap of faith is that its security is based on my understanding of https://www.iacr.org/archive/eurocrypt2006/40040377/40040377.pdf , which defines the requirements of MAC and (DE)CIPHER for SIV mode.
From what I understand, the requirements are that MAC be a PRF from vectors of bitstrings (OK for Truncate-192(HMAC-SHA256(k, v)) as we know all our vectors are going to be exactly 1 bitstring, based on the fact we have no header), and (DE)CIPHER be an IV-based encryption scheme.
This second point is not really verified: technically, the nonce in XSalsa20 is not an IV. Now, I can't see in the proof of correctness where this assumption is used, then I may be wrong, especially since I haven't had time to read and understand the whole paper and every argument in it.

Now, to what attacks does such a scheme open way?

First one and most obvious: as nix provides an encryption oracle (by building a derivation containing to-be-encrypted data as a user), to attacks bruteforcing the content of the secret and comparing returned values. However, this is actually required by any deterministic encryption scheme we could put in place. Solutions to mitigate this could be either to use one (bunch of) symmetric key(s) per user (the preferred in my opinion, but most likely way harder to implement) or to rate-limit encryption to a point where bruteforcing is not an issue.

Second one is that it's possible to know when the same thing is encrypted twice. It is possible to mitigate this by adding some additional data in the encryption process, like the filename wherein the encrypted data will reside and the index of the encrypted data inside the file. Or just encrypt entire files at once, then the leaked information would only be whether two files are exactly equal.

Third, as the encryption process becomes fully deterministic, side-channel attacks may become easier to accomplish. I don't see any way of mitigating this, as it is an issue naturally associated with side-channel attacks.

Fourth, I may have completely misunderstood https://www.iacr.org/archive/eurocrypt2006/40040377/40040377.pdf and told complete nonsense.

As this is likely to be way easier to implement than a full-featured HS1-SIV (which, as I learned recently, was rejected after round2 of CAESAR, despite the judges not giving any compelling argument against its use) using only libsodium's primitives, I hope this may help!

Ekleog commented May 7, 2017

Ok, so... First, disclaimer: I'm not a cryptographer.

That said, after looking a bit further into SIV via http://csrc.nist.gov/groups/ST/toolkit/BCM/documents/proposedmodes/siv/siv.pdf , it looks like it's possible to implement (a variant of) it in a simple way by using the fact that we need no associated cleartext data along with the encrypted data: in pseudocode,

encrypt(key1, key2, value) =
    let nonce = MAC(key1, value) in
    nonce || CIPHER(key2, value, nonce)
decrypt(key1, key2, value) =
    let nonce || cipher = value in
    let clear = DECIPHER(key2, cipher, nonce) in
    if MAC(key1, value) != nonce then
        FAIL()
    else
        clear

(note that using this is not optimal from a performance point of view, as CIPHER will likely store the nonce someplace we could reuse it; real SIV uses it only as an IV that can be checked then but this would require internal knowledge of the encryption algorithms, thing that libsodium doesn't provide us with)

The thing to note is that there is no real violation of (DE)CIPHER's requirement for a nonce is not reused twice: it will be reused multiple times, but always with the same keys and values, which means that it will necessarily return the same result, leaking no information about the cleartext.

Now, the issue remaining is what the paper I linked uses as MAC, CIPHER and DECIPHER operations: they use MAC(k, v) = AES-CMAC(k, C_k ^ v) (with v padded with 10* if need be and C_k a constant depending on k), and (DE)CIPHER that are AES-CTR.

Here is the leap of faith: I think implementing a SIV mode of operation using MAC(k, v) = Truncate-192(HMAC-SHA256(k, v)) and (DE)CIPHER that are XSalsa20 (that is, the primitives libsodium offers) should not be an issue, as the overall idea of the algorithm is preserved, only the MAC and (DE)CIPHER primitives are changed.
The reason why I am saying this is a leap of faith is that its security is based on my understanding of https://www.iacr.org/archive/eurocrypt2006/40040377/40040377.pdf , which defines the requirements of MAC and (DE)CIPHER for SIV mode.
From what I understand, the requirements are that MAC be a PRF from vectors of bitstrings (OK for Truncate-192(HMAC-SHA256(k, v)) as we know all our vectors are going to be exactly 1 bitstring, based on the fact we have no header), and (DE)CIPHER be an IV-based encryption scheme.
This second point is not really verified: technically, the nonce in XSalsa20 is not an IV. Now, I can't see in the proof of correctness where this assumption is used, then I may be wrong, especially since I haven't had time to read and understand the whole paper and every argument in it.

Now, to what attacks does such a scheme open way?

First one and most obvious: as nix provides an encryption oracle (by building a derivation containing to-be-encrypted data as a user), to attacks bruteforcing the content of the secret and comparing returned values. However, this is actually required by any deterministic encryption scheme we could put in place. Solutions to mitigate this could be either to use one (bunch of) symmetric key(s) per user (the preferred in my opinion, but most likely way harder to implement) or to rate-limit encryption to a point where bruteforcing is not an issue.

Second one is that it's possible to know when the same thing is encrypted twice. It is possible to mitigate this by adding some additional data in the encryption process, like the filename wherein the encrypted data will reside and the index of the encrypted data inside the file. Or just encrypt entire files at once, then the leaked information would only be whether two files are exactly equal.

Third, as the encryption process becomes fully deterministic, side-channel attacks may become easier to accomplish. I don't see any way of mitigating this, as it is an issue naturally associated with side-channel attacks.

Fourth, I may have completely misunderstood https://www.iacr.org/archive/eurocrypt2006/40040377/40040377.pdf and told complete nonsense.

As this is likely to be way easier to implement than a full-featured HS1-SIV (which, as I learned recently, was rejected after round2 of CAESAR, despite the judges not giving any compelling argument against its use) using only libsodium's primitives, I hope this may help!

@paragonie-scott

This comment has been minimized.

Show comment
Hide comment
@paragonie-scott

paragonie-scott May 7, 2017

I think I'm missing something: Why do you want reproducible encryption?

Is this for unit tests? If so, that can be handled totally separately. Your nonce can be a counter (i.e. start with \x00\x00\x00... then \x01\x00\x00... then \x02\x00\x00 using sodium_increment(). If you have the same nonce-key pair, encryption becomes deterministic.

If you're concerned about weak RNGs, here's a simpler idea:

  1. Calculate crypto_generichash() of the message.
  2. Feed the hash into the kernel's entropy pool as an additional seed. (Write it to /dev/random.)
  3. Just use a random nonce as you normally would.

If your RNG is weak, you'll get NMR by virtue of the RNG state being seeded by the hash of the message. If the RNG is strong, feeding the hash of the message into your RNG doesn't harm anything. If you're concerned about the RNG leaking data about your message, you're only feeding a hash of the message and not the message itself.

A lot of these SIV schemes add a lot of additional complexity (and virtually lock you into violating the cryptographic doom principle) for a small win only in corner cases.

paragonie-scott commented May 7, 2017

I think I'm missing something: Why do you want reproducible encryption?

Is this for unit tests? If so, that can be handled totally separately. Your nonce can be a counter (i.e. start with \x00\x00\x00... then \x01\x00\x00... then \x02\x00\x00 using sodium_increment(). If you have the same nonce-key pair, encryption becomes deterministic.

If you're concerned about weak RNGs, here's a simpler idea:

  1. Calculate crypto_generichash() of the message.
  2. Feed the hash into the kernel's entropy pool as an additional seed. (Write it to /dev/random.)
  3. Just use a random nonce as you normally would.

If your RNG is weak, you'll get NMR by virtue of the RNG state being seeded by the hash of the message. If the RNG is strong, feeding the hash of the message into your RNG doesn't harm anything. If you're concerned about the RNG leaking data about your message, you're only feeding a hash of the message and not the message itself.

A lot of these SIV schemes add a lot of additional complexity (and virtually lock you into violating the cryptographic doom principle) for a small win only in corner cases.

@grahamc

This comment has been minimized.

Show comment
Hide comment
@grahamc

grahamc May 7, 2017

Member

For context I've solicited cryptography experts to review this proposal, and specifically asked my friend @paragonie-scott to take a look. @paragonie-scott, elsewhere in Nix is a strong desire and effort to have bitwise reproducibility. If that is at odds with having our mechanism remain secure, then it would be good to know that early.

Would it be a bad idea to use a hash of the secret input to create & cache the store path?

Member

grahamc commented May 7, 2017

For context I've solicited cryptography experts to review this proposal, and specifically asked my friend @paragonie-scott to take a look. @paragonie-scott, elsewhere in Nix is a strong desire and effort to have bitwise reproducibility. If that is at odds with having our mechanism remain secure, then it would be good to know that early.

Would it be a bad idea to use a hash of the secret input to create & cache the store path?

```
The above can also be extended to support access by non-root users,
e.g.

This comment has been minimized.

@grahamc

grahamc May 7, 2017

Member

This seems to be a critically important component, so individual services don't need (poorly) re-implement this behavior.

@grahamc

grahamc May 7, 2017

Member

This seems to be a critically important component, so individual services don't need (poorly) re-implement this behavior.

This comment has been minimized.

@nbp

nbp May 26, 2017

Member

I agree, and ideally this would need some way to get a path for each configuration file. For security reasons systemd services can switch users, and we would want these files to be read only by these users.

I would prefer to avoid FUSE as the first iteration, as this would imply trusting more code than necessary. (we are talking about the manipulation of secrets)

I think we could have something like:

{ security.encryptedFiles.file = {
    user = "foo"; /* uses users.users.<name>.uid or assert if missing */
    file = ...;
  };
  systemd.services.wpa_supplicant.serviceConfig.Exec =
     "wpa_supplicant -c ${security.encryptedFiles.file}";
}

The problem of this scheme is that the author of the module would have to think ahead that the file has to be secure.

For example, simple configuration can provide a default configuration based on pam, but leave complex option which could be provided via extraConfig to set password in the configuration files. In such case we might want as a user to retrofit security in an expression which did not expected it first.

To handle such cases, I think we should make a submodule which is global to NixOS to let people register secure files anywhere, by adding an option type which would be a submodule for declaring files. This submodule would then be in charge of finding the proper renaming of the files path, and these entries could be listed in the security.encryptedFiles option to let the activation script decipher these files as root, before changing the ownership. Thus a service would look like:

{ lib, config, ... }:
{ options.services.foo = {
    enable = ...;
    extraConfig = ...;
    configFile = lib.mkOption { type = lib.types.file; };
  };
  config = with config.services.foo; lib.mkIf enable {
    security.secureFiles = [ configFile ]; # Can be added by another module
    systemd.services.foo.serviceConfig.Exec =
       "command --conf ${configFile}";
    services.foo.configFile.content = extraConfig;
  };
}

Doing it that way, ensure that the security.maybeSecureFiles can be registered by anybody who needs to set the services.foo.configFile.secure = true;. Thus, the submodule lib.types.file will contain the logic to translate any file from a store path to a /var or /tmp path if the file has the secure flag. The decoding would be handled by the security.secureFiles option, which would decipher the files as root, and change the ownership as part of the activation script of NixOS.

Thus, instead of forcing something special for secure files only, we would force something special for files in general, which would make it less special, and easier to extend in a modular fashion.

@nbp

nbp May 26, 2017

Member

I agree, and ideally this would need some way to get a path for each configuration file. For security reasons systemd services can switch users, and we would want these files to be read only by these users.

I would prefer to avoid FUSE as the first iteration, as this would imply trusting more code than necessary. (we are talking about the manipulation of secrets)

I think we could have something like:

{ security.encryptedFiles.file = {
    user = "foo"; /* uses users.users.<name>.uid or assert if missing */
    file = ...;
  };
  systemd.services.wpa_supplicant.serviceConfig.Exec =
     "wpa_supplicant -c ${security.encryptedFiles.file}";
}

The problem of this scheme is that the author of the module would have to think ahead that the file has to be secure.

For example, simple configuration can provide a default configuration based on pam, but leave complex option which could be provided via extraConfig to set password in the configuration files. In such case we might want as a user to retrofit security in an expression which did not expected it first.

To handle such cases, I think we should make a submodule which is global to NixOS to let people register secure files anywhere, by adding an option type which would be a submodule for declaring files. This submodule would then be in charge of finding the proper renaming of the files path, and these entries could be listed in the security.encryptedFiles option to let the activation script decipher these files as root, before changing the ownership. Thus a service would look like:

{ lib, config, ... }:
{ options.services.foo = {
    enable = ...;
    extraConfig = ...;
    configFile = lib.mkOption { type = lib.types.file; };
  };
  config = with config.services.foo; lib.mkIf enable {
    security.secureFiles = [ configFile ]; # Can be added by another module
    systemd.services.foo.serviceConfig.Exec =
       "command --conf ${configFile}";
    services.foo.configFile.content = extraConfig;
  };
}

Doing it that way, ensure that the security.maybeSecureFiles can be registered by anybody who needs to set the services.foo.configFile.secure = true;. Thus, the submodule lib.types.file will contain the logic to translate any file from a store path to a /var or /tmp path if the file has the secure flag. The decoding would be handled by the security.secureFiles option, which would decipher the files as root, and change the ownership as part of the activation script of NixOS.

Thus, instead of forcing something special for secure files only, we would force something special for files in general, which would make it less special, and easier to extend in a modular fashion.

```
security.encryptedFiles = [
{ file = httpdConf;
owner = "httpd";

This comment has been minimized.

@grahamc

grahamc May 7, 2017

Member

maybe group (string, default null) and groupReadable (boolean, default false) as options too.

@grahamc

grahamc May 7, 2017

Member

maybe group (string, default null) and groupReadable (boolean, default false) as options too.

@eternaleye

This comment has been minimized.

Show comment
Hide comment
@eternaleye

eternaleye May 7, 2017

@Ekleog

First things first, "implementing a variant of SIV" is a rather fraught exercise. Google has been attempting this, with AES-GCM-SIV, and the result actually has significantly weaker security bounds than the AES-SIV construction they have based it on.

Second, the assumption that you "need no authenticated cleartext data" may not be true. The section of the RFC talking about reproducible encryption raises the question of leaking information about rollbacks; this can be mitigated by including more information in the authenticated data - because if anything in the AD or plaintext changes, the no-leaks property kicks in.

Third, your framing of what is done by SIV is not entirely correct. It omits the dbl operation, which while simple, is cryptographically important. I suspect your proposed variant has cryptographic weaknesses as a result.

I'm curious as to what side-channel attacks you feel might become relevant, as any reasonable implementation of AES-SIV or HS1-SIV will be constant-time.

Fourth, HS1-SIV was dropped from CAESAR round 3, yes. Given that asking the author if he had been told why resulted in being informed that he had not been told of any actual issues with the cipher mode, but that it had rather not been innovative enough (and was instead too well-trodden ground), this has caused me to change my view of CAESAR (to an "exploratory" or "experimental" competition rather than "select a future standard") somewhat more than it has changed my view of HS1-SIV.

@paragonie-scott

Reproducible encryption is valuable because it fits into the broader reproducibility story of Nix. In addition, MRAE algorithms are the best-in-breed option security-wise, which provably satisfy the maximal security properties possible for deterministic encryption, given the security of the underlying primitives. As a result, using them is prudent anywhere they can be used (i.e., where streaming/online encryption/decryption are not necessary. For that, Rogaway's related paper provides optimal security.)

In addition, the cryptographic doom principle is a heuristic. Moreover, it is one that applies when stronger results are not available. The security reduction from AES-SIV to AES does not contain assumptions that are subject to being invalidated by the cryptographic doom principle.

Finally, the mitigation you propose is suboptimal - in particular, it still uses a cipher mode that does not meet the optimal security guarantees, and if you don't want deterministic encryption with an MRAE, you can simply... include random data, sourced however you want (such as from /dev/[u]random) as AD. Because of the optimality properties of MRAE, this will effectively randomize the entire ciphertext.

And no, your assertion about getting NMR from seeding with the hash even if the RNG is weak does not hold. There are many forms of weakness in RNGs, including some that converge to a small set of output values. The Debian weak-keys bug was a case of this. In such a case, the problem isn't with the seeding of the RNG, it's that it can never output the full range of values no matter what the seed,.

eternaleye commented May 7, 2017

@Ekleog

First things first, "implementing a variant of SIV" is a rather fraught exercise. Google has been attempting this, with AES-GCM-SIV, and the result actually has significantly weaker security bounds than the AES-SIV construction they have based it on.

Second, the assumption that you "need no authenticated cleartext data" may not be true. The section of the RFC talking about reproducible encryption raises the question of leaking information about rollbacks; this can be mitigated by including more information in the authenticated data - because if anything in the AD or plaintext changes, the no-leaks property kicks in.

Third, your framing of what is done by SIV is not entirely correct. It omits the dbl operation, which while simple, is cryptographically important. I suspect your proposed variant has cryptographic weaknesses as a result.

I'm curious as to what side-channel attacks you feel might become relevant, as any reasonable implementation of AES-SIV or HS1-SIV will be constant-time.

Fourth, HS1-SIV was dropped from CAESAR round 3, yes. Given that asking the author if he had been told why resulted in being informed that he had not been told of any actual issues with the cipher mode, but that it had rather not been innovative enough (and was instead too well-trodden ground), this has caused me to change my view of CAESAR (to an "exploratory" or "experimental" competition rather than "select a future standard") somewhat more than it has changed my view of HS1-SIV.

@paragonie-scott

Reproducible encryption is valuable because it fits into the broader reproducibility story of Nix. In addition, MRAE algorithms are the best-in-breed option security-wise, which provably satisfy the maximal security properties possible for deterministic encryption, given the security of the underlying primitives. As a result, using them is prudent anywhere they can be used (i.e., where streaming/online encryption/decryption are not necessary. For that, Rogaway's related paper provides optimal security.)

In addition, the cryptographic doom principle is a heuristic. Moreover, it is one that applies when stronger results are not available. The security reduction from AES-SIV to AES does not contain assumptions that are subject to being invalidated by the cryptographic doom principle.

Finally, the mitigation you propose is suboptimal - in particular, it still uses a cipher mode that does not meet the optimal security guarantees, and if you don't want deterministic encryption with an MRAE, you can simply... include random data, sourced however you want (such as from /dev/[u]random) as AD. Because of the optimality properties of MRAE, this will effectively randomize the entire ciphertext.

And no, your assertion about getting NMR from seeding with the hash even if the RNG is weak does not hold. There are many forms of weakness in RNGs, including some that converge to a small set of output values. The Debian weak-keys bug was a case of this. In such a case, the problem isn't with the seeding of the RNG, it's that it can never output the full range of values no matter what the seed,.

@Ekleog

This comment has been minimized.

Show comment
Hide comment
@Ekleog

Ekleog May 7, 2017

@eternaleye

About the dbl operation, I guess you are taking it from http://csrc.nist.gov/groups/ST/toolkit/BCM/documents/proposedmodes/siv/siv.pdf . This operation is included in the C_k I talked about (because t = 0 so m = 1), and is thus a no-op when there is no AD (xor-ing with a k-dependent constant should have no importance when the result immediately goes into a PRF). What's more, https://www.iacr.org/archive/eurocrypt2006/40040377/40040377.pdf (the actual SIV paper) makes mention of it only in section 4 "Enriching a PRF to take Vectors of Strings as Input" if I read correctly, which is not needed thanks to the absence of AD in the use nix would make of SIV.

As for your proposal of actually using AD for including random data, then it's just as good to use a non-MRAE non-deterministic scheme, as nix is not lacking any entropy.

In my opinion there are two possibilities.

  1. Either use deterministic encryption, which gives bitwise reproducibility but loses the property that identical cleartexts have identical ciphertexts (by design).

  2. Or do not, and forget about all that stuff of SIV, just use crypto_secretbox_easy.

The question of how to implement deterministic encryption is another issue, important only if choice 1 is picked, and is limited mostly by the primitives libsodium offers, if there is no will to use another library. At best a library implementing AES-SIV (or HS1-SIV or whatever) would be used, but even openssl doesn't seem to offer an implementation of it. So the solution I proposed is a stop-gap solution, that achieves far less than SIV (by not proposing associated data that could be used as a nonce, basically not proposing MRAE but only deterministic encryption), but should be far easier to implement. It can also avoid the "cryptographic doom principle" by simply using AE like XSalsa20-Poly1305 instead of just XSalsa20, by simply changing the encryption algorithm, and thus checking the MAC twice (because why not? performance is not really an issue here afaict). That said, I'm not sure it's OK, especially due to the fact XSalsa20 is not an IV-based encryption but a nonce-based encryption, which means all the requirements of https://www.iacr.org/archive/eurocrypt2006/40040377/40040377.pdf are not strictly-speaking met, but libsodium doesn't have an implementation of AES...).

Ekleog commented May 7, 2017

@eternaleye

About the dbl operation, I guess you are taking it from http://csrc.nist.gov/groups/ST/toolkit/BCM/documents/proposedmodes/siv/siv.pdf . This operation is included in the C_k I talked about (because t = 0 so m = 1), and is thus a no-op when there is no AD (xor-ing with a k-dependent constant should have no importance when the result immediately goes into a PRF). What's more, https://www.iacr.org/archive/eurocrypt2006/40040377/40040377.pdf (the actual SIV paper) makes mention of it only in section 4 "Enriching a PRF to take Vectors of Strings as Input" if I read correctly, which is not needed thanks to the absence of AD in the use nix would make of SIV.

As for your proposal of actually using AD for including random data, then it's just as good to use a non-MRAE non-deterministic scheme, as nix is not lacking any entropy.

In my opinion there are two possibilities.

  1. Either use deterministic encryption, which gives bitwise reproducibility but loses the property that identical cleartexts have identical ciphertexts (by design).

  2. Or do not, and forget about all that stuff of SIV, just use crypto_secretbox_easy.

The question of how to implement deterministic encryption is another issue, important only if choice 1 is picked, and is limited mostly by the primitives libsodium offers, if there is no will to use another library. At best a library implementing AES-SIV (or HS1-SIV or whatever) would be used, but even openssl doesn't seem to offer an implementation of it. So the solution I proposed is a stop-gap solution, that achieves far less than SIV (by not proposing associated data that could be used as a nonce, basically not proposing MRAE but only deterministic encryption), but should be far easier to implement. It can also avoid the "cryptographic doom principle" by simply using AE like XSalsa20-Poly1305 instead of just XSalsa20, by simply changing the encryption algorithm, and thus checking the MAC twice (because why not? performance is not really an issue here afaict). That said, I'm not sure it's OK, especially due to the fact XSalsa20 is not an IV-based encryption but a nonce-based encryption, which means all the requirements of https://www.iacr.org/archive/eurocrypt2006/40040377/40040377.pdf are not strictly-speaking met, but libsodium doesn't have an implementation of AES...).

@eternaleye

This comment has been minimized.

Show comment
Hide comment
@eternaleye

eternaleye May 7, 2017

@Ekleog

I'm not necessarily proposing that AD be used to include random data. Rather, by including more data besides just the secret, it acts as a confounder against rewind leakage. If any of the other data has changed in the meantime, the AD would change, changing the ciphertext - preventing the rewind from being visible. This preserves determinism (it just increases how many inputs the ciphertext depends on), but reduces the risk.

eternaleye commented May 7, 2017

@Ekleog

I'm not necessarily proposing that AD be used to include random data. Rather, by including more data besides just the secret, it acts as a confounder against rewind leakage. If any of the other data has changed in the meantime, the AD would change, changing the ciphertext - preventing the rewind from being visible. This preserves determinism (it just increases how many inputs the ciphertext depends on), but reduces the risk.

@Ekleog

This comment has been minimized.

Show comment
Hide comment
@Ekleog

Ekleog May 7, 2017

@eternaleye Point made, indeed! I was thinking of encrypting entire files at once to mitigate that threat by just including all relevant data in the encrypted data. Using AD has the advantage of spending less time encrypting stuff, at the drawback of requiring a significantly more complex implementation... if it's possible to find an enterprise-grade implementation of SIV that's clearly better :)

Ekleog commented May 7, 2017

@eternaleye Point made, indeed! I was thinking of encrypting entire files at once to mitigate that threat by just including all relevant data in the encrypted data. Using AD has the advantage of spending less time encrypting stuff, at the drawback of requiring a significantly more complex implementation... if it's possible to find an enterprise-grade implementation of SIV that's clearly better :)

@taktoa

This comment has been minimized.

Show comment
Hide comment
@taktoa

taktoa May 9, 2017

@Ekleog I'm not a cryptographer, but I did see this video a little while back that has led me to believe that you generally want /dev/urandom unless your application can be affected by lack of entropy in early boot. Is this wrong, or are you just addressing aspects of the POSIX specification that aren't actually reflected in Linux?

taktoa commented May 9, 2017

@Ekleog I'm not a cryptographer, but I did see this video a little while back that has led me to believe that you generally want /dev/urandom unless your application can be affected by lack of entropy in early boot. Is this wrong, or are you just addressing aspects of the POSIX specification that aren't actually reflected in Linux?

@paragonie-scott

This comment has been minimized.

Show comment
Hide comment
@taktoa

This comment has been minimized.

Show comment
Hide comment
@taktoa

taktoa May 9, 2017

According to that article, it seems like getrandom(2) might be the best option on Linux 3.17 and up, as it avoids the issues that /dev/urandom has during early boot.

taktoa commented May 9, 2017

According to that article, it seems like getrandom(2) might be the best option on Linux 3.17 and up, as it avoids the issues that /dev/urandom has during early boot.

@Ekleog

This comment has been minimized.

Show comment
Hide comment
@Ekleog

Ekleog May 9, 2017

OK, so I fear we're moving to an unrelated debate (I'd be saying just use whatever primitives libsodium gives), just a one remark (I just erased a block of text that would be really clearly off-topic, just can't leave my last remark unexplained).

@taktoa, in the video you linked the speaker states https://blog.cr.yp.to/20140205-entropy.html agrees with him. Seeing DJB's blog, I wanted to check out what point he exactly made. It turns out he's not at all advocating for /dev/urandom, quite the contrary: he states adding entropy in the mix is harmful. Which is exactly what both /dev/random and /dev/urandom do. A malicious device could just as well compromise /dev/urandom as /dev/random. Actually, what he advocates seems to be first seeding the RNG with entropy, then acting deterministically... which is exactly what libgcrypt (gnupg's crypto library, the only one I'm at least a bit familiar with the internals) does. With /dev/random as a seed for strong randomness, and /dev/urandom as a seed for weaker randomness.

Please note I'm not advocating the use of /dev/random for everything or whatever, just saying there are still valid use cases for it, and this specific one (that is, creating once a... year? a long-term key) appears to be exactly one for it. Preferably with a POW and a deterministic PRNG behind, as DJB calls for.

Anyway, it's not like any of this matters, the best way to go is clearly to use libsodium's implementation and let them struggle with keeping everything secure. (it turns out they use getrandom internally, I didn't check whether with GRND_RANDOM or not)

Ekleog commented May 9, 2017

OK, so I fear we're moving to an unrelated debate (I'd be saying just use whatever primitives libsodium gives), just a one remark (I just erased a block of text that would be really clearly off-topic, just can't leave my last remark unexplained).

@taktoa, in the video you linked the speaker states https://blog.cr.yp.to/20140205-entropy.html agrees with him. Seeing DJB's blog, I wanted to check out what point he exactly made. It turns out he's not at all advocating for /dev/urandom, quite the contrary: he states adding entropy in the mix is harmful. Which is exactly what both /dev/random and /dev/urandom do. A malicious device could just as well compromise /dev/urandom as /dev/random. Actually, what he advocates seems to be first seeding the RNG with entropy, then acting deterministically... which is exactly what libgcrypt (gnupg's crypto library, the only one I'm at least a bit familiar with the internals) does. With /dev/random as a seed for strong randomness, and /dev/urandom as a seed for weaker randomness.

Please note I'm not advocating the use of /dev/random for everything or whatever, just saying there are still valid use cases for it, and this specific one (that is, creating once a... year? a long-term key) appears to be exactly one for it. Preferably with a POW and a deterministic PRNG behind, as DJB calls for.

Anyway, it's not like any of this matters, the best way to go is clearly to use libsodium's implementation and let them struggle with keeping everything secure. (it turns out they use getrandom internally, I didn't check whether with GRND_RANDOM or not)

@tarcieri

This comment has been minimized.

Show comment
Hide comment
@tarcieri

tarcieri May 9, 2017

I'll just leave this here: https://sockpuppet.org/blog/2014/02/25/safely-generate-random-numbers/

In order of preference: getrandom(2), /dev/urandom, and I'll just end there. Telling someone "use /dev/random instead of /dev/urandom because it makes better random numbers" is outright incorrect.

The source of confusion regarding a preference for /dev/random over /dev/urandom seems to be a very out-of-date manpage which was recently updated:

https://bugzilla.kernel.org/show_bug.cgi?id=71211

See:

http://man7.org/linux/man-pages/man4/random.4.html
http://man7.org/linux/man-pages/man7/random.7.html

tarcieri commented May 9, 2017

I'll just leave this here: https://sockpuppet.org/blog/2014/02/25/safely-generate-random-numbers/

In order of preference: getrandom(2), /dev/urandom, and I'll just end there. Telling someone "use /dev/random instead of /dev/urandom because it makes better random numbers" is outright incorrect.

The source of confusion regarding a preference for /dev/random over /dev/urandom seems to be a very out-of-date manpage which was recently updated:

https://bugzilla.kernel.org/show_bug.cgi?id=71211

See:

http://man7.org/linux/man-pages/man4/random.4.html
http://man7.org/linux/man-pages/man7/random.7.html

@zx2c4

This comment has been minimized.

Show comment
Hide comment
@zx2c4

zx2c4 May 9, 2017

Seeing as this thread has descended into /dev/random madness -- a common black hole in message threads -- I'll duck out. But my advice remains the same: decide your actual requirements for this system, and then crypto people can help you. At the moment, the requirements aren't actually clearly determined, and so in the midst of confusion about this comes even more confusion about the crypto. When you've got your ideas together, write a clear document, and then we can get rolling again. @grahamc, who asked me to take a look at this thread in the first place, knows where to find me.

zx2c4 commented May 9, 2017

Seeing as this thread has descended into /dev/random madness -- a common black hole in message threads -- I'll duck out. But my advice remains the same: decide your actual requirements for this system, and then crypto people can help you. At the moment, the requirements aren't actually clearly determined, and so in the midst of confusion about this comes even more confusion about the crypto. When you've got your ideas together, write a clear document, and then we can get rolling again. @grahamc, who asked me to take a look at this thread in the first place, knows where to find me.

## Key generation
Currently, we assume that `<nixos-store-key>` exists beforehand. It

This comment has been minimized.

@joepie91

joepie91 May 10, 2017

How would this work in a multi-user environment? Could each user feasibly have their own key? And how would this integrate with something like NixOps, where one might want to have a separate key for NixOps deployments, and possibly share that key with others managing the deployment, without exposing their local key?

@joepie91

joepie91 May 10, 2017

How would this work in a multi-user environment? Could each user feasibly have their own key? And how would this integrate with something like NixOps, where one might want to have a separate key for NixOps deployments, and possibly share that key with others managing the deployment, without exposing their local key?

This comment has been minimized.

@Ekleog

Ekleog Jun 13, 2017

This RFC only adds builtins.encryptString and a nix-store --decrypt helper, so as far as I understand this issue is left to future work.

I believe it's best to first get this in as soon as we can all agree on what's wanted, and then try thinking more about how to generate keys and how to use them, once the primitives will be in place (and when we will have moved a bit further in the process of getting nix encryption in production, making steps one by one so we don't just stumble on agreeing on a RFC for three months like what is currently happening). Don't you think? :)

@Ekleog

Ekleog Jun 13, 2017

This RFC only adds builtins.encryptString and a nix-store --decrypt helper, so as far as I understand this issue is left to future work.

I believe it's best to first get this in as soon as we can all agree on what's wanted, and then try thinking more about how to generate keys and how to use them, once the primitives will be in place (and when we will have moved a bit further in the process of getting nix encryption in production, making steps one by one so we don't just stumble on agreeing on a RFC for three months like what is currently happening). Don't you think? :)

@Ekleog

This comment has been minimized.

Show comment
Hide comment
@Ekleog

Ekleog Jun 13, 2017

Not having seen a single comment arying to summarize what is still to do in order to make things move forward:

  1. Decide whether we want deterministic encryption (I'd vote for it, in order to not re-generate all the config files that include encrypted bits at every single system rebuild, but it leaks the rebuilds when encrypted bits change)

  2. In parallel, think over @zimbatm's comment here. I think an encrypted attribute would indeed be better, but not required to get things to a useful first step: it just removes a help the module developer could have to not generate broken configuration files. This sure is useful, but not needed in a first step.
    Also, I'm wondering whether it wouldn't be possible to bind-mount the unencrypted file with the right permissions over the normal file in the store, making this attribute unnecessary, as well as the need to fix all dependent paths. This is a completely separate solution, and doesn't require special support from the RFC, building only using the tools it offers.

  3. Once the choice in 1 is made, if we want deterministic encryption, write it in the RFC and leave the issue of how to implement it to a future PR on the nix repo -- it's unrelated to the content of the RFC, which should just notify what is to be done.

  4. Accept this RFC, unless someone has something more to say.

Hoping to move this forward, what is your opinion on this matter?

Ekleog commented Jun 13, 2017

Not having seen a single comment arying to summarize what is still to do in order to make things move forward:

  1. Decide whether we want deterministic encryption (I'd vote for it, in order to not re-generate all the config files that include encrypted bits at every single system rebuild, but it leaks the rebuilds when encrypted bits change)

  2. In parallel, think over @zimbatm's comment here. I think an encrypted attribute would indeed be better, but not required to get things to a useful first step: it just removes a help the module developer could have to not generate broken configuration files. This sure is useful, but not needed in a first step.
    Also, I'm wondering whether it wouldn't be possible to bind-mount the unencrypted file with the right permissions over the normal file in the store, making this attribute unnecessary, as well as the need to fix all dependent paths. This is a completely separate solution, and doesn't require special support from the RFC, building only using the tools it offers.

  3. Once the choice in 1 is made, if we want deterministic encryption, write it in the RFC and leave the issue of how to implement it to a future PR on the nix repo -- it's unrelated to the content of the RFC, which should just notify what is to be done.

  4. Accept this RFC, unless someone has something more to say.

Hoping to move this forward, what is your opinion on this matter?

@0xABAB

This comment has been minimized.

Show comment
Hide comment
@0xABAB

0xABAB Jun 30, 2017

I think the RFC is ill conceived, but there is a real need for this feature.

I am not seeing any extension mechanisms for the encryption system to be used, which already makes this unsuitable as an enterprise solution. On top of that there are some of the concerns already raised by the author of the RFC.

The core feature which needs to be designed is a primitive where a reference can be injected into any kind of file which can be replaced at run-time when some data is provided to it via a variety of mechanisms. This would still be a system where the builds are reproducible, because the mechanism is a constant. No keys are ever part of the "build".

0xABAB commented Jun 30, 2017

I think the RFC is ill conceived, but there is a real need for this feature.

I am not seeing any extension mechanisms for the encryption system to be used, which already makes this unsuitable as an enterprise solution. On top of that there are some of the concerns already raised by the author of the RFC.

The core feature which needs to be designed is a primitive where a reference can be injected into any kind of file which can be replaced at run-time when some data is provided to it via a variety of mechanisms. This would still be a system where the builds are reproducible, because the mechanism is a constant. No keys are ever part of the "build".

@Ekleog Ekleog referenced this pull request Jun 30, 2017

Closed

[RFC 0009] Nix rapid release #9

@Nadrieril

This comment has been minimized.

Show comment
Hide comment
@Nadrieril

Nadrieril Aug 30, 2017

The discussion about deterministic encryption is stalling this RFC, but think it shouldn't.
It seems to me that we usually want encryption for configuration and deterministic builds for packages. In that sense, having non-deterministic crypto shouldn't break package reproducibility much since packages don't need encrypted secrets (AFAIU). Isn't the whole point of deterministic builds being able to check that a given (publically-available) build output matches a given (publically-available) build input ? Besides, the non-determinism is very localised since there is only a specific primitive that can introduce it.
If there really are things we want to be both deterministic and storing secrets, why not have an extensible encryption primitive, as @0xABAB suggests ? The first implementation of this RFC could implement only standard non-deterministic crypto, and we could make another RFC to add deterministic crypto, that could be enabled on a case-by-case basis so as not to weaken the whole system's security. Also, replaceable crypto is good engineering regardless.
I however disagree with @0xABAB on the fact that we need a primitive to have run-time injection of data. I would very much like to have such primitive, but I believe the core of this RFC is the encryption primitive, which by itself would be super cool, and which I would love to have as soon as we can.

Nadrieril commented Aug 30, 2017

The discussion about deterministic encryption is stalling this RFC, but think it shouldn't.
It seems to me that we usually want encryption for configuration and deterministic builds for packages. In that sense, having non-deterministic crypto shouldn't break package reproducibility much since packages don't need encrypted secrets (AFAIU). Isn't the whole point of deterministic builds being able to check that a given (publically-available) build output matches a given (publically-available) build input ? Besides, the non-determinism is very localised since there is only a specific primitive that can introduce it.
If there really are things we want to be both deterministic and storing secrets, why not have an extensible encryption primitive, as @0xABAB suggests ? The first implementation of this RFC could implement only standard non-deterministic crypto, and we could make another RFC to add deterministic crypto, that could be enabled on a case-by-case basis so as not to weaken the whole system's security. Also, replaceable crypto is good engineering regardless.
I however disagree with @0xABAB on the fact that we need a primitive to have run-time injection of data. I would very much like to have such primitive, but I believe the core of this RFC is the encryption primitive, which by itself would be super cool, and which I would love to have as soon as we can.

@Nadrieril

This comment has been minimized.

Show comment
Hide comment
@Nadrieril

Nadrieril Sep 17, 2017

So, what should happen now ? Has any kind of consensus been reached ? Maybe the authors of the RFC want to amend the RFC to reflect the discussion and then we can vote on something ?

Nadrieril commented Sep 17, 2017

So, what should happen now ? Has any kind of consensus been reached ? Maybe the authors of the RFC want to amend the RFC to reflect the discussion and then we can vote on something ?

@zimbatm

This comment has been minimized.

Show comment
Hide comment
@zimbatm

zimbatm Sep 18, 2017

Member

sounds good. Making both the crypto deterministic and secure seems to be at antithesis.

Member

zimbatm commented Sep 18, 2017

sounds good. Making both the crypto deterministic and secure seems to be at antithesis.

@Mic92 Mic92 referenced this pull request Jan 16, 2018

Merged

openvpn: add option to store credentials #33890

2 of 8 tasks complete
@dasJ

This comment has been minimized.

Show comment
Hide comment
@dasJ

dasJ Apr 23, 2018

Bumping this RFC up again because I also really need it. Why is this PR stalled? What is the process to continue? It's still tagged as "status: draft" so what are the changes @edolstra still needs to make? Or is there already a mailing-list discussion that's not linked here? Is there any place more active where people actually talk about this PR?

dasJ commented Apr 23, 2018

Bumping this RFC up again because I also really need it. Why is this PR stalled? What is the process to continue? It's still tagged as "status: draft" so what are the changes @edolstra still needs to make? Or is there already a mailing-list discussion that's not linked here? Is there any place more active where people actually talk about this PR?

@Scriptkiddi

This comment has been minimized.

Show comment
Hide comment
@Scriptkiddi

Scriptkiddi Apr 27, 2018

I was also wondering what needs to be done here? Can I do anything to help?

Scriptkiddi commented Apr 27, 2018

I was also wondering what needs to be done here? Can I do anything to help?

@Ekleog

This comment has been minimized.

Show comment
Hide comment
@Ekleog

Ekleog Apr 27, 2018

So. I've just re-read through the whole discussion, and here are, I think, all the points that have been raised and not answered yet (and that are relevant to the RFC in itself, as discussion has strayed a bit at times). Sorry if I missed something, please up it.

Summary

  1. @zimbatm (here) proposed that encrypted files are tagged so they cannot be used except for being decrypted, so that an encrypted file could not be referenced in an unencrypted file, in order to reduce the likelihood of misconfiguration.
  2. A point I just noticed: if <nixos-store-key> is a path, it means "${<nixos-store-key>}" would put the key into the store. Would it be possible to make it a string, in order to reduce the likelihood of this happening?
  3. @domenkozar (here) proposed that keys are referred to by name -- which would also solve the issue just above.
  4. @taktoa (here) noticed a typo: s/obverse/observe/ line 245
  5. The question of reproducible encryption ended up with the answers that there are cryptographic solutions to do it, but they are not easy-to-use in libsodium. The question of whether we would want it or not anyway stays open.
  6. @0xABAB and @Nadrieril (here) raised the questions of the possibility to extend the cryptographic primitive, as primitives come and go, and having an easy way to replace them is often good engineering.

Additionally, @nbp (here) started thinking about how to integrate this into nixpkgs. I don't think it's necessarily an important point to debate right now, but it will sure become after this RFC has been completed, so putting it here so as not to forget it.

So could we just try to get to an agreement on these 6 points, merge the RFC, maybe adapt a bit the current implementation, and finally have a way to remove some passwords for the nix-store? :)

Ekleog commented Apr 27, 2018

So. I've just re-read through the whole discussion, and here are, I think, all the points that have been raised and not answered yet (and that are relevant to the RFC in itself, as discussion has strayed a bit at times). Sorry if I missed something, please up it.

Summary

  1. @zimbatm (here) proposed that encrypted files are tagged so they cannot be used except for being decrypted, so that an encrypted file could not be referenced in an unencrypted file, in order to reduce the likelihood of misconfiguration.
  2. A point I just noticed: if <nixos-store-key> is a path, it means "${<nixos-store-key>}" would put the key into the store. Would it be possible to make it a string, in order to reduce the likelihood of this happening?
  3. @domenkozar (here) proposed that keys are referred to by name -- which would also solve the issue just above.
  4. @taktoa (here) noticed a typo: s/obverse/observe/ line 245
  5. The question of reproducible encryption ended up with the answers that there are cryptographic solutions to do it, but they are not easy-to-use in libsodium. The question of whether we would want it or not anyway stays open.
  6. @0xABAB and @Nadrieril (here) raised the questions of the possibility to extend the cryptographic primitive, as primitives come and go, and having an easy way to replace them is often good engineering.

Additionally, @nbp (here) started thinking about how to integrate this into nixpkgs. I don't think it's necessarily an important point to debate right now, but it will sure become after this RFC has been completed, so putting it here so as not to forget it.

So could we just try to get to an agreement on these 6 points, merge the RFC, maybe adapt a bit the current implementation, and finally have a way to remove some passwords for the nix-store? :)

@Ekleog

This comment has been minimized.

Show comment
Hide comment
@Ekleog

Ekleog Apr 27, 2018

So now my opinions about these 6 points:

  1. I think this is not required for a first iteration. At worst, what will happen is that modules badly written will try to start with encrypted data, and will fail. That's not a security issue, and won't affect modules that were working before. So I don't think that's an urgent issue, contrarily to finally getting an encryption builtin.
  2. I think the idea of referring to keys by name would solve this issue.
  3. Well, same as 2.
  4. Simple typo, I don't think there's much space left for debate.
  5. I don't think we need deterministic encryption for the first iteration. It'd sure be a nice-to-have, but there is no need like there is a need for at least encryption. And with extensible cryptographic primitives, it's easy to add it afterwards, so I'd say let's postpone this debate for later.
  6. For extensibility of the cryptographic primitive, I think this RFC as defined implicitly caters for it. Indeed, changing the cryptographic primitive will require changing the key. So the choice of cryptographic primitive could just be embedded into the key file, in a format maybe like (for the first libsodium-using iteration): chacha20-poly1305:[base64-encoded key] (note that this is only for illustration purposes, as I don't think the RFC needs to describe this format). This way the implementation would be easily extensible, with exactly the same syntax for the user.

So, to me, the required changes in the current RFC are:

  • s/obverse/observe/ line 245
  • Change the encryption primitive to take, instead of the path, a name (that will be looked for under $NIX_STATE_DIR/keys/[key_name])
  • Write anywhere that extensibility of the cryptographic primitive will be handled by changing the format of the key.

Ekleog commented Apr 27, 2018

So now my opinions about these 6 points:

  1. I think this is not required for a first iteration. At worst, what will happen is that modules badly written will try to start with encrypted data, and will fail. That's not a security issue, and won't affect modules that were working before. So I don't think that's an urgent issue, contrarily to finally getting an encryption builtin.
  2. I think the idea of referring to keys by name would solve this issue.
  3. Well, same as 2.
  4. Simple typo, I don't think there's much space left for debate.
  5. I don't think we need deterministic encryption for the first iteration. It'd sure be a nice-to-have, but there is no need like there is a need for at least encryption. And with extensible cryptographic primitives, it's easy to add it afterwards, so I'd say let's postpone this debate for later.
  6. For extensibility of the cryptographic primitive, I think this RFC as defined implicitly caters for it. Indeed, changing the cryptographic primitive will require changing the key. So the choice of cryptographic primitive could just be embedded into the key file, in a format maybe like (for the first libsodium-using iteration): chacha20-poly1305:[base64-encoded key] (note that this is only for illustration purposes, as I don't think the RFC needs to describe this format). This way the implementation would be easily extensible, with exactly the same syntax for the user.

So, to me, the required changes in the current RFC are:

  • s/obverse/observe/ line 245
  • Change the encryption primitive to take, instead of the path, a name (that will be looked for under $NIX_STATE_DIR/keys/[key_name])
  • Write anywhere that extensibility of the cryptographic primitive will be handled by changing the format of the key.
@lschuermann

This comment has been minimized.

Show comment
Hide comment
@lschuermann

lschuermann Sep 30, 2018

I'm not quite sure whether this PR already implies this:

I think it's quite important to not only encrypt secrets using builtins.encryptString, but to also provide a utility for encrypting before evaluating / building the config. This could then allow using the ciphertext in the configuration itself. In addition to having them encrypted in the nix-store, this would allow me to push my config to a public version control, without exposing all of my secrets. (As far as I know a pretty common issue)

This could then go along with the ability to copy a key to another machine, and being able to use multiple keys (addressable by name, etc.). For example, multiple machines could share the WiFi configuration + WiFi secrets encryption key.

I'm already doing something similar with openssl (still world-readable in the nix store), but it would be much better if this were somehow integrated.

lschuermann commented Sep 30, 2018

I'm not quite sure whether this PR already implies this:

I think it's quite important to not only encrypt secrets using builtins.encryptString, but to also provide a utility for encrypting before evaluating / building the config. This could then allow using the ciphertext in the configuration itself. In addition to having them encrypted in the nix-store, this would allow me to push my config to a public version control, without exposing all of my secrets. (As far as I know a pretty common issue)

This could then go along with the ability to copy a key to another machine, and being able to use multiple keys (addressable by name, etc.). For example, multiple machines could share the WiFi configuration + WiFi secrets encryption key.

I'm already doing something similar with openssl (still world-readable in the nix store), but it would be much better if this were somehow integrated.

@Ekleog

This comment has been minimized.

Show comment
Hide comment
@Ekleog

Ekleog Oct 1, 2018

@lschuermann This would be solved by doing nix repl then builtins.encryptString <nixos-key> "mysecret", and using the output of this command instead of the value.

However, your message is important in that it reminds us that in nixpkgs, we must not only accept eg. a password argument that will be transparently encrypted, but we must also support a encryptedPassword that'd assume encryption has already been performed.

tl;dr: I don't think there's anything to change in the RFC for this, but it'll be good to keep in mind when adapting modules to the new system :)

Ekleog commented Oct 1, 2018

@lschuermann This would be solved by doing nix repl then builtins.encryptString <nixos-key> "mysecret", and using the output of this command instead of the value.

However, your message is important in that it reminds us that in nixpkgs, we must not only accept eg. a password argument that will be transparently encrypted, but we must also support a encryptedPassword that'd assume encryption has already been performed.

tl;dr: I don't think there's anything to change in the RFC for this, but it'll be good to keep in mind when adapting modules to the new system :)

@Ekleog

This comment has been minimized.

Show comment
Hide comment
@Ekleog

Ekleog Oct 5, 2018

@edolstra Could you handle the 3 concerns pointed out in #5 (comment) and merge this RFC? I think no disagreement and a few +1's after ~5/6 months is enough to say that everyone agree. Or do you want me to do a PR to your RFC branch with the changes?

Ekleog commented Oct 5, 2018

@edolstra Could you handle the 3 concerns pointed out in #5 (comment) and merge this RFC? I think no disagreement and a few +1's after ~5/6 months is enough to say that everyone agree. Or do you want me to do a PR to your RFC branch with the changes?

@Ekleog

This comment has been minimized.

Show comment
Hide comment
@Ekleog

Ekleog Oct 5, 2018

(also, I can co-author if there is a need for it)

Ekleog commented Oct 5, 2018

(also, I can co-author if there is a need for it)

@edolstra

This comment has been minimized.

Show comment
Hide comment
@edolstra

edolstra Oct 7, 2018

Member

@Ekleog The main issue with this RFC is the lack of deterministic encryption, which is incompatible with the goal of binary reproducibility. I'll need to think a bit more about what to do about that.

Member

edolstra commented Oct 7, 2018

@Ekleog The main issue with this RFC is the lack of deterministic encryption, which is incompatible with the goal of binary reproducibility. I'll need to think a bit more about what to do about that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment