Improve qvm-backup key derivation/management #971

Open
andrewdavidwong opened this Issue Apr 26, 2015 · 139 comments

Projects

None yet
@marmarek marmarek added this to the Release 3.0 milestone May 3, 2015
@marmarek marmarek modified the milestone: Release 3.1, Release 3.0 May 13, 2015
@marmarek
Member

We need someone who know cryptography to check discussed key derivation scheme. Especially answer for questions in this message: https://groups.google.com/d/msg/qubes-devel/CZ7WRwLXcnk/1N0sYf6lVvUJ

@andrewdavidwong
Member

We may never get the assistance of a cryptographer on this. For now, we should at least pass the -md option to openssl and specify sha256 (to match the key length of aes256). In other words:

openssl enc -md sha256 -e -aes-256 [...]

This would be better than nothing. Currently, md5 is used by default, which is actually weakening security for users who select passphrases larger than 128 bits (probably almost all Qubes users).

@marmarek
Member

On Thu, Oct 22, 2015 at 02:54:10AM -0700, Axon wrote:

We may never get the assistance of a cryptographer on this. For now, we should at least pass the -md option to openssl and specify sha256 (to match the key length of aes256). In other words:

openssl enc -md sha256 -e -aes-256 [...]

This would be better than nothing. Currently, md5 is used by default, which is actually weakening security for users who select passphrases larger than 128 bits (probably almost all Qubes users).

As the user can (theoretically) freely choose encryption algorithm, just
sha256 may not be an universal option. Do you know any way to get key
length from openssl cipher name? Or just search for some numbers in
it?...

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

@andrewdavidwong
Member

As the user can (theoretically) freely choose encryption algorithm, just sha256 may not be an universal option.

Two ideas:

  1. Set the default to sha256, but give the user the option to change it. If the user wants to choose a different encryption algorithm, then it's up to her to also choose an appropriate message digest algorithm to accompany it.
  2. Hard code sha512 for everything, then let each encryption algorithm take as many truncated bits as it can (assuming that would work).

Do you know any way to get key length from openssl cipher name? Or just search for some numbers in it?...

I'm not aware of any. That would be my guess too...

@andrewdavidwong
Member

Based on this and after further research, I think it would be best to go with GPG for encryption (but not for integrity-verification).

Ideally, we would read from the user-defined preferences in ~/.gnupg/gpg.conf in dom0. This would allow users to set the following options (among others):

--s2k-cipher-algo name
      Use name as the cipher algorithm used to protect secret keys.  The default cipher
      is CAST5. This cipher is also used for  conventional  encryption  if  --personal-
      cipher-preferences and --cipher-algo is not given.

--s2k-digest-algo name
      Use  name  as  the  digest algorithm used to mangle the passphrases.  The default
      algorithm is SHA-1.

--s2k-mode n
      Selects how passphrases are mangled. If n is 0 a plain passphrase (which  is  not
      recommended)  will  be  used,  a  1  adds  a  salt to the passphrase and a 3 (the
      default) iterates the whole process a number of times (see --s2k-count).   Unless
      --rfc1991 is used, this mode is also used for conventional encryption.

--s2k-count n
      Specify how many times the passphrase mangling is repeated.  This value may range
      between 1024 and 65011712 inclusive.  The default  is  inquired  from  gpg-agent.
      Note  that  not all values in the 1024-65011712 range are legal and if an illegal
      value is selected, GnuPG will round up to the nearest legal value.   This  option
      is only meaningful if --s2k-mode is 3.

Recommended defaults:

gpg --encrypt --symmetric --s2k-cipher-algo AES256 --s2k-digest-algo SHA512 --s2k-mode 3 --s2k-count 65011712 --compress-algo bzip2
@marmarek
Member

This seems to be the best option.
Some minor questions/improvements/thoughts:

  1. Should be no --encrypt as it is for asymmetric encryption
  2. We currently have compression handled independently of encryption, I think we should keep that and use --compress-algo none. Our current approach allows to specify any "filter" program to do the actual compression. Personally I like pigz (parallel gzip) for much better performance on 8-core system :)
  3. Generally I'm fine with --s2k-count 65011712; on my system it needs 1.00s to encrypt just "test" string (so mostly key derivation benchmark), but it may be somehow longer on older systems; do we care? On Thinkpad T61 (Core 2 Duo T7300) - 1.42s. Ok, not bad at all.
  4. What about integrity protection (openssl)? is it a problem that the passphrase is used directly here (at least I think it is)? If yes and if we want to change that, it would require some more work to make it compatible with older formats, because integrity protection of backup header (only after its verification we can get some properties like backup format version, encryption algorithm etc).
  5. Backup data will be integrity-protected twice - once by openssl, and additionally by gpg. I don't think it is a problem, but I'm not a cryptographer.
  6. I don't think we should parse ~/.gnupg/gpg.conf. Either let use the defaults (possibly changed by user in that file), or override some option.
@adrelanos
Member

As for most secure gpg settings, check this out.

Quote https://github.com/Whonix/anon-gpg-tweaks

"Security and privacy enhancements for gnupg's config file for user
"user" in /home/user/.gnupg/gpg.conf." See also:
https://raw.github.com/ioerror/torbirdy/master/gpg.conf and
ioerror/torbirdy#11.

  1. Backup data will be integrity-protected twice - once by openssl, and additionally by gpg. I don't think it is a problem, but I'm not a cryptographer.

Not a bad thought. In past there were "known plaintext attacks", where
it was recommended against to encrypt known plaintext as this would
weaken the ciphertext. Any serious algorithm nowadays must pass this.

Normally it should not. If it did, it would be a critical cipher weakness.

I am just worried of the extra complexity of double encryption wrt to
manual backup recovery. But if you think it's fine, go for it.

  1. I don't think we should parse ~/.gnupg/gpg.conf. Either let use the defaults (possibly changed by user in that file), or override some option.

Yes, programs should not rely on user settings in ~/.gnupg/gpg.conf
indeed. Configurable options is a pony.

gpg supports --no-options (skip ~/.gnupg/gpg.conf) and --homedir.

By the way, in Qubes Q3 (and earlier) you cannot continue to backup if
you uncheck the encryption box. [I personally use full disk encryption
for all my disks, so the encryption of QVMM is nice, but I prefer to
disable it to ease manual recovery [in case the backup is ever damaged,
version conflict or w/e).]

@marmarek
Member
  1. Backup data will be integrity-protected twice - once by openssl, and additionally by gpg. I don't think it is a problem, but I'm not a cryptographer.

Not a bad thought. In past there were "known plaintext attacks", where
it was recommended against to encrypt known plaintext as this would
weaken the ciphertext. Any serious algorithm nowadays must pass this.

Normally it should not. If it did, it would be a critical cipher weakness.

I am just worried of the extra complexity of double encryption wrt to
manual backup recovery. But if you think it's fine, go for it.

Not encryption, just integrity protection - (.hmac files).

By the way, in Qubes Q3 (and earlier) you cannot continue to backup if
you uncheck the encryption box. [I personally use full disk encryption
for all my disks, so the encryption of QVMM is nice, but I prefer to
disable it to ease manual recovery [in case the backup is ever damaged,
version conflict or w/e).]

Can you create separate ticket for it? I was not aware of this
problem...

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

@andrewdavidwong
Member

@marmarek:

  1. Should be no --encrypt as it is for asymmetric encryption

Oops. Right.

  1. We currently have compression handled independently of encryption, I think we should keep that and use --compress-algo none. Our current approach allows to specify any "filter" program to do the actual compression. Personally I like pigz (parallel gzip) for much better performance on 8-core system :)

Sounds good to me. :)

  1. Generally I'm fine with --s2k-count 65011712; on my system it needs 1.00s to encrypt just "test" string (so mostly key derivation benchmark), but it may be somehow longer on older systems; do we care? On Thinkpad T61 (Core 2 Duo T7300) - 1.42s. Ok, not bad at all.

I suggested 65011712 as the default because it's the maximum. I think we should assume Qubes users want the "most secure" options by default. If they value backward compatibility more, they can override the defaults. (The security gain of the "most secure defaults" strategy is in some cases debatable, but the choice of defaults acts as an important reputational signal for a security-oriented OS.)

  1. What about integrity protection (openssl)? is it a problem that the passphrase is used directly here (at least I think it is)?

Yes, this is the other part of the problem.

If yes and if we want to change that, it would require some more work to make it compatible with older formats, because integrity protection of backup header (only after its verification we can get some properties like backup format version, encryption algorithm etc).

True...

  1. Backup data will be integrity-protected twice - once by openssl, and additionally by gpg. I don't think it is a problem, but I'm not a cryptographer.

IANAC either. I take it integrity-protection from gpg can't be turned off?

  1. I don't think we should parse ~/.gnupg/gpg.conf.

The only reason I suggested parsing ~/.gnupg/gpg.conf is because that seemed like the most logical place for the user to put their custom settings, but certainly if it's dangerous to parse it (or undesirable for another reason), then let's not do it. I really only meant to suggest that users should be able to specify their own options for s2k-cipher-algo, s2k-digest-algo, s2k-mode, and s2k-count (and any others that should be available).

Either let use the defaults

Sorry, I don't understand. Can you rephrase?

(possibly changed by user in that file),

Which file?

or override some option.

Would it be something like this?

qvm-backup --encrypt --s2k-count 1024 [...]
@andrewdavidwong
Member

@adrelanos:

Yes, programs should not rely on user settings in ~/.gnupg/gpg.conf indeed.

I'm glad you and @marmarek pointed this out to me. As mentioned in my previous message, my suggestion about reading from ~/.gnupg/gpg.conf was really just an expression of my desire for users to be able to choose their own custom settings for gpg to use.

However, now I'm curious: Why is it such a bad idea for programs to read from/rely on ~/.gnupg/gpg.conf?

Configurable options is a pony.

What does this mean?

@marmarek
Member

On Fri, Oct 23, 2015 at 07:16:42PM -0700, Axon wrote:

  1. Backup data will be integrity-protected twice - once by openssl, and additionally by gpg. I don't think it is a problem, but I'm not a cryptographer.

IANAC either. I take it integrity-protection from gpg can't be turned off?

Yes, I think it can't be disabled.

  1. I don't think we should parse ~/.gnupg/gpg.conf.

The only reason I suggested parsing ~/.gnupg/gpg.conf is because that seemed like the most logical place for the user to put their custom settings, but certainly if it's dangerous to parse it (or undesirable for another reason), then let's not do it. I really only meant to suggest that users should be able to specify their own options for s2k-cipher-algo, s2k-digest-algo, s2k-mode, and s2k-count (and any others that should be available).

Either let use the defaults

Sorry, I don't understand. Can you rephrase?

(possibly changed by user in that file),

Which file?

qvm-backup (one program) should not read ~/.gnupg/gpg.conf
(configuration of another program). We have no control over it, syntax
can change, its location may change, some other options may be
introduced (including another file?) etc.

So for each option we should either set it explicitly on command line
(always), or not set at all. Do not try to guess whether user set some
custom value in gpg.conf or not.

or override some option.

Would it be something like this?

qvm-backup --encrypt --s2k-count 1024 [...]

Yes.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

@adrelanos
Member

Axon:

@adrelanos:

Yes, programs should not rely on user settings in ~/.gnupg/gpg.conf indeed.

I'm glad you and @marmarek pointed this out to me. As mentioned in my previous message, my suggestion about reading from ~/.gnupg/gpg.conf was really just an expression of my desire for users to be able to choose their own custom settings for gpg to use.

However, now I'm curious: Why is it such a bad idea for programs to read from/rely on ~/.gnupg/gpg.conf?

It's not super bad, not a huge issue, just bad practice. Because that
file is for user settings. Not arbitrary tools. Avoiding conflicts with
whatever the user did. It's hard to modify a user config file.
Specifically if there is no .d folder and especially in the home
folder. By avoiding it, it gets easier to configure it tailed for the
tool in question.

Configurable options is a pony.

What does this mean?

An optional, non-essential, nice to have, thing if someone contributes it.

@adrelanos
Member

Axon:

integrity-protection from gpg can't be turned off?

I don't know if it should be turned off (why?) but it can be configured:

--force-mdc / --disable-mdc

@andrewdavidwong
Member

qvm-backup (one program) should not read ~/.gnupg/gpg.conf (configuration of another program). We have no control over it, syntax can change, its location may change, some other options may be introduced (including another file?) etc.

So for each option we should either set it explicitly on command line (always), or not set at all. Do not try to guess whether user set some custom value in gpg.conf or not.

Ok, that makes sense. Thank you for explaining it to me. :)

@andrewdavidwong
Member

It's not super bad, not a huge issue, just bad practice. Because that file is for user settings. Not arbitrary tools. Avoiding conflicts with whatever the user did. It's hard to modify a user config file. Specifically if there is no .d folder and especially in the home folder. By avoiding it, it gets easier to configure it tailed for the tool in question.

Right, makes sense.

An optional, non-essential, nice to have, thing if someone contributes it.

Oh, I see. Thanks. :)

I don't know if it should be turned off (why?) but it can be configured:[...]

Oh, ok. I guess the only reason to turn it off would be if we're worried about double integrity-protection causing problems, but it sounds like that's not really a concern.

@pqg
pqg commented Oct 25, 2015

FWIW, I wonder whether the minimum effort variation would be to apply the key stretching just once, to encrypt a random master key.

e.g.:

  1. A random Master Key, MK, of 512 bits is generated.
  2. This key is recorded in a file in the archive, say ./master-key, encrypted under the user's passphrase, using a key stretching method of some variety (I'll get back to the details of such later).
  3. The encryption key EK is derived as EK = HMAC_SHA512(MK, "qubes_backup_encryption_key")
  4. EK, truncated to 256 bits, is used to encrypt files in the current manner, but it is supplied to openssl via the parameter -K (i.e. as a raw key); I suspect this also means that the IV will have to be generated by the backup program and be manually fed in to openssl via -iv.
  5. The Authentication Master Key AMK is derived as AMK = HMAC_SHA512(MK, "qubes_backup_authentication_key")
  6. For each file, an Authentication Key AK_<filepath> is derived as AK_<filepath> = HMAC_SHA512(AMK, "<filepath>")
  7. Each file has it's HMAC taken and stored in an adjacent <blah>.hmac file, in the current manner.

Using a unique key for each file's MAC resolves a possible issue where an attacker could swap files around in the archive, potentially migrating an infection from a lower-security VM to a higher-security VM.

This also avoids having an attacker brute-force the user's passphrase against one/any of the HMACs.

A NIST-specified KDF could be used on the Master Key to derive the other keys instead, but none lend themselves to being specified in an openssl one-liner. Also, 2 of the 3 algos in NIST 800-108 more or less reduce to the above HMAC invocations if we don't need subkeys longer than the HMAC's block size.

Alas, this proposal requires more operations before verifying the ./backup-header file, which is bad both for code exposure concerns and backwards compatibility. However, preserving the ./backup-header.hmac in its current form will leave a method by which, as mentioned, an attacker can bypass the key stretching and brute-force the bare passphrase (HashCat has an "HMAC-SHA256 (key = $pass)" mode already that might be able to be dropped in to serve such a purpose, though I've not tested it).

I've not looked at the code yet, but if GnuPG does not do /too much/ parsing, maybe it's still okay to use it for key stretching?

There was actually a competition held recently to pick a good key stretching ("password hashing") algorithm. The winner was an entrant called Argon2, which beat out other entrants including yescrypt, which is an improved scrypt variant. Note that a key issue with scrypt is that it allows time/memory trade-offs to an extent that's undesirable in an optimal key stretching algorithm, though this issue can be mitigated by tuning the scrypt invocation to require sufficiently large memory. And all of these key stretching algorithms wipe the floor with GnuPG's S2K Mode 3, which is just an iterated hash, and therefore not memory-hard at all (i.e. GPUs and ASICs can attack it much faster than the CPUs that will be computing it under normal circumstances).

Of course, the problem with exotic key stretching algos is that it's hard to use them without making manual backup recovery dependent on non-ubiquitous software. And a million iterations of SHA512 is much better than no stretching at all.

I'll look into this a bit more later, and report back with any developments :)

@marmarek
Member

On Sun, Oct 25, 2015 at 10:13:47AM -0700, pqg wrote:

FWIW, I wonder whether the minimum effort variation would be to apply the key stretching just once, to encrypt a random master key.

Generally we want to avoid doing to much low-level crypto handling,
because we are not cryptographers.

Using a unique key for each file's MAC resolves a possible issue where an attacker could swap files around in the archive, potentially migrating an infection from a lower-security VM to a higher-security VM.

This isn't a problem, because paths are stored inside of archive(s). Outer layer is only for convenience (streaming the backup while creating it, partial restore). Take a look at point 7 here:
https://www.qubes-os.org/en/doc/backup-emergency-restore-v3/

Alas, this proposal requires more operations before verifying the ./backup-header file, which is bad both for code exposure concerns and backwards compatibility. However, preserving the ./backup-header.hmac in its current form will leave a method by which, as mentioned, an attacker can bypass the key stretching and brute-force the bare passphrase (HashCat has an "HMAC-SHA256 (key = $pass)" mode already that might be able to be dropped in to serve such a purpose, though I've not tested it).

Yes, IMO this is currently the main concern. When we get to the ./backup-header, we can have KDF paramenters, salt, iteration count, or any other configuration needed for restoring the rest of the backup. But before, we need something working without such knowledge...

I've not looked at the code yet, but if GnuPG does not do /too much/ parsing, maybe it's still okay to use it for key stretching?

I think we previously concluded, that we don't want to expose gpg to any unverified input.

Of course, the problem with exotic key stretching algos is that it's hard to use them without making manual backup recovery dependent on non-ubiquitous software. And a million iterations of SHA512 is much better than no stretching at all.

And also using exotic algorithms isn't a good idea, because those didn't received that much review/analysis/tests.

But generally IMHO scrypt as a KDF (exactly the thing it was designed for, right?) should be ok. If we'd need to employ KDF ourself at all...

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

@andrewdavidwong
Member

There was actually a competition held recently to pick a good key stretching ("password hashing") algorithm. The winner was an entrant called Argon2, which beat out other entrants including yescrypt, which is an improved scrypt variant. Note that a key issue with scrypt is that it allows time/memory trade-offs to an extent that's undesirable in an optimal key stretching algorithm, though this issue can be mitigated by tuning the scrypt invocation to require sufficiently large memory. And all of these key stretching algorithms wipe the floor with GnuPG's S2K Mode 3, which is just an iterated hash, and therefore not memory-hard at all (i.e. GPUs and ASICs can attack it much faster than the CPUs that will be computing it under normal circumstances).

Of course, the problem with exotic key stretching algos is that it's hard to use them without making manual backup recovery dependent on non-ubiquitous software. And a million iterations of SHA512 is much better than no stretching at all.

Exactly. I'm skeptical that the practical security benefits of using a newer KDF will outweigh the benefits of being able to do manual backup recovery in a variety of disaster scenarios. Security-concious users (which almost by definition includes all Qubes users) ought to be using reasonably long, high-entropy passphrases (I'll refer to these as "good" passphrases for the sake of brevity).

The current system (in which openssl applies a single round of md5) punishes users who pick good passphrases by capping maximum entropy at a measly 128 bits. We should fix that. However, my understanding (correct me if I'm wrong) is that S2K or PBKDF2, applied to good passphrases, will be uncrackable for the foreseeable long-term future (i.e., even if we assume many quadrillions of guesses per second, it would still take at least millions of years). If that's correct, then it seems to me that we should instead select the most trusted, tested, ubiquitous tools available. My backup is no good to me if I can't recover the data because the only software available to me is whatever's on an old Ubuntu disc, for example. (We can't predict in advance what kinds of emergency scenarios Qubes users might find themselves in, and under what conditions they might need manual access to their data.)

@andrewdavidwong
Member

Having said all that, if scrypt provides exactly what we're looking for, then it may be worthwhile to go with that. (And advise users to store a copy of scrypt with their backups, or something.) If anyone can get this right, it's Colin Percival.

@mfc
Member
mfc commented Oct 26, 2015

we should add a crypto tag to this (and any other similar issues) and I will create a thread with Cure53 (or NCC / iSec Partners) about this. They can provide free crypto advice to OTF-funded projects.

@marmarek marmarek added the crypto label Oct 26, 2015
@marmarek
Member

Great idea!

@pqg
pqg commented Oct 26, 2015

@axon-qubes

I concur with your position that, if at all possible, the data should be recoverable with an "old Ubuntu disc".

The scrypt binary and python-scrypt module don't seem to be currently packaged with Fedora anyway (they are available on Debian Jessie; I've not checked elsewhere). And keeping scrypt alongside the backup would be problematic if we need to invoke scrypt /before/ we can verify the integrity of anything.

As an aside, I think we should distinguish between 2 uses of "KDF":

  1. A means by which to derive subkeys of arbitrary length from a master key of fixed length.
  2. A means by which to perform "key stretching" (and salting) so as to make brute force attacks more costly on user-supplied keying material.

For instance, scrypt could provide function 2, but not function 1. Function 1 is served by things like NIST 800-108, which in the special case of all subkeys being of equal or shorter length than the underlying hash function, reduces to HMAC(MK, "\x01<subkey_purpose_specifier_string>\x00\x20"), where the \x01 and \x00 are no longer serving a particular purpose in this reduced form, and \x20 is the length of the resulting subkey. More conservative KDFs (purpose 1) use a single keystream that feeds back the result of the previous HMAC invocations or, more conservative still, they create one feedback pipeline whose output is passed to a second HMAC invocation to produce the actual keystream (TLSv1.2 does this, see section 5). However, all of these are prohibitively awkward in the "old Ubuntu disc" scenario. By contrast:

echo -n qubes_backup_authentication_key | openssl dgst -sha256 -hmac <Hex(Master Key)>

...should work almost anywhere, yield the desired different keys for different purposes, and be just as secure as long as SHA-256 is a strong hash function. (The fancier derivation methods are just meant to give some additional protection if it should turn out that the underlying hash function is weaker than expected.) Though note that the Master Key (randomly generated or the result of scrypt or some other key stretching) will be a binary blob, which will be tough to pass via -hmac unless encoded by hex (as above) or base64, which is kind of unsightly protocol-wise. On OpenSSL >= v1.0.0 (Ubuntu since at least 2011) the following invocation could be used instead to pass the master key hex-encoded:

echo -n qubes_backup_authentication_key | openssl dgst -sha256 -mac hmac -macopt hexkey:<Hex(Master Key)>

So that the output is actually HMAC_SHA256(MK, "qubes_backup_authentication_key") rather than HMAC_SHA256(Hex(MK), "qubes_backup_authentication_key") as it would have to be with -hmac.

However, my understanding (correct me if I'm wrong) is that S2K or PBKDF2, applied to good passphrases, will be uncrackable for the foreseeable long-term future (i.e., even if we assume many quadrillions of guesses per second, it would still take at least millions of years).

With no key stretching, e.g. just take the SHA-256 of the passphrase, an attacker capable of 10 quadrillion tries a second for 10 million years would exhaust an ~101-bit keyspace. This represents approximately an 8-word diceware passphrase (~103 bits). With ~1million rounds of key stretching, as in S2K Mode 3 at max settings, ~81 bits of input keyspace could be searched, which requires a diceware passphrase of 7 words (~90 bits) though you would get pretty close with 6 words (~78 bits).

A GPU running hashcat can currently trial on the order of 2 billion SHA-256 sums per second (2*10^9/sec) so 10^16/sec represents a powerful adversary indeed. 10 million years is of course just providing a robust margin of error, since in 30 years the crypto will be completely obsolete, and in 100 years we'll all be dead.

So 128 bits is still enough for anyone. Unless your threat model includes the arrival of quantum computers in the very near term, in which case you should double all your symmetric crypto key length requirements.

That said, as you originally advanced, even if nothing else changes from this discussion, -md sha256 should be added to the openssl enc invocations. It's much more conservative.

None of this, of course, helps with the MACs, where we really want to salt and stretch the user's passphrase before doing anything, even if the same resulting key is used for both verification and encryption.

Adding a salt not only avoids precomputaion attacks, but binds the files to the particular backup. While the renaming attack I previously feared is indeed impossible, I don't /think/ that there's currently any protection against shuffling in an identically-named VM from a different backup, as long as both backups were made with the same passphrase.

However, there then needs to be some way to store the salt and, if variable, any key stretching parameters, and load them before verification. Would it be an acceptable risk to parse just the key=values out of the ./backup-header and interpret those necessary to establish the key, then check the ./backup-header.hmac, and finally interpret the rest of the values (e.g. if, in the future, a timestamp is added to the header, it would be good to not have to parse that until after verification)? Setting hard line and file length limits might help limit the attack surface.

This certainly presents less attack surface than any way I can think of using GPG to do pre-verification stretching. Though, I'm aware I haven't actually proposed another way to perform key stretching yet. Python has baked-in hashlib.pbkdf2_hmac(), but only since v3.4 and v2.7.8, which are far too recent (mid 2014). I'd hoped it might be possible to use gpg --symmetric on a fixed value, hash the resulting "file", and use that as the key, but I've not been able to find a way to pass a salt and/or IV to GPG, so it unfortunately generates a random one on each invocation. The scrypt binary has the same problem, assuming its availability issues could be overcome.

The bcrypt functionality in python-passlib may be a reasonable bet. I can't guarantee how ubiquitous it is, but it's in Debian >=7 and backports for 6, definitely in Ubuntu >= 12.04, and available on Fedora 21. bcrypt has been fairly heavily popularized and so should common enough; I suspect that a working implementation should be more readily obtainable for our hypothetical frazzled power user than scrypt.

On the other hand, the scrypt binary is pretty easy to build if you've got gcc and autotools. As mentioned, it's only good for encrypting/decrypting files, not plain key stretching, so we'd be back to the ./master-key idea or similar, but it still presents a considerably smaller attack surface than GnuPG.

@marmarek
Member

Adding a salt not only avoids precomputaion attacks, but binds the files to the particular backup. While the renaming attack I previously feared is indeed impossible, I don't /think/ that there's currently any protection against shuffling in an identically-named VM from a different backup, as long as both backups were made with the same passphrase.

Yes, such attack probably is possible.

However, there then needs to be some way to store the salt and, if variable, any key stretching parameters, and load them before verification. Would it be an acceptable risk to parse just the key=values out of the ./backup-header and interpret those necessary to establish the key, then check the ./backup-header.hmac, and finally interpret the rest of the values (e.g. if, in the future, a timestamp is added to the header, it would be good to not have to parse that until after verification)? Setting hard line and file length limits might help limit the attack surface.

This might be some option if nothing better comes to us. Those
pre-verification options needs to be clearly marked (some prefix in
name?) and reduced to absolutely minimum. If we'd have to go that way, I
think this should include version field, which would determine most of
parameters. For example it would be bad idea to include hash algorithm
in those pre-authentication set, because the attacker would be able to
replace it with some weak value (vide "cipher_null" in LUKS header[1]).

The bcrypt functionality in python-passlib may be a reasonable bet. I can't guarantee how ubiquitous it is, but it's in Debian >=7 and backports for 6, definitely in Ubuntu >= 12.04, and available on Fedora 21. bcrypt has been fairly heavily popularized and so should common enough; I suspect that a working implementation should be more readily obtainable for our hypothetical frazzled power user than scrypt.

And according to previous comments, bcrypt is also somehow sensible
choice, right?

On the other hand, the scrypt binary is pretty easy to build if you've got gcc and autotools. As mentioned, it's only good for encrypting/decrypting files, not plain key stretching, so we'd be back to the ./master-key idea or similar, but it still presents a considerably smaller attack surface than GnuPG.

I think we can simply have hex-encoded, encrypted master key included in
the backup header.

And as the backup format is going to be more and more complex, maybe
it's a good idea to include emergency restoration script as one of
plain text, integrity protected files (similar to backup-header)? So the
emergency procedure would include manual verification of backup header
and that script, and then simply run the script.

[1]
https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-019-2015.txt

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

@andrewdavidwong
Member

I notice that all of pqg's messages have disappeared from this thread. pqg, did you intentionally delete them? If so, may I ask why? (Feel free to send me an encrypted message if you prefer.) Note that this may create difficulties for people who try to read this thread in the future (e.g., cryptographers interested in helping with this issue).

@marmarek
Member

I notice that all of pqg's messages have disappeared from this thread.

As the whole account...

@andrewdavidwong
Member

As the whole account...

Ah. In light of that, I'm not sure what to make of his/her/their previous suggestions...

@pqg
pqg commented Oct 27, 2015

Apparently I tripped over some bot heuristics and was hellbanned. Presumably because I am using Tor and posted some comments with some links in them.

I trust that you can see me again?

@marmarek
Member

I trust that you can see me again?

Yes.

@Rudd-O
Rudd-O commented Jan 17, 2016

Hi. Use the scrypt source code for key derivation. It's simple, good license, and lets you create arbitrarily hard derivations. I'm using it for a project I'm working on, and it works exactly fine.

@marmarek
Member

Take a look at #971 (comment) and #971 (comment) mostly about scrypt availability and usage for sole key derivation. One of important requirements for backup format here is ability to restore the data using commonly available tools, not necessary on Qubes OS.

@marmarek
Member

Python has baked-in hashlib.pbkdf2_hmac(), but only since v3.4 and v2.7.8, which are far too recent (mid 2014).

There is pbkdf2 implementation in passlib python module (http://pythonhosted.org/passlib/lib/passlib.utils.pbkdf2.html), for quite a long time. As for bcrypt in that library, I can see only an API to create a password hash, not sure how can it be applied as key derivation...

@marmarek marmarek modified the milestone: Release 4.0, Release 3.1 Feb 8, 2016
@Rudd-O
Rudd-O commented Feb 10, 2016

What about including an implementation of a command line program that simulates the six words of diceware using /dev/random? If that was a thing, then we could even have a backup option --diceware that generates and shows those six words on the standard output, telling the user to memorize them or back those words up somewhere safe. And that way you need no passphrase stretching like pbkdf2 because the entropy is good enough as is.

@andrewdavidwong
Member

I think that would be a poor substitute for proper key derivation. It's basically forcing the user to choose between two bad options (three including the status quo):

  1. Memorize a set of six words for every backup created (unrealistic if you make a new backup every week and want to be able to recover them years from now).
  2. Back up the word lists somehow, either in plaintext (significantly decreases security) or user-encrypted (in which case we're basically shifting our problem onto the user). It also makes the backups more likely to be unrecoverable in many disaster models, since now there are three different things needed to decrypt (the passphrase to decrypt the word list backup, the word list backup itself, and the associated Qubes backup).

And that way you need no passphrase stretching like pbkdf2 because the entropy is good enough as is.

But remember that low-entropy passphrases are not the main problem here. We tell users that they ought to have reasonably high-entropy passphrases and assume they will. The problem is that the current implementation takes the high-entropy passphrase a user provides and, in many cases, reduces its entropy.

@andrewdavidwong
Member

Thinking out loud:

  • A number of devastating bugs have been found in OpenSSL over the past couple of years.
  • OpenSSL still has not been audited.
  • There is strong evidence that openssl enc (the symmetric encryption module we use to encrypt Qubes backups) has not received much developer attention.
    • Basically, we encountered a bug where OpenSSL tries to encrypt data, then compress it. But of course encrypted data can't be compressed. The fact that this obvious bug has been known for so many years and still not been fixed (or the option removed) suggests that very few other organizations rely on openssl enc and that developers don't spend much time on the code.

Given all of this, I think it's not inconceivable (perhaps even likely) that there are more bad bugs lurking in openssl enc that could expose Qubes users' data in the future. If moving away from OpenSSL very soon isn't a viable option (due to lack of a crypto expert), we should at least consider alternatives like double-encryption schemes. For example, we initially backed away from using GPG because it did too much parsing of untrusted data before verifying authenticity, but what's preventing us from piping data through GPG, then through the current OpenSSL setup (perhaps even using two different algos, i.e., cascade encryption)? Presumably, we wouldn't be giving up anything on the authentication front, since the backup would look externally the same as it does now. We'd just be feeding openssl ciphertext instead of plaintext.

@Rudd-O
Rudd-O commented Mar 15, 2016

tbh, if all you want is a wrapper for the encryption with verification, I think you can do well with the plain scrypt command from the scrypt distribution. it encrypts and verifies, and I believe it does so in a streamable way (which is what you want), but if I am in error, you can always spit blocks of a specific size to the output file (or to multiple files) and only decrypt the first block to get to the header. The same file can then be scrypted --decrypt in the same order. You don't have to deal with gruyeressl, you don't have to deal with monstrous different algorithms, check the source and see what it does, it's very simple and it's the same shit used on the client side for the tarsnap service prior to sending it to the cloud, which is secure as far as we know

Build a package, ship it in the Qubes repos, drama solved.

@andrewdavidwong
Member

tbh, if all you want is a wrapper for the encryption with verification, I think you can do well with the plain scrypt command from the scrypt distribution. [...]

Sounds promising. @marmarek, what do you think?

@andrewdavidwong
Member

I'm curious to get others' opinions on some general questions:

  1. Do you think this whole "OpenSSL / weak key derivation scheme" thing is a significant problem, or not?
  2. If not, is it because...
    • You don't use qvm-backup as your main backup solution?
    • You already encrypt your data some other way before/after using qvm-backup?
    • You don't store your backups in a vulnerable place (e.g., only on local external hard drives, not in the cloud)?
    • You consider openssl enc secure enough as is?
    • Some other reason?

If it turns out that, e.g., hardly anyone else actually uses qvm-backup, then maybe this is a moot point. Or, even if people do use it, but I'm the only one worried about this, maybe I'm just being too paranoid. :)

@marmarek
Member
  • You already encrypt your data some other way before/after using qvm-backup?
  • You don't store your backups in a vulnerable place (e.g., only on local external hard drives, not in the cloud)?

For me - those two ;)

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

@andrewdavidwong
Member

For me - those two ;)

Ah, interesting. You encrypt before or after (or both)? (I'm guessing the whole external drive is LUKS encrypted?)

@marmarek
Member

(I'm guessing the whole external drive is LUKS encrypted?)

Exactly.

@marmarek
Member

(I'm guessing the whole external drive is LUKS encrypted?)

In fact this is an artifact of using old backup format, which didn't support encryption itself...

@andrewdavidwong
Member

In fact this is an artifact of using old backup format, which didn't support encryption itself...

Ah, yes, I remember those days. :)

But what, if anything, do you do about offsite backups, then?

@marmarek
Member

But what, if anything, do you do about offsite backups, then?

Keep that disk far enough...

@andrewdavidwong
Member

Keep that disk far enough...

Oh, ok!

@andrewdavidwong
Member

Should we just recommend that users do the same, I wonder? IOW, display a warning that says something like, "Do not rely on Qubes backup for strong encryption. Instead, encrypt the data yourself"?

@marmarek
Member

Sounds promising. @marmarek, what do you think?

AFAIR the whole point of using openssl enc was to use something broadly available for easy emergency restore on non-Qubes system. It looks like it is available in Debian, but not Fedora. Which in theory pass the "old Ubuntu" test.
While the tool is indeed quite simple, it is advertised as a demonstration of the scrypt key derivation function. Not sure if something meant for real-world usage. But the more I read that code the more I think it is good idea to use it.

What we'd still need, is some verification of backup header, which IMO shouldn't be encrypted for compatibility reasons (if we ever decide to change encryption again, to have ability to detect that during restore). Maybe backup-header.hmac file can be simply scrypt-encrypted version of the header (which should handle authentication itself)?

@andrewdavidwong
Member

AFAIR the whole point of using openssl enc was to use something broadly available for easy emergency restore on non-Qubes system. It looks like it is available in Debian, but not Fedora. Which in theory pass the "old Ubuntu" test.

It seems like scrypt is prevalent enough that it strikes a reasonable balance between availability and security. To my mind, the "old Ubuntu test" is passed if the user can, e.g., store a small binary with her backups that can be installed on commonly available systems without Internet access. The point of the test is just to make sure that we give reasonable consideration to data availability across plausible disaster scenarios. I don't think we should take the requirement too literally.

Maybe one possibility is to give the user the option to create backups using the current system (e.g., --version=3), in case they require the ubiquity of OpenSSL.

While the tool is indeed quite simple, it is advertised as a demonstration of the scrypt key derivation function. Not sure if something meant for real-world usage.

Yeah, that part is a bit worrying... When it comes to something fundamental like backup encryption, I would prefer to see Qubes stay on the conservative side and employ widely-acknowledged best practices. Unfortunately, the consensus in this area seems to be "use GPG."

What we'd still need, is some verification of backup header, which IMO shouldn't be encrypted for compatibility reasons (if we ever decide to change encryption again, to have ability to detect that during restore).
Maybe backup-header.hmac file can be simply scrypt-encrypted version of the header (which should handle authentication itself)?

Hm, I'm not sure I follow. Here's my understanding (tell me if I'm going wrong somewhere):

The current order:

  1. Verify backup-header.hmac.
  2. Read backup-header.
  3. [Continue...]

We do it this way because we don't want to parse backup-header before verifying that it's non-malicious.

So, suppose we decide to make backup-header.hmac by scrypt-encrypting backup-header (since it includes authentication). Then, sometime in the future, we decide to switch from scrypt to foocrypt (which also includes authentication). Won't FutureQubes have to try decrypting backup-header.hmac with foocrypt, then scrypt (or vice versa) until one works (authenticates), before reading backup-header?

If so, then we always have to decrypt backup-header.hmac before reading backup-header. But if backup-header.hmac is just the ciphertext of backup-header, then this first step will always yield the plaintext of backup-header. If that's correct, then why store a duplicate plaintext of backup-header alongside backup-header.hmac? What does that gain us?

@marmarek
Member

Maybe one possibility is to give the user the option to create backups using the current system (e.g., --version=3), in case they require the ubiquity of OpenSSL.

Current design is to "create always latest backup version, be able to restore all of them". Support for creating multiple backup versions will make the code more error prone (more paths to test, more corner cases etc). This is something we really don't want to happen in backup handling code...

Hm, I'm not sure I follow. Here's my understanding (tell me if I'm going wrong somewhere):

The current order:

  1. Verify backup-header.hmac.
  2. Read backup-header.
  3. [Continue...]

Correct.

Won't FutureQubes have to try decrypting backup-header.hmac with foocrypt, then scrypt (or vice versa) until one works (authenticates), before reading backup-header?

Yes, indeed.

If that's correct, then why store a duplicate plaintext of backup-header alongside backup-header.hmac? What does that gain us?

To use the same header extraction code for every backup version. If backup-header.hmac turns out to be scrypt-ed backup header, the plaintext one can be simply deleted (or better - compared for authenticity). But if not, it may be hmac of the actual header stored in backup-header file (the current backup format). Also having plaintext backup-header will help with disaster recovery - user may look at it (especially version field) to see what instruction use to manually restore the data (including header verification...).

@andrewdavidwong
Member

Current design is to "create always latest backup version, be able to restore all of them". Support for creating multiple backup versions will make the code more error prone (more paths to test, more corner cases etc). This is something we really don't want to happen in backup handling code...

Ah, good point. (Especially when the the potential benefit is so dubious.)

To use the same header extraction code for every backup version. [...]

Ok, that makes sense.

Also having plaintext backup-header will help with disaster recovery [...]

True. Even though we advise against reading it before verifying authenticity, in many disaster scenarios authenticity may not matter much.

@andrewdavidwong
Member

One reason not to use scrypt enc as-is: It relies on a homebrew implementation of AES. Granted, it was written by Colin Percival, so it is much more trustworthy than most roll-your-own AES implementations. But still, it is not as thoroughly tested and vetted as the AES implementations in, e.g., GPG and dm-crypt. So, the safer way to go would be to use scrypt for the KDF and use GPG for the crypto, IMHO.

@andrewdavidwong
Member

Just to summarize everything so far, it sounds like there are two three main options on the table:

1. Use GPG for encryption and OpenSSL for verification

Pros:

  • Uses existing tools
  • Requires no crypto expertise and no building; just using pre-installed binaries
  • These tools are ubiquitous and widely-used
  • These tools are well-tested and widely-respected (for the proposed functions)

Cons:

  • Does not result in the strongest possible KDF
    • (It is debatable whether this is really a "con." As we established above, GPG's KDF is fine if you're using a decent passphrase, which you should be. And it's also not much of a con if you're skeptical of newfangled KDFs which haven't yet stood the test of time.)
  • Requires two different tools
    • (This may not matter, since the two tools are so ubiquitous that there is unlikely to be a situation where you won't have access to both.)
  • Still relies on OpenSSL
    • The problems with OpenSSL are with the enc function (poorly-written, little-used). If, by contrast, the HMAC-related functions are better-written and more widely-used, then this may not be a problem.

2. Use scrypt for everything

Pros:

  • Includes a very strong, modern KDF
  • Requires no crypto expertise
  • One tool for everything
  • Reputable author

Cons:

  • Significantly less ubiquitous than the other tools, making manual backup recovery less certain and harder for users
  • Uses a custom implementation of AES
    • (Less tested, not widely-used, fewer eyes on the code to catch bugs)
  • Still not the strongest, most modern KDF
    • (As mentioned above, this may not be a con.)
  • Introduces more complexity than simply using pre-installed tools
    • (More chances for mistakes to be made)
  • [Marek] It isn't easy for scripted usage - reads password from /dev/tty. I can workaround this, but it will not be nice code...

3. Use the current system, but pass -md sha256 to openssl enc

Pros:

  • This is the new default in OpenSSL 1.1.0
  • Requires the least amount of developer time and effort (and minimal code changes)
  • Allows us to keep using the same tool without introducing any new ones
  • sha256 is much less likely to reduce entropy of user's passphrase (compared to md5)

Cons:

  • Still hardcoded to one iteration
  • Still uses the same passphrase for verification and encryption
    • As discussed below, this could be avoided by passing sha512(passphrase) to openssl dgst.
  • Still relies on OpenSSL
    • But at least now we evidence that enc isn't totally neglected.

(As I mentioned above, when it comes to something like backup encryption, I'd prefer to see Qubes stay on the conservative side, so, FWIW, I'm leaning toward option 1. Option 3 might be better for now, if it can be done immediately at little cost yet provide a pareto improvement over the current system.)

@marmarek
Member
  • Still relies on OpenSSL
    • The problems with OpenSSL are with the enc function (poorly-written, little-used). If, by contrast, the HMAC-related functions are better-written and more widely-used, then this may not be a problem.

If I understand correctly, using openssl for verification still requires
some KDF...

One more scrypt cons:

  • it isn't easy for scripted usage - reads password from /dev/tty. I
    can workaround this, but it will not be nice code...

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

@andrewdavidwong
Member

If I understand correctly, using openssl for verification still requires some KDF...

I think that depends on what we consider the "requirements" to be. Currently, we use OpenSSL for verification without any kind of KDF. We just feed the user's passphrase directly to openssl dgst -hmac.

One option would be to keep doing this. Since GPG applies its own KDF (S2K) to the passphrase, the resultant verification and encryption keys would be significantly different. However, the passphrases fed to openssl and gpg would still be the same, and that's undesirable. (E.g., exploiting a flaw in OpenSSL's implementation of HMAC-SHA512 which allows the attacker to recover the passphrase would allow her to then immediately decrypt the GPG-encrypted data. Unlikely, perhaps, but easily avoided by simply by using two different passphrases.)

OpenSSL is really just there to protect GPG from unverified input. So, if all we really want is for the two passphrases to be different, a simple solution could be something like: feed sha512(passphrase) to openssl and feed passphrase to GPG (on which GPG will then apply its own KDF).

IMHO, our goal isn't to apply our own key-stretching (we trust the user to supply a reasonable passphrase). Rather, it's just to avoid "key-shortening" (which the openssl enc's single round of md5 currently does in many cases) and to avoid the "same passphrase" problem mentioned above.

Either way, it would still better than what we do right now. Plus, a solution like this ensures that disaster recovery remains possible without yet another external tool (i.e,. the user doesn't need a copy of exotic-kdf-tool, just the ability to compute sha512(their_passphrase)).

It may not be "ideal," but it seems like it would fix the main problems we currently have, which are: (1) the issues with openssl enc (entropy-reducing "KDF," code neglect) and (2) using the same passphrase for verification and encryption. I worry that any more "ideal" solution would require much more code and complexity (and therefore not really be ideal).

One more scrypt cons:

  • it isn't easy for scripted usage - reads password from /dev/tty. I can workaround this, but it will not be nice code...

Added.

@andrewdavidwong
Member

OpenSSL changes between 1.0.2g and 1.1.0:

  *) Changed default digest for the dgst and enc commands from MD5 to
     sha256
     [Rich Salz]

This reminds me that there's a third option: simply start passing -md sha256. I'll edit this back into the previous post, along with pros and cons.

@marmarek
Member

I think that depends on what we consider the "requirements" to be. Currently, we use OpenSSL for verification without any kind of KDF. We just feed the user's passphrase directly to openssl dgst -hmac.

Isn't that exactly what we're trying to solve in this ticket? Weak/no KDF usage for (any of) authentication or encryption.
If I understand correctly, this makes attack easier because someone may launch attack on password by just attacking hmac, then use guessed password for decryption. So if any of those will be weak (cheap to launch dictionary/bruteforce attack), it will help with attacking the other part, even if decent KDF is used in that other part.

As for -md sha256 idea - we already pass -sha512 to openssl dgst. But not openssl enc - here it will indeed somehow improve the situation. At least it will not reduce passphrase to 128 bits.
Also, indeed some idea may be using SHA512(passphrase) for both operations. Or even SHA512(passphrase + "hmac") and SHA512(passphrase + "enc"). This will produce different keys. This looks like solution for the original problem, but as I'm not a cryptographer, I don't know if generally a good idea... Similar idea was raised in #971 (comment)

@andrewdavidwong
Member

Isn't that exactly what we're trying to solve in this ticket? Weak/no KDF usage for (any of) authentication or encryption.

Well, it's one of the issues. The current three problems are:

  1. md5(passphrase) is capping entropy at 128 bits.
  2. Same passphrase is being fed to dgst and enc.
  3. openssl enc seems shoddy (admittedly this one is debatable).

If I understand correctly, this makes attack easier because someone may launch attack on password by just attacking hmac, then use guessed password for decryption. So if any of those will be weak (cheap to launch dictionary/bruteforce attack), it will help with attacking the other part, even if decent KDF is used in that other part.

Yes, this is almost exactly the same situation I mentioned above.

But, again, there is a big middle-ground between (a) passing the same raw passphrase to dgst and enc, and (b) adding some full-blown KDF to the process. One example of such a middle-ground solution is using, e.g., sha512(passphrase) (discussed below).

As for -md sha256 idea - we already pass -sha512 to openssl dgst. But not openssl enc - here it will indeed somehow improve the situation. At least it will not reduce passphrase to 128 bits.

Yes, and it seems like there's no reason not to do this immediately. After all, it will happen by default if/when we upgrade to OpenSSL 1.1.0. Might as well start now and reap some benefit in the meantime.

Also, indeed some idea may be using SHA512(passphrase) for both operations. Or even SHA512(passphrase + "hmac") and SHA512(passphrase + "enc"). This will produce different keys. This looks like solution for the original problem, but as I'm not a cryptographer, I don't know if generally a good idea... Similar idea was raised in #971 (comment)

It may not be optimal from a cryptography standpoint, but surely it is better than what we do now (passing the same bare passphrase to both dgst and enc). If it would be trivial to implement, then it seems like we have nothing to lose and a fair amount to gain. Why not pick the low-hanging fruit?

@ag4ve
ag4ve commented Mar 17, 2016

Any reason not to just use a 3rd party OSS tool for this like duplicity?
Alternatively, I believe rsync and LUKS containers would fit the bill as
well.
On Mar 17, 2016 6:22 PM, "Axon" notifications@github.com wrote:

Isn't that exactly what we're trying to solve in this ticket? Weak/no KDF
usage for (any of) authentication or encryption.

Well, it's one of the issues. The current three problems are:

  1. md5(passphrase) is capping entropy at 128 bits.
  2. Same passphrase is being fed to dgst and enc.
  3. openssl enc seems shoddy (admittedly this one is debatable).

If I understand correctly, this makes attack easier because someone may
launch attack on password by just attacking hmac, then use guessed password
for decryption. So if any of those will be weak (cheap to launch
dictionary/bruteforce attack), it will help with attacking the other part,
even if decent KDF is used in that other part.

Yes, this is almost exactly the same situation I mentioned above.

But, again, there is a big middle-ground between (a) passing the same raw
passphrase to dgst and enc, and (b) adding some full-blown KDF to the
process. One example of such a middle-ground solution is using, e.g.,
sha512(passphrase) (discussed below).

As for -md sha256 idea - we already pass -sha512 to openssl dgst. But not openssl
enc - here it will indeed somehow improve the situation. At least it will
not reduce passphrase to 128 bits.

Yes, and it seems like there's no reason not to do this immediately. After
all, it will happen by default if/when we upgrade to OpenSSL 1.1.0. Might
as well start now and reap some benefit in the meantime.

Also, indeed some idea may be using SHA512(passphrase) for both
operations. Or even SHA512(passphrase + "hmac") and SHA512(passphrase +
"enc"). This will produce different keys. This looks like solution for the
original problem, but as I'm not a cryptographer, I don't know if generally
a good idea... Similar idea was raised in #971
#971 (comment)

It may not be optimal from a cryptography standpoint, but surely it is
better than what we do now (passing the same bare passphrase to both dgst
and enc). If it would be trivial to implement, then it seems like we have
nothing to lose and a fair amount to gain. Why not pick the low-hanging
fruit?


You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#971 (comment)

@Rudd-O
Rudd-O commented Mar 18, 2016

If I may for a moment explain -- I do use qvm-backup but I am unhappy with it, because there is no support for encrypted incremental backups. This means I have to back up close to a gig of stuff EVERY TIME. This is slow, it prevents me from using the machine being backed up, and it's so fucking tedious, I rarely do it.

We need a better solution that will (a) support running the backup concurrently with the underlying VMs running (b) support scripting and sending off to external storage beyond just USB drives (c) support incremental backup in the right way, so that backups can be finished faster. I believe what this means is embracing some sort of WAFL / COW file system, and backing it up by replicating (send/recv) to a remote, encrypted disk or disk image. qvm-backup just doesn't cut the mustard.

@marmarek
Member

I agree that requirement to shutdown all the VMs for the backup is not convenient at least...

(c) support incremental backup in the right way, so that backups can be finished faster.

There was a discussion about it linked to #858

(a) support running the backup concurrently with the underlying VMs running

This is also considered as part of #858

(b) support scripting and sending off to external storage beyond just USB drives

This is already possible - you can enter some command instead of directory path and the backup will be streamed to its stdin. For example ssh somehost dd of=/path/on/remote/machine.

@marmarek
Member

It may not be optimal from a cryptography standpoint, but surely it is better than what we do now (passing the same bare passphrase to both dgst and enc). If it would be trivial to implement, then it seems like we have nothing to lose and a fair amount to gain. Why not pick the low-hanging fruit?

One reason: to not change the backup format too frequently
(compatibility, number of combinations to test). There will be new
backup format (version += 1) in Qubes 4.0, because qubes.xml format is
changed. So we can bundle this change with fixing problem discussed in
this ticket.

For now I'll ignore the first option from your list (gpg+openssl), as it looks to only introduce complexity while the same gains can be achieved by much simpler option "3" (openssl enc -md sha512 + sha512(passphrase + 'hmac') for openssl dgst).

So, we have two options:

  1. use scrypt (option "2") - probably the best of those considered from cryptography POV, but with some practical drawbacks as described in #971 (comment)
  2. Slightly modify the current system (option "3") - easiest to implement

The question is, whether option "3" is good enough? It is surely better than the current implementation...

@andrewdavidwong
Member

@Rudd-O: Thank you for sharing your experience! I agree with you and @marmarek about the inconvenience of the current system. I also think that an incremental backup system is desirable, but there would still be a need for the ability to create immediate full backups (e.g., for system migration).


One reason: to not change the backup format too frequently
(compatibility, number of combinations to test). There will be new
backup format (version += 1) in Qubes 4.0, because qubes.xml format is
changed. So we can bundle this change with fixing problem discussed in
this ticket.

Ah, that's a good point. Changing the format too frequently would be bad. Ok, so bundling with 4.0 sounds good.

The question is, whether option "3" is good enough? It is surely better than the current implementation...

IMHO, we need to be careful not to let the perfect be the enemy of the good. Option 3 makes the current system better without making anything worse. So, if option 3 is not good enough, then the current system falls even shorter of being good enough. In light of this, I can only think of a few reasons not to go with option 3 (but maybe there are more):

  • We'd rather put the development time into other things.
  • We expect a better solution will fall into our laps soon.
  • We don't want to change the backup format only to have to change it again soon after.

All can be legitimate reasons, of course, but IIUC, the development time would be pretty minimal, and we don't expect a better solution to fall into our laps anytime soon. The last two reasons are likely to be related. (If we thought a better solution was about to fall into our laps, we'd want to hold off so that we don't have to change the format again so quickly afterward.) But this attitude can also be taken to an extreme. It can paralyze us with the fear that something better is always over the horizon, so we never make the easy improvements that we can make.

@andrewdavidwong
Member

Ok, I thought of one more potential concern about option 3:

Even though option 3 seems simple, nothing in cryptography is ever really simple. Our proposal is to compute sha512(passphrase + 'hmac') and feed it to openssl dgst as a passphrase. There are various problems associated with using a short, static salt (e.g., the birthday attack) and using a standard digest algorithm to hash passwords (instead of a true KDF). Many of these problems are more relevant to protecting online databases than to file encryption at rest, but the point is that we don't know enough about cryptography to know whether a seemingly innocuous move (like adding 'hmac' as a salt) could somehow decrease security. For example, there are even tricky problems that arise with how you concatenate strings.

In other words, maybe even the simplest option (3) requires more crypto expertise than we have available to us.

The standard advice is to use an actual KDF rather than trying to roll your own. Our problem is that we need something commonly available that can be used from the command-line, and apparently nothing like that is available.

Except that's not entirely true. OpenSSL includes the command passwd. From the man page:

The passwd command computes the hash of a password typed at run-time or the hash of each password in a list. The password list is taken from the named file for option -in file, from stdin for option -stdin, or from the command line, or from the terminal otherwise. The Unix standard algorithm crypt and the MD5-based BSD password algorithm 1 and its Apache variant apr1 are available.

The problem is that it uses only very old and weak algorithms (md5 and crypt), which would defeat the purpose of passing -md sha256 to enc, since the initial step would cap user passphrase entropy at 128 bits (and use a weak set of algorithms, to boot).

@marmarek
Member

openssl passwd by default uses a random salt, so it isn't good for
KDF. There is an option for using static salt (provided on command
line), but then again - we'd need somehow to carry this salt (in backup
header?). The same problem which is solved internally by scrypt (it
uses own header in encrypted files), or in case of just scrypt KDF
(connected with openssl enc) - by storing its parameters in backup
header.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

@andrewdavidwong
Member

@ag4ve:

Any reason not to just use a 3rd party OSS tool for this like duplicity? Alternatively, I believe rsync and LUKS containers would fit the bill as well.

IIUC, the main problems with those options are:

  • Dom0 has no network access (by design).
  • The backup shouldn't have to be stored in dom0 (in its entirety, for any length of time)(since there might not be enough space, among other reasons).
  • The backup must be encrypted before it leaves dom0.
  • Upon restore, the backup needs to be verified (for integrity and authenticity) before the data is significantly parsed (GPG fails here). (Otherwise, an attacker could swap your backup with a malicious payload, which then compromises dom0 when you attempt to restore from it.)
@andrewdavidwong
Member

In other words, maybe even the simplest option (3) requires more crypto expertise than we have available to us.

Clarification: This applies only to the passphrase hashing part. The part about passing -md sha256 to enc should still be done, if nothing else.

@defuse
defuse commented May 6, 2016 edited

Hey! I use Qubes backups heavily (I'm restoring from a backup as I type this), so I'm willing to provide some free consulting on this issue. I just skimmed this thread, but I understand:

  1. We're worried about a weakness in openssl enc's password-to-key KDF.
  2. It's crucial to retain ease of decrypting with standard tools.

The main concern I see being brought up is OpenSSL's use of MD5 and that this might cap security at 128 bits. This isn't the right thing to be worrying about. I'm looking at the command line we use for encryption and the source code for openssl enc which calls EVP_BytesToKey. According to the algorithm described in this documentation, no entropy would actually be lost assuming MD5 is secure. I am fairly confident that the known weaknesses in MD5 do not significantly decrease security here either. In other words, option 3 of adding -md sha256 to the openssl enc command gains us negligible security benefits. In either case, a good 256-bit passphrase is still 256 bits of security, not 128. (But using -md sha256 is a good thing to do anyway, just in case.)

I'm concerned about the following things:

  1. Is a random salt being used right now? openssl enc should spit out a salt unless you pass the nosalt option, and then that salt should be provided upon decryption. Could you point me to the code that does this? This is probably the most important thing Qubes needs to be doing. (Edit: Actually I may be misreading the OpenSSL code.)
  2. Using separate systems (GnuPG and OpenSSL) for encryption and authentication, where one does key stretching and the other doesn't, negates the benefit of key stretching. Key stretching is really important, mandatory I would even say, when you're turning a passphrase into a key.
  3. I'd like to verify how the HMAC is being computed/checked (please point me to code!).

I strongly recommend switching to something with decent key stretching (e.g. at least PBKDF2, or whatever the thing in GnuPG is). I'm willing to provide advice on this. Could someone point me to where the relevant code is (for creating a backup, and then restoring it)? Thanks!

@defuse
defuse commented May 6, 2016 edited

To answer my own concern (1) above, the passphrase effectively isn't protected by a salt. According to these instructions for manual restoration of the backup, there's a file called backup-header whose contents are the same for all users having the same Qubes version. Inside a backup-header.hmac file there's the result of running openssl dgst -sha512 -hmac "your_passphrase" backup-header.

This enables precomputation attacks: An attacker could take the contents of backup-header for the most common Qubes version and build up a database (or rainbow tables) of what the backup-header.hmac contents would be under different passphrases. They could then use that database to quickly map the victim's backup-header.hmac contents back to their password, if it's in the database (or rainbow table).

Edit: use better wording

@defuse
defuse commented May 6, 2016 edited

Strong +1 to option 2 of using scrypt. Speaking as a user, I'm very willing to download and build the scrypt tarball to get the increased security. It's even in the debian repositories.

@andrewdavidwong
Member

I use Qubes backups heavily (I'm restoring from a backup as I type this), so I'm willing to provide some free consulting on this issue.

Thank you!

The main concern I see being brought up is OpenSSL's use of MD5 and that this might cap security at 128 bits. This isn't the right thing to be worrying about. [...] In either case, a good 256-bit passphrase is still 256 bits of security, not 128.

Can you clarify that? Suppose my passphrase is a truly random 256-bit string. openssl enc takes md5(passphrase), resulting in a 128-bit string. How have I not lost entropy here?

Here's the backup code:
https://github.com/QubesOS/qubes-core-admin/blob/master/core/backup.py

@soatok
soatok commented May 6, 2016

How have I not lost entropy here?

See:

This isn't the right thing to be worrying about.

@andrewdavidwong
Member

How have I not lost entropy here?

See:

This isn't the right thing to be worrying about.

These are two logically distinct claims:

  1. md5(passphrase) doesn't cap entropy at 128 bits.
  2. md5(passphrase) caps entropy at 128 bits, but that's not the right thing to be worrying about.
@defuse
defuse commented May 6, 2016 edited

@andrewdavidwong: I'm making claim number (2). md5(passphrase) certainly caps entropy at 128 bits. openssl enc is doing something different. What openssl enc uses as the key is...

md5(passphrase || salt) || md5(md5(passphrase || salt) || passphrase || salt)

...according to the fact it uses EVP_BytesToKey and these EVP_BytesToKey docs. (The source code is here but it's pretty terrible to read.). There are probably other problems with this KDF, but it's not obvious that entropy is capped at 128 bits, since the passphrase is used once to make the first 128-bit half and then again in a different way to make the second 128-bit half. The first and second halves are both capped at 128 bits but the end result probably isn't.

@andrewdavidwong
Member

Ok, that helps clarify things for me. Thanks!

@tomrittervg

Hi all. I was also pointed at this thread. I run NCC Group's Cryptography Services practice. (And actually see a reference to NCC earlier - but we never got connected apparently. That's a shame, I'd have been happy to help.)

@defuse is pretty smart, and his statements all seem reasonable to me, but I might reinforce/add to them:

  1. Why are you using CBC+HMAC separately? Why not use GCM mode? Even if there's a reason to have the authenticity check separate, I would recommend encrypting with GCM mode anyway to avoid some sort of time-of-check-time-of-use bitflipping attack.

  2. One passphrase is used to feed into both the HMAC and the encryption? If the key derivation steps are the same for both, you'd be using the same key for both. That's not ideal. The correct way to derive separate keys from identical input entropy is with HKDF (https://tools.ietf.org/html/rfc5869) which, haha, is not trivial to implement. At the bare minimum, one should do something like HMAC(entropy, "encryption key") and HMAC(entropy, "hmac key") and use the output of those as the respective keys.

That said - this type of academic analysis (why exactly HKDF is good, and HMAC(entropy, "encryption key") or SHA(entropy | "encryption key") are bad) has never been my strong suit. I don't believe there's practical attacks here but... well that's why we go with the safe stuff in crypto.

  1. I agree that Key Stretching, and the use of a unique salt when using a passphrase, is crucial for security. Salt to avoid precomputation and stretching to make brute forcing much more painful. PGP's stretching algorithm (S2K Iterations) is an odd standard, but it's a standard. It's about on par with PBKDF2 in terms of security. Scrypt is better - you get (some) memory hardness. I'm not familiar enough with scrypt to recommend parameters though, but I bet you could find someone to help you tweak things to your needs. Maybe a judge or participant in https://password-hashing.net/

Omitting key stretching removes the safety net for users. With key stretching a user with an 'okay' passphrase might be safe, without there's a much higher chance of getting cracked. So perhaps you decide that operating without a safety net is okay for the time being, and you come back to it later - but I wouldn't choose to forgoe it forever.

@andrewdavidwong
Member

Thanks, @tomrittervg and @defuse! We fully agree that the system is flawed in the ways you two have helpfully pointed out. Our current problem is that we don't have anyone with the relevant expertise who's willing to help us implement a better system. Is that something that either of you, or anyone you know, would be willing to do?

@defuse
defuse commented Jun 7, 2016

I'm super busy at the moment, so I can't commit to being able to do anything right now. Using Colin Percival's scrypt tool should be pretty straightforward and hard to mess up, and I'd be happy to review any implementation that comes into existence.

@andrewdavidwong andrewdavidwong added a commit that referenced this issue Jun 8, 2016
@andrewdavidwong andrewdavidwong Track #971 f7e563c
@andrewdavidwong
Member

Using Colin Percival's scrypt tool should be pretty straightforward and hard to mess up, and I'd be happy to review any implementation that comes into existence.

@marmarek, what do you think?

@marmarek
Member

Last time I've checked it wasn't easy to provide passphrase from anything but terminal (it was reading it from /dev/tty). So not trivial to integrate. But probably doable.

@Rudd-O
Rudd-O commented Jun 12, 2016

Duplicity does not need to have network access to work. It just needs to have a backend specific to inter-VM backup:

https://bazaar.launchpad.net/~duplicity-team/duplicity/0.7-series/view/head:/duplicity/backends/ssh_pexpect_backend.py

@Rudd-O
Rudd-O commented Jun 13, 2016

SRC RPM for tarsnap (containing scrypt) https://koji.rpmfusion.org/koji/buildinfo?buildID=688

@marmarek
Member

SRC RPM for tarsnap (containing scrypt) https://koji.rpmfusion.org/koji/buildinfo?buildID=688

There is also package for fc23, tagged with "f23-nonfree". I wonder what this means in practice... Will it be in some separate repository?

@Rudd-O
Rudd-O commented Jun 21, 2016 edited

Nonfree means things built outside Amerika because of intellectual monopoly bullshit.

To the extent that I know, that may just be a miscategorization, as tarsnap is bona fide open source which you can download from the tarsnap site.

@marmarek
Member

If going with scrypt (which looks like the best option according to above comments), I see it:

  • would replace having separate .hmac files, as scrypt handle this already
  • as a consequence will make impossible to create non-encrypted backup (not a problem at all)
  • not having backup-header.hmac will make slightly harder to handle both old and new backups with the same tool, but this is still doable.

This change makes security of scrypt tool very critical to security of Qubes backups. Both in terms of used encryption, and correctly handling (potentially malicious) data during decryption.

At the same time we can simplify backup format even more: get rid of inner tar layer. Currently it serves two purposes:

  1. Efficiently handle sparse files.
    This can be replaced by a compression. Somehow less effective, but still - stream of compressed zeroes isn't that big. And when using pigz instead of just gzip it isn't that bad (but of course still much slower than simply not storing it at all)
  2. Authenticate file name inside (inner tar archive is integrity protected, while outer one isn't).
    This can be replaced by prefixing password (given to scrypt for given file) by expected file name (and some separator, like \x01). scrypt authenticate file content using this password, so swapping files should be mitigated here.

Attack not mitigated by any of those, is replacing whole VMs with the same VM from older/newer backup (created with the same passphrase). This is not much different than replacing the whole backup archive. Can be mitigated by user by using different passphrases for different backups (like append a number, or a date).

Another reason to drop inner tar layer - it is no longer effective if source isn't sparse file, but LVM thin volume (which will be the case in Qubes 4.0). Actually I haven't found out yet how to create tar archive from block device content, without dumping it to a file first (using python tarfile module isn't any better, as it doesn't support sparse files).

Some benchmark about tar/gzip/pigz:

[user@testvm ~]$ truncate -s 1G sparse.file
[user@testvm ~]$ time tar cS sparse.file |wc -c
10240

real    0m0.041s
user    0m0.002s
sys 0m0.030s
[user@testvm ~]$ time gzip < sparse.file |wc -c
1042069

real    0m15.087s
user    0m14.211s
sys 0m0.881s
[user@testvm ~]$ time pigz < sparse.file |wc -c
1171473

real    0m9.444s
user    0m31.015s
sys 0m2.699s
@andrewdavidwong andrewdavidwong added a commit to QubesOS/qubes-doc that referenced this issue Sep 26, 2016
@andrewdavidwong andrewdavidwong Add considerations on choosing a backup passphrase
Includes Marek's point about malicious backup substitution in
QubesOS/qubes-issues#971
b64f4b9
@andrewdavidwong
Member

Attack not mitigated by any of those, is replacing whole VMs with the same VM from older/newer backup (created with the same passphrase). This is not much different than replacing the whole backup archive. Can be mitigated by user by using different passphrases for different backups (like append a number, or a date).

That's a good point (added to documentation).

IMHO, neither variety should be considered a critical attack, since in any case a backup authenticated by the user's passphrase is trusted insofar as the user has chosen to create the backup using that passphrase.

One thing to note is that it's possible to "DoS" someone who uses a different passphrase for each backup (e.g., date appended) by changing around the file names of their backups. This is a different kind of DoS from simply deleting all of their backups, since the victim won't realize what's happened unless/until they try to restore from one of the backups.

Some benchmark about tar/gzip/pigz: [...]

How should these results be read? For example, what's the difference between "real" and "user"?

@marmarek
Member

One thing to note is that it's possible to "DoS" someone who uses a different passphrase for each backup (e.g., date appended) by changing around the file names of their backups. This is a different kind of DoS from simply deleting all of their backups, since the victim won't realize what's happened unless/until they try to restore from one of the backups.

Mostly the same as filling the backup file with junk data...

Some benchmark about tar/gzip/pigz: [...]

How should these results be read? For example, what's the difference between "real" and "user"?

"real" is actual time spent on the operation (end_time-start_time), "user" is CPU time used (including multiple cores etc - 1s of fully utilizing 4 cores is 4s "user" time and 1s "real" time). "sys" is time spent on system calls (kernel code).

@Rudd-O
Rudd-O commented Sep 27, 2016 edited

@marmarek the gains from sparse file storage aren't so much on the read (backup) side (though, unlike your short benchmark, the are quite big if you have rotational disks like I do). They are mostly on the write side. When your system has to write an enormous file to disk, and allocate those zeroes that should be unallocated, you end up spending a MONSTROUS amount of disk space and disk activity just to store those zeroes. On a system like Qubes, which relies on thin provisioned storage, getting rid of sparse file storage is a bad idea.

Maybe a purpose-built, short C or Go program, that reads from device files and writes tar format to its output, is the right thing to use here. It avoids using tar directly, it can detect rows of zeroes and output them as sparse blocks, and it isn't needed during restore (as you can use tar directly in that case). Those are my thoughts. What do you think?

@marmarek
Member

@marmarek the gains from sparse file storage aren't so much on the read (backup) side (though, unlike your short benchmark, the are quite big if you have rotational disks like I do). They are mostly on the write side. When your system has to write an enormous file to disk, and allocate those zeroes that should be unallocated, you end up spending a MONSTROUS amount of disk space and disk activity just to store those zeroes. On a system like Qubes, which relies on thin provisioned storage, getting rid of sparse file storage is a bad idea.

This isn't a problem. dd conv=sparse in the middle does the trick.

Maybe a purpose-built, short C or Go program, that reads from device files and writes tar format to its output, is the right thing to use here. It avoids using tar directly, it can detect rows of zeroes and output them as sparse blocks, and it isn't needed during restore (as you can use tar directly in that case). Those are my thoughts. What do you think?

This is what I'm currently exploring, as I've failed to find any other method (tried many tar implementations, other archive formats etc).

@Rudd-O
Rudd-O commented Sep 28, 2016

Maybe take a look at Go for that custom program. It's batteries-built-in, it's very efficient, it's a safe language. It's got what you need.

Look at what I wrote in it during the past few days: https://github.com/Rudd-O/curvetls .

@Rudd-O
Rudd-O commented Sep 28, 2016 edited

Come to think of it, the same crypto primitives I am using in the program above (Go's implementation of NaCL secretbox) can be used to seal disk image contents in tamper-resistant containers. You really should check it out — it doesn't have the cache / timing leaks that AES has, it's Salsa and Poly, very good stuff that has a number of implementations and is not known to be weak.

Presumably, the key you pass to secretbox.Seal would be the output of scrypt's hash function.

Nice, eh?

@Rudd-O
Rudd-O commented Sep 28, 2016

I'm actually writing two demo programs to explain what I mean. Super simple, for you to read. Gimme 15 more minutes.

@Rudd-O
Rudd-O commented Sep 28, 2016 edited

There you go: brain dead simple:

https://github.com/Rudd-O/simple-nacl-crypto

The only remaining thing to do, is to write io.Reader and io.Writer that will "packetize" rows of zeroes (as sparse files are wont to contain) and package that data into the secret boxes. It's fairly easy to do, and the Go implementation of files allows seeking, thus it allows constructing sparse files on disk.

@Rudd-O
Rudd-O commented Sep 28, 2016

Great news:

Though I have to go to sleep and I still need to put a few finishing touches on it, the encryptor and decryptor programs have evolved to pack zeroes (512-byte blocks to be accurate) in a run-length format (that should not be vulnerable to malicious manipulation, because verifiable encryption goes around it).

A file 1 GB in size reduces itself to about 21 bytes. And it's those 21 bytes that get encrypted. No need to do gzip or anything of the sort. Of course, this packing format can in principle be piped to gzip for compression as well.

Naturally, the decoder will use disk seeks to skip writing zeroes as it decodes. This will give us sparse files on decryption for free.

I will finish the decoder for this packing format later today. Right now I must sleep.

@marmarek
Member

Come to think of it, the same crypto primitives I am using in the program above (Go's implementation of NaCL secretbox) can be used to seal disk image contents in tamper-resistant containers. You really should check it out — it doesn't have the cache / timing leaks that AES has, it's Salsa and Poly, very good stuff that has a number of implementations and is not known to be weak.

@Rudd-O please don't. Writing our own program to handle crypto is the last thing we want to do, somewhere near "inventing our own crypto algorithm". Actually this very ticket is result of "being smart" and using openssl enc/openssl dgst directly, instead of some higher layer application, designed by experienced cryptographer.

@Rudd-O
Rudd-O commented Sep 28, 2016 edited

@marmarek I'm not "writing my own crypto". I'm merely writing a program that wraps well-tested cryptography. NaCL's secretbox is that higher layer crypto (a layer above enc + dgst, with proper authentication and handling of streams), designed by experienced cryptographers. My program only uses it.

Anyway, you should know that the point of these programs isn't to be used as full-fledged backup programs — they are meant as demos within a memory-safe language of (a) crypto box (b) sparse file handling. Much like scrypt enc and scrypt dec are meant to be demos of the scrypt hashing algorithm, and they are not meant to be fully-fledged encryption programs. I'm not going to expand them into backup programs.

@marmarek
Member

@Rudd-O But you are inventing own file format. The other problem is introducing new language into code base. While I see why you propose Go, we should stick to those currently used (Python, C). Otherwise maintaining and auditing it would be a nightmare (good luck finding skilled developer fluent in Go, Python, C, Ocaml, Ruby and whatnot). Please don't go offtopic here on advocating why Go is better (even when technically you may be right) - use mailing list for this.

@Rudd-O
Rudd-O commented Sep 28, 2016 edited

I understand. There seems to be a misunderstanding here.

I'm not saying "use this code as part of the backup for Qubes".

I'm saying that you should look at this a demo — demo, key word, I used it repeatedly — of how:

  • a crypto solution (not a primitive) like NaCL secretbox can be used to safely store files
  • a format for a backup can be made such that it is secure and not tamperable
  • a file can be packed as sparse almost effortlessly

Anyone is 100% free to look at how the code solves these three problems, and write a Python implementation of the same concepts. That can then be used in Qubes.

Note that you are free to use the code directly, if you later change your mind.

Edit: the demo project is done. It encrypts and decrypts files, packing and unpacking sparse files. The code won't let tampered files screw with the computer — security-critical properties are validated before data is parsed. I'm happy with how it turned out. See for yourself:

[user@machine simple-nacl-crypto]$ dd if=/dev/urandom bs=30M of=original count=1 seek=21+0 records in
1+0 records out
31457280 bytes (31 MB, 30 MiB) copied, 2.19849 s, 14.3 MB/s
[user@ring2-projects simple-nacl-crypto]$ encbufsize=1048576 ; decbufsize=1048576 ; make && (bin/nacl-crypt -s -b $encbufsize enc original encrypted abc ; (echo ---------- ; bin/nacl-crypt -b $decbufsize dec encrypted new abc ) ; echo ;  (md5sum original new ; ls -la original ; ls -la  encrypted ; ls -la new ; du original ; du encrypted ; du new ))
GOPATH=/home/user/Projects/simple-nacl-crypto go install github.com/Rudd-O/simple-nacl-crypto/cmd/`echo bin/nacl-crypt | sed 's|bin/||'`
----------

ae76a32ca5b75f4c4f276a2f08750bc7  original
ae76a32ca5b75f4c4f276a2f08750bc7  new
-rw-rw-r-- 1 user user 94371840 Sep 29 02:02 original
-rw-rw-r-- 1 user user 31458320 Sep 29 02:02 encrypted
-rw-rw-r-- 1 user user 94371840 Sep 29 02:02 new
30720   original
30724   encrypted
30724   new
@marmarek marmarek added a commit to QubesOS/qubes-linux-scrypt that referenced this issue Oct 6, 2016
@marmarek marmarek Apply patch (and it dependents) to add -P option 71dfce0
@marmarek
Member

Here is draft of emergency backup restore v4, which is informal backup format specification. It uses tar for storing sparse files, but encrypted and integrity protected using scrypt utility.

@Rudd-O
Rudd-O commented Oct 11, 2016

I dislike the use of tar, to be honest. tar takes forever when reading a file that has sparse sectors, because it has to read the entire file before actually beginning to spit out the data to the calling process. A utility that was written for the purpose, which doesn't have this problem, and made available on Github for emergency restore purposes, should be much better.

@marmarek
Member

I see. As for reading entire file - bsdtar does it better (for the same file format). But it works only for sparse files, not LVM. Not sure how it works on btrfs. In fact I think it is impossible to effectively get LVM thin volume content (without reading it all). But if possible, it can be implemented in our tool.
And also - as tar tool can't get block device content, I've written simply python script for it: https://github.com/marmarek/qubes-core-admin/blob/core3-backup/qubes/tarwriter.py
Extraction (either normal, or emergency) is still handled by standard tool. As discussed in this year+ long thread, the best compromise for encryption + integrity protection is to use scrypt tool, I don't want to reinvent anything here.

@Rudd-O
Rudd-O commented Oct 12, 2016

On 10/11/2016 10:32 PM, Marek Marczykowski-Górecki wrote:

I see. As for reading entire file - |bsdtar| does it better (for the
same file format). But it works only for sparse /files/, not LVM. Not
sure how it works on btrfs. In fact I think it is impossible to
effectively get LVM thin volume content (without reading it all). But
if possible, it can be implemented in our tool.
And also - as |tar| /tool/ can't get block device content, I've
written simply python script for it:
https://github.com/marmarek/qubes-core-admin/blob/core3-backup/qubes/tarwriter.py
Extraction (either normal, or emergency) is still handled by standard
tool. As discussed in this year+ long thread, the best compromise for
encryption + integrity protection is to use |scrypt| tool, I don't
want to reinvent anything here.

Qubes OS should also not be using LVM AT ALL. There are no data
integrity guarantees with it.

If Qubes OS used btrfs, for example, efficient clones of VMs would be
trivial, cp --reflink would work, and FIEMAP (discovery of holes in VM
images) would also be implementable.

tar still sucks. The file needs to be read whole because the format
requires it upfront.

Scrypt is fine. It's effectively the same thing I am doing with the
demo program that I posted above, except it doesn't handle sparse files.

Rudd-O
http://rudd-o.com/
@marmarek
Member

On Tue, Oct 11, 2016 at 05:25:43PM -0700, Rudd-O wrote:

Qubes OS should also not be using LVM AT ALL. There are no data
integrity guarantees with it.

What data integrity features would you expect? It's just about storage
of VM images - something that dom0 should not look into at all.

If Qubes OS used btrfs, for example, efficient clones of VMs would be
trivial, cp --reflink would work,

And very similar with LVM thin (lvcreate -s).

and FIEMAP (discovery of holes in VM
images) would also be implementable.

Indeed this would be better on btrfs.

tar still sucks. The file needs to be read whole because the format
requires it upfront.

If you have FIEMAP-compatible storage, it isn't a problem, as you get
the file map at the start. But for incompatible storages, you're right.

Scrypt is fine. It's effectively the same thing I am doing with the
demo program that I posted above, except it doesn't handle sparse files.

But also it has been written some time ago and is properly maintained.
We don't want to maintain (one more) our own tool.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

@Rudd-O
Rudd-O commented Oct 12, 2016

What data integrity features would you expect? It's just about storage of VM images - something that dom0 should not look into at all.

That's not a backup thing I was referring to. I was referring to guarantees that the live data stored in dom0 is never corrupted. LVM cannot guarantee that. Qubes OS should use either ZFS or btrfs.

And very similar with LVM thin (lvcreate -s).

Reflink does not work with LVM.

But also it has been written some time ago and is properly maintained. We don't want to maintain (one more) our own tool.

Yes, I understand, I am not suggesting the adoption of my tool.

@rustybird

@Rudd-O:

I dislike the use of tar, to be honest. tar takes forever when reading a file that has sparse sectors, because it has to read the entire file before actually beginning to spit out the data to the calling process.

You can massively speed up qvm-backup by installing tar >= 1.29, which uses SEEK_DATA/SEEK_HOLE instead. (I built mine from source, but it looks like the f25 package works in f23 too.)

@Rudd-O
Rudd-O commented Oct 12, 2016

I would hope SEEK_DATA / SEEK_HOLE is implemented in ZFS. :-)

@jpouellet
Contributor
jpouellet commented Oct 16, 2016 edited

#971 (comment) re tarsnap:

Nonfree means things built outside Amerika because of intellectual monopoly bullshit.

To the extent that I know, that may just be a miscategorization, as tarsnap is bona fide open source which you can download from the tarsnap site.

is not correct. See https://www.tarsnap.com/open-source.html

The tarsnap source itself is not open source, only some of its components (dependent libraries) are, including the scrypt implementation. This is a somewhat arbitrary distinction, as Colin Percival is also the author of the bsd-licensed dependencies.

It appears as though a line was apparently drawn somewhere between "this will benefit the community" and "this is how i pay the bills, and my business model requires the tarsnap client i've written to use my server-side code and not some arbitrary possibly-self-hosted cheaper (free?) one".

The above is in no way meant to be an argument against using scrypt, only a clarification about the classification of tarsnap as non-free.

@Rudd-O
Rudd-O commented Oct 17, 2016

Understood.

@marmarek marmarek added a commit to marmarek/qubes-core-admin that referenced this issue Oct 28, 2016
@marmarek marmarek backup: use 'scrypt' tool for backup encryption and integrity protection
`openssl dgst` and `openssl enc` used previously poorly handle key
stretching - in case of `openssl enc` encryption key is derived using
single MD5 iteration, without even any salt. This hardly prevent
brute force or even rainbow tables attacks. To make things worse, the
same key is used for encryption and integrity protection which ease
brute force even further.
All this is still about brute force attacks, so when using long, high
entropy passphrase, it should be still relatively safe. But lets do
better.
According to discussion in QubesOS/qubes-issues#971, scrypt algorithm is
a good choice for key stretching (it isn't the best of all existing, but
a good one and widely adopted). At the same time, lets switch away from
`openssl` tool, as it is very limited and apparently not designed for
production use. Use `scrypt` tool, which is very simple and does exactly
what we need - encrypt the data and integrity protect it. Its archive
format have own (simple) header with data required by the `scrypt`
algorithm, including salt. Internally data is encrypted with AES256-CTR
and integrity protected with HMAC-SHA256. For details see:
https://github.com/tarsnap/scrypt/blob/master/FORMAT

This means change of backup format. Mainly:

1. HMAC is stored in scrypt header, so don't use separate file for it.
Instead have data in files with `.enc` extension.
2. For compatibility leave `backup-header` and `backup-header.hmac`. But
`backup-header.hmac` is really scrypt-encrypted version of `backup-header`.
3. For each file, prepend its identifier to the passphrase, to
authenticate filename itself too. Having this we can guard against
reordering archive files within a single backup and across backups. This
identifier is built as:

        backup ID (from backup-header)!filename!

For backup-header itself, there is no backup ID (just 'backup-header!').

Fixes QubesOS/qubes-issues#971
418d749
@marmarek marmarek added a commit to marmarek/qubes-core-admin that referenced this issue Oct 28, 2016
@marmarek marmarek backup: add 'backup_id' to integrity protection
This prevent switching parts of backup of the same VM between different
backups made by the same user (or actually: with the same passphrase).

QubesOS/qubes-issues#971
4ad15c0
@marmarek marmarek added a commit to marmarek/qubes-core-admin that referenced this issue Oct 28, 2016
@marmarek marmarek backup: verify if archive chunks are not reordered
Now, when file name is also integrity protected (prefixed to the
passphrase), we can make sure that input files are given in the same
order. And are parts of the same VM.

QubesOS/qubes-issues#971
51b6620
@marmarek marmarek added a commit to marmarek/qubes-core-admin that referenced this issue Oct 28, 2016
@marmarek marmarek backup: mark 'encryption' option as deprecated - all backups are encr… 49e718c
@marmarek marmarek referenced this issue in QubesOS/qubes-core-admin Oct 28, 2016
Merged

Core3 backup2 #64

@andrewdavidwong
Member

@marmarek: What's the easiest way for (non-programmer) users to store a copy of tarsnap along with their backups such that it can be used to restore on the largest variety of platforms (i.e., the "old Ubuntu" test discussed in #971 (comment) and previously)?

For example, is it possible to store a few different signed scrypt packages (.deb, .rpm, etc.) along with my backups, then, in an emergency situation with no internet access, install scrypt using one of the packages and use it to decrypt my data?

@marmarek
Member

I've already included instruction from where download it: https://www.qubes-os.org/doc/backup-emergency-restore-v4/

@andrewdavidwong
Member

My question was prompted by reading exactly that page. You link to this page, which offers the source code as a tarball and links to several libraries and alternate implementations. It's still not clear what the best way is for non-programmer users to store a copy of the tool along with their backups.

@marmarek
Member

It is possible to build a statically linked scrypt binary, by calling configure this way:

CFLAGS=-static LIBS='-lz -ldl' ./configure

To do this, you need some build environment (compiler, headers etc) installed. And in case of Fedora, also: glibc-static, zlib-static, openssl-static.
Such binary should run on any system as it doesn't depend on external libraries.

@andrewdavidwong
Member

Thank you! Would it be worthwhile for us to build this, then offer the binary (signed by an appropriate Qubes signing key) to users as an optional convenience? I fear that many users will not be able to figure this part out for themselves, and therefore be forced to risk having inaccessible backups.

@marmarek
Member

Good idea, will add to the package and update instruction.

@Rudd-O
Rudd-O commented Oct 31, 2016

Please read this whitepaper for Qubes OS backup next version (3):

http://www.duplicati.com/assets/Block-basedstorageformat.pdf

This is what Qubes backup ought to have been all along.

Enjoy. It's a really great read.

@jpouellet
Contributor
jpouellet commented Oct 31, 2016 edited

According to that paper, duplicati considers neither confidentiality nor integrity.

Statements from that paper like:

"We chose the SHA­256 hash as that seems to be a safe bet, although the cryptographic properties of SHA are not required."

and

"If a required block is not found after scanning the index files, block files are downloaded at
random, until all known blocks are found. This enables a restore, even in cases without any index files."

are rather concerning when considering an adversary who maliciously modifies your backups.

I like the idea of a simple incremental backup solution which treats the actual blocks as completely opaque (the content-parsing to determine optimal chunking boundaries of other backup solutions sounds like an interesting attack surface to me), but IMO solution proposed in the paper used alone appears dangerously insufficient.

@marmarek
Member

@Rudd-O indeed interesting, but offtopic here. It says nothing about encryption and integrity protection which is main issue discussed here. It's mostly on topic of #858
EDIT: @jpouellet was faster about details, so not repeating it here.

@Rudd-O
Rudd-O commented Oct 31, 2016

On 10/31/2016 09:43 PM, Jean-Philippe Ouellet wrote:

According to that paper, duplicati considers neither confidentiality
nor integrity.

Statements from that paper like:

"We chose the SHA­256 hash as that seems to be a safe bet,
although the cryptographic properties of SHA are not required."

and

This is an unfair criticism.

Duplicati lets you choose plain AES like Qubes backup today, or GPG if
you dislike the choice of SHA256. Furthermore, the software is open
source — LGPL 2.1 to be exact — so if you want to plug your own
algorithm, you're free to do so.

"If a required block is not found after scanning the index files,
block files are downloaded at
random, until all known blocks are found. This enables a restore,
even in cases without any index files."

are rather concerning when considering an adversary who maliciously
modifies your backups.

Duh, if an adversary modifies your files, it does not matter whether you
have to download a thousand 1 MB blocks or 1 GB file, you still can't
restore! This is a 100% frivolous criticism, my man.

I like the idea of a simple incremental backup solution which treats
the actual blocks as completely opaque

That is precisely what Duplicati is — the blocks are opaque to the
storage medium — except it isn't in the full+incremental scheme — it's
fully content-addressable, which is superior in many ways.

(the content-parsing to determine optimal chunking boundaries of other
backup solutions sounds like an interesting attack surface to me), but
to me this solution alone is dangerously insufficient.

I do not believe you have formed that opinion based on valid facts.

Rudd-O
http://rudd-o.com/
@Rudd-O
Rudd-O commented Oct 31, 2016

On 10/31/2016 09:46 PM, Marek Marczykowski-Górecki wrote:

@Rudd-O https://github.com/Rudd-O indeed interesting, but offtopic
here. It says nothing about encryption and integrity protection which
is main issue discussed here. It's mostly on topic of #858
#858
EDIT: @jpouellet https://github.com/jpouellet was faster about
details, so not repeating it here.

I recognize it's off topic, as the paper does not speak of integrity
protection. The integrity protection of the implementation is provided
by the choice of encryption and hashing algorithm, which is not the
topic of the paper (the paper focuses on explaining the unencrypted data
structures, which I thought would be a good read for you).

I will be trying to implement something like this for Qubes OS, since I
need this storage system for a variety of reasons. I will report back
with progress.

Rudd-O
http://rudd-o.com/
@marmarek
Member

I don't think deduplication (especially across VMs) is a good idea for secure backup solution at all. There are multiple things that can go wrong. For example - if attacker control one of the VMs, can store some data there, and later observing backup size guess if the data is present also in some other VM, thus leaking information about content of another VM. Probably not feasible for leaking private keys (something not known to the attacker), but very real for finding who have some known data (like breaking anonymity of some Tor user).
Also, today SHA256 is considered safe, but this format rely on it to have never, ever any collision, not even unintentional one. This drawback is even mentioned in that paper.

And this is just what I can think of in 5 minutes. There are probably more. So, any solution must not share anything across VMs.

@jpouellet
Contributor
jpouellet commented Nov 1, 2016 edited

On Mon, Oct 31, 2016 at 6:40 PM, Rudd-O notifications@github.com wrote:

On 10/31/2016 09:43 PM, Jean-Philippe Ouellet wrote:

"If a required block is not found after scanning the index files,
block files are downloaded at
random, until all known blocks are found. This enables a restore,
even in cases without any index files."

are rather concerning when considering an adversary who maliciously
modifies your backups.

Duh, if an adversary modifies your files, it does not matter whether you
have to download a thousand 1 MB blocks or 1 GB file, you still can't
restore! This is a 100% frivolous criticism, my man.

Sure, they could harm availability, but if properly authenticated, the attacker's capabilities end there.

But, if you don't have a trusted way to identify and authenticate the root of your merkle-tree index thing, (which sounds to be the case with this system as described when restoring without an already trusted index) then your adversary may also gain arbitrary write on restore, and that is plenty of leverage for further attacks.

I like the idea of a simple incremental backup solution which treats
the actual blocks as completely opaque

That is precisely what Duplicati is — the blocks are opaque to the
storage medium — except it isn't in the full+incremental scheme — it's
fully content-addressable, which is superior in many ways.

Right. I was saying that's the main concept I like from it, as opposed to other systems.

(the content-parsing to determine optimal chunking boundaries of other
backup solutions sounds like an interesting attack surface to me), but
to me this solution alone is dangerously insufficient.

I do not believe you have formed that opinion based on valid facts.

Sorry to disappoint.

It makes sense from the perspective of one optimizing for storage above all else to want to split streams on intelligent boundaries such that you can re-use as many already-stored content-addressed chunks as possible.

Unfortunately, determining the most intelligent boundary is seriously non-trivial, and as one tries to achieve better results by incorporating more format-specific knowledge, complexity (and with it opportunity for vulnerabilities) necessarily explodes.

Consider libchop's anchor-based chopper based on this paper.

Upon a cursory examination, the implementation appears to include a partial domain-specific JIT compiler (!!!) just to choose where to split data!! I'd much prefer something which treated things as completely opaque blobs.

Sure, one could argue it could be effectively sandboxed and pass back only a list of indeces to chop at, but still... for stuff likely running in Dom0, definitely do not want!

And for those who say "yeah, but... how likely is it for it to actually have a vuln in that... It can't be that complex... it's just some minor string manipulation", just look at the libarchive exploit that recently affected FreeBSD's update system.

@Rudd-O
Rudd-O commented Nov 2, 2016

On 10/31/2016 11:09 PM, Marek Marczykowski-Górecki wrote:

I don't think deduplication (especially across VMs) is a good idea for
secure backup solution at all. There are multiple things that can go
wrong. For example - if attacker control one of the VMs, can store
some data there, and later observing backup size guess if the data is
present also in some other VM, thus leaking information about content
of another VM.

This is an objection that is incredibly far-fetched. First of all, you
are suggesting that an attacker controls a VM that is backed up, AND
also controls the backup data. That's already incredibly improbable.
Second, you're also suggesting that the attacker writes some that
happens to be deduplicated, which means the attacker ALREADY HAS the
data he was trying to discover BY DEFINITION.

I do not think this is a valid objection.

But, if you fear this extremely improbable thing, you can just switch
off deduplication. Presto, problem gone.

Below I explain how that can be accomplished.

Probably not feasible for leaking private keys (something not known to
the attacker), but very real for finding who have some known data
(like breaking anonymity of some Tor user).
Also, today SHA256 is considered safe, but this format rely on it to
have /never, ever/ any collision, not even unintentional one. This
drawback is even mentioned in that paper.

I'm sure new algorithms can be added to the code. It's open source.

And this is just what I can think of in 5 minutes. There are probably
more. So, any solution must not share anything across VMs.

Just turn dedup off!

It's super simple to do that.

Here's just one way to do it:

During backup, at the point where backup is hashing the data to see if
there are any duplicates, simply hash the data with the path name of the
file being stored.

Bam, dedup between VMs is now off.

Best thing: all of this is implementable because the code is open source.

Another way to turn dedup off: always do a full backup. That way this
mysterious attacker you posit cannot compare before+after backup sizes.

Bam, problem solved!

Fundamentally, some people care about the remote scenarios you propose,
and others do not. To cater for them, there should be a setting that
allows them to toggle the behavior you dislike.

Rudd-O
http://rudd-o.com/
@Rudd-O
Rudd-O commented Nov 2, 2016

On 11/01/2016 12:19 AM, Jean-Philippe Ouellet wrote:

Sure, they could harm availability, but if properly authenticated, the
attacker's capabilities end there.

But, if you don't have a trusted way to identify and authenticate the
root of your merkle-tree index thing, (which sounds to be the case
with this system as described when restoring without an already
trusted index)

I don't think that's the case. That merkle root will not decrypt if you
give it the wrong password.

Feel free to correct me with the relevant technical details.

Rudd-O
http://rudd-o.com/
@jpouellet
Contributor

On Wed, Nov 2, 2016 at 6:42 AM, Rudd-O notifications@github.com wrote:

On 10/31/2016 11:09 PM, Marek Marczykowski-Górecki wrote:

I don't think deduplication (especially across VMs) is a good idea for
secure backup solution at all.

This is an objection that is incredibly far-fetched. First of all, you
are suggesting that an attacker controls a VM that is backed up, AND
also controls the backup data. That's already incredibly improbable.
Second, you're also suggesting that the attacker writes some that
happens to be deduplicated, which means the attacker ALREADY HAS the
data he was trying to discover BY DEFINITION.

Yes, the adversary knows what the data is, but does not know you
have it (and you wish to keep that a secret).

I do not think this is a valid objection.

I think this is a very valid objection.

Consider the case where you are a whistle-blower trying to expose some
corruption or whatever. You have a non-networked "secrets" vm you use
to analyze some cache of documents exposing said corruption that you
stole from some database. The adversary suspects you may possess said
documents, but wants proof before making you disappear.

By injecting a single block of said data into the disk a different
(less-trusted) vm (plenty of ways to do that... via browser cache for
example) and observing deduplication, then the adversary can confirm
the presence of data you wished to keep the existence of a secret.

But, if you fear this extremely improbable thing, you can just switch
off deduplication. Presto, problem gone.

Fundamentally, some people care about the remote scenarios you propose,
and others do not. To cater for them, there should be a setting that
allows them to toggle the behavior you dislike.

I believe that is not the correct philosophy for an open source
project aiming to provide improved security and privacy for an
audience including less-technical users.

You expect some journalist wanting to hide the fact that they have
certain documents to be aware of duplication-based
content-confirmation attacks, be aware that qubes' backup system is
vulnerable by default, be aware that there is an option to turn that
off, and actually do so before it matters? I do not think this is a
reasonable model for the intended users.

Personally, I'm a great fan of OpenBSD-style decision rationale, where
choices are made to be secure by default and have few knobs to turn.
Consistent application of that philosophy protects users.

@jpouellet
Contributor

On Wed, Nov 2, 2016 at 6:43 AM, Rudd-O notifications@github.com wrote:

I don't think that's the case. That merkle root will not decrypt if you
give it the wrong password.

Feel free to correct me with the relevant technical details.

But, if you don't have a trusted way to identify and authenticate the
root of your merkle-tree index thing, (which sounds to be the case

I was assuming things were implemented as described in the paper, with
no additional integrity protection mechanisms wrapping the indexes.

If this is not the case, then I am mistaken.

Content-confirmation attacks associated with
convergent-encryption-based deduplication schemes like this one still
remain problematic though.

@Rudd-O
Rudd-O commented Nov 3, 2016

On 11/02/2016 02:07 PM, Jean-Philippe Ouellet wrote:

Fundamentally, some people care about the remote scenarios you propose,
and others do not. To cater for them, there should be a setting that
allows them to toggle the behavior you dislike.

I believe that is not the correct philosophy for an open source
project aiming to provide improved security and privacy for an
audience including less-technical users.

What's wrong with the setting, if the setting is toggled to the secure
value by default?

Settings are a perfectly valid way of changing how software behaves.

Rudd-O
http://rudd-o.com/
@defuse
defuse commented Nov 3, 2016

@Rudd-O: I was going to reply here about why settings are bad in general, but I felt it was off-topic, so I moved it into a blog post: https://bqp.io/on-software-settings.html (here it's the "second problem" that applies most).

@jpouellet
Contributor
jpouellet commented Nov 3, 2016 edited

Another post that explains things more eloquently than I could: http://www.daemonology.net/blog/2009-09-04-complexity-is-insecurity.html

That post also happens to be from the author of scrypt :)

EDIT: This also applies to my argument against trying to chunk data at intelligent boundaries rather than fixed-size blocks. While the former clearly results in superior deduplication, the latter is more secure.

@defuse
defuse commented Nov 14, 2016

Heads up that I added a feature to crack Qubes v3.1 (and think, but have not verified v3.2) backup encryption (default settings) to CrackStation: https://bqp.io/qubes-backup-encryption-added-to-crackstation.html I hope by doing this I'll attract some more cryptographers in here making sure the next version is rock solid.

In case anyone wants a short-term workaround, what I've done personally is to encrypt my backup disk with LUKS, and then store my Qubes-encrypted backups on there.

@andrewdavidwong
Member
andrewdavidwong commented Dec 11, 2016 edited

Heads up that I added a feature to crack Qubes v3.1 (and think, but have not verified v3.2) backup encryption (default settings) to CrackStation: https://bqp.io/qubes-backup-encryption-added-to-crackstation.html

Yes, this applies to 3.2 also.

I hope by doing this I'll attract some more cryptographers in here making sure the next version is rock solid.

Thanks, @defuse.

Related question: Theoretically, what's the fastest way to brute force a Qubes backup under the current scheme? There are at least two ways to go, right?

  1. Keep trying 32-digit hexadecimal strings (i.e., possible outputs of md5(passphrase)) as aes-256-cbc keys until one works. (Assuming that the passphrase has >= 128-bits of entropy.)
  2. Keep computing openssl dgst -sha512 -hmac '<passphrase>' backup-header until one matches backup-header.hmac.

Is 1 theoretically faster because it entails a smaller keyspace? Or is there another, faster way?

The reason I ask is because this would seem to affect estimates of how strong/weak the current system is (i.e., how long it would take to brute force given different values of passphrase entropy).

@jpouellet
Contributor
jpouellet commented Dec 11, 2016 edited
  1. Keep trying 32-digit hexadecimal strings (i.e., possible outputs of md5(passphrase)) as aes-256-cbc keys until one works. (Assuming that the passphrase has >= 128-bits of entropy.)

No, 128 bits is an unthinkably large state space to try to brute-force. The set of md5(likely_passphrases) is sparse within that space, so you would definitely want to start with likely passphrase candidates and pass those through md5 to generate candidate aes keys, not try to randomly guess a correct 128-bit value (assumed to be so incredibly unlikely that it is why many standards target the 128-bit effective security level).

The larger problem with this approach though is checking whether or not a resulting passphrase is correct. You then need to use the resulting key to try to decrypt the filesystem, and check if the plaintext looks like a filesystem. The HMAC-cracking approach has the benefit that it is immediately obvious whether or not the result is correct.

  1. Keep computing openssl dgst -sha512 -hmac '' backup-header until one matches backup-header.hmac.

Yes. This.

Is 1 theoretically faster because it entails a smaller keyspace? Or is there another, faster way?

No. The question is not about 128-bits vs 256-bits (both are assumed to be sufficient in crypto literature). The design flaw here is that is the HMAC and encryption key are derived from exactly the same input, and an HMAC construction allows fast (cheap to attack) checking of whether or not a key is correct (because it is designed for cheap message authentication using shared secrets), whereas a strong KDF is designed to be expensive-to-compute (and that expense does not need to be paid by an attacker who can bypass it by computing the HMAC for each candidate passphrase instead).

Disclaimer: I am not a cryptographer, I just like reading and thinking about applied crypto, and occasionally have to crack passwords

@jpouellet
Contributor

The reason I ask is because this would seem to affect estimates of how strong/weak the current system is (i.e., how long it would take to brute force given different values of passphrase entropy).

In its current (3.2) state, we essentially have the cheapest KDF possible, so I think the only safe way to use it is to have a passphrase that is already sufficiently high entropy for use directly as a key.

When I used qvm-backup (to transfer VMs from old laptop to new laptop), I used a key from dd if=/dev/random, encrypted the result asymmetrically to a key generated on the destination machine, signed that with a key generated on the origin machine, performed the backup & transfer, then decrypted the used symmetric key on the destination machine.

I think such an asymmetric scheme is really the way to go for various reasons, but I've yet to put my arguments together and formally make a case for it. tl;dr - allows to do away with passphrases and have a safely-stored offline backup-recovery key (which is not needed at backup-creation time), and a backup-origin signing key. This would enable the creation of encrypted and authenticated backups in a hostile environment where entering a backup passphrase would be problematic.

@jpouellet
Contributor

tl;dr - allows to do away with passphrases and have a safely-stored offline backup-recovery key

That recovery key could (should) of course be protected with a passphrase itself, but it need not be used (or even known) at backup-creation time.

@andrewdavidwong
Member
  1. Keep trying 32-digit hexadecimal strings (i.e., possible outputs of md5(passphrase)) as aes-256-cbc keys until one works. (Assuming that the passphrase has >= 128-bits of entropy.)

No, 128 bits is an unthinkably large state space to try to brute-force. The set of md5(likely_passphrases) is sparse within that space, so you would definitely want to start with likely passphrase candidates and pass those through md5 to generate candidate aes keys, not try to randomly guess a correct 128-bit value (assumed to be so incredibly unlikely that it is why many standards target the 128-bit effective security level).

Of course. I didn't say anything about randomly guessing 128-bit values. It goes without saying that you would prioritize the possible outputs of likely passphrases before unlikely ones. (This is, of course, an oversimplification of what sophisticated password crackers actually do.)

The larger problem with this approach though is checking whether or not a resulting passphrase is correct. You then need to use the resulting key to try to decrypt the filesystem, and check if the plaintext looks like a filesystem. The HMAC-cracking approach has the benefit that it is immediately obvious whether or not the result is correct.

That's one thing I was wondering about. I suppose it depends on how much additional time it takes to attempt to decrypt the file with each key.

Is 1 theoretically faster because it entails a smaller keyspace? Or is there another, faster way?

No. The question is not about 128-bits vs 256-bits (both are assumed to be sufficient in crypto literature).

You're missing the point. The question is which one is theoretically faster. The question is not whether both would take a sufficiently long time.

The design flaw here is that is the HMAC and encryption key are derived from exactly the same input, and an HMAC construction allows fast (cheap to attack) checking of whether or not a key is correct (because it is designed for cheap message authentication using shared secrets), whereas a strong KDF is designed to be expensive-to-compute (and that expense does not need to be paid by an attacker who can bypass it by computing the HMAC for each candidate passphrase instead).

I know. You just repeated the issue that I originally reported.

@jpouellet
Contributor

I know. You just repeated the issue that I originally reported.

Err, oops. Indeed. This thread has been so long I'd forgotten you were the OP. :) My apologies if you feel I've insulted your intelligence by my explanation.

@defuse
defuse commented Dec 11, 2016

Related question: Theoretically, what's the fastest way to brute force a Qubes backup under the current scheme? There are at least two ways to go, right?

Keep trying 32-digit hexadecimal strings (i.e., possible outputs of md5(passphrase)) as aes-256-cbc keys until one works. (Assuming that the passphrase has >= 128-bits of entropy.)
Keep computing openssl dgst -sha512 -hmac '<passphrase>' backup-header until one matches backup-header.hmac.

These are about equivalent, if you ignore the difference in time it takes to compute MD5 then AES vs. HMAC-SHA512.

The reason I ask is because this would seem to affect estimates of how strong/weak the current system is (i.e., how long it would take to brute force given different values of passphrase entropy).

The worst problem with the current system is that the password isn't salted. Without a salt, the attacker can perform their brute-force attack once, saving all the (HMAC, password) pairs they compute into a lookup table, then quickly crack any HMAC in that list with a fast (<1 second) lookup. That's a lot cheaper than having to re-run the brute-force attack for each hash (which using salt ensures). Fixing this changes the asymptotics of the best attack -- O(N) precomputation then O(1) for each hash to crack into no precomputation and O(N) for each hash to crack.

The second problem is that we want to use some sort of slow key derivation function on the password, like Argon2, scrypt, bcrypt, etc, to increase the cost of attacks further.

@marmarek
Member

See #971 (comment) I think it all have been covered already (mostly by using scrypt utility).

@marmarek
Member

One issue with new backup format is that each chunk (currently 100M) has key derived separately - and it takes time (by design). I think the easiest think to do is to increase chunk size to for example 1GB, to minimize the impact.
Other solution would be to derive key once use it for all the chunks, but it will require more work because:

  • cannot use scrypt tool directly - for example go back to openssl enc/openssl dgst - this time with explicit keys (derived using scrypt KDF) instead of very weak built-in KDF; or find some other tool
  • needs some other way to bind encrypted chunk to its name (currently done by prefixing chunk name to the passphrase feed into scrypt KDF)
    And this will also make the manual restore harder.
    Looking at this long discussion, I don't want to go through it all along again...
@andrewdavidwong
Member

One issue with new backup format is that each chunk (currently 100M) has key derived separately - and it takes time (by design). I think the easiest think to do is to increase chunk size to for example 1GB, to minimize the impact.

Is this "just" a performance issue, or is it also a security issue? How much of a performance impact are we talking about? Why is the current chunk size 100M in the first place?

Other solution would be to derive key once use it for all the chunks, but it will require more work because: [...]

Both of those options sound undesirable. Are there any downsides to just using a 1GB chunk size?

Looking at this long discussion, I don't want to go through it all along again...

Not sure what you mean here. Why would we have to go through it all again?

@marmarek
Member
@andrewdavidwong
Member

In that case, I agree with you. Better to keep it simple and just increase the chunk size to 1GB (or some similarly suitable size).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment