Can I use AEAD without the integirty tag? Or why AEAD is not supported in command enc

[bowensd]

I run openssl -enc -aes-128-gcm and shows that AEAD ciphers not supported
Why AEAD is not supported?Is there any consideration in designing the command line tools? Is there any other command to test or run AEAD ciphers in command line?

[slontis]

openssl-enc.pod explains why this is not supported from the commandline..

i.e:
'This command does not support authenticated encryption modes
like CCM and GCM, and will not support such modes in the future.
This is due to having to begin streaming output (e.g., to standard output
when B<-out> is not used) before the authentication tag could be validated.
When this command is used in a pipeline, the receiving end will not be
able to roll back upon authentication failure. The AEAD modes currently in
common use also suffer from catastrophic failure of confidentiality and/or
integrity upon reuse of key/iv/nonce, and since B places the
entire burden of key/iv/nonce management upon the user, the risk of
exposing AEAD modes is too great to allow. These key/iv/nonce
management issues also affect other modes currently exposed in this command,
but the failure modes are less extreme in these cases, and the
functionality cannot be removed with a stable release branch.
For bulk encryption of data, whether using authenticated encryption
modes or other modes, L<openssl-cms(1)> is recommended, as it provides a
standard data format and performs the needed key/iv/nonce management.'

If you want a see a test that uses AEAD run

./test/acvp_test -test 2
./test/acvp_test -test 3

There are 2 tests that use fixed tables from acvp_test.inc.

[paulidale]

As @slontis wrote. To support AEAD ciphers in the enc command, all of the output needs to be buffered before any is emitted. Letting something downstream use the data before it has been authenticated is a critical vulnerability.

[jmdesp]

It's a bit insane to allow things that are a lot more dangerous (unauthenticated broken CBC encryption, and all the other modes where you have no idea of the integrity of the data at all) but block this.
The command lines tools in Openssl are in many different ways extremly efficient to shoot yourself in the foot if you don't know what you're doing.
This could be restricted instead to allow only output to a file, and delete this file if the authentication fails.

[markg85]

What is the adviced way to do AES-256-GCM encryption with command line tooling like openssl enc? I can vaguely find "openssl cms" but that tells me nothing and examples for that seems to be hidden quite well (can't quite find it).

[markg85]

邮件已收到，感谢来信。

Would you mind repeating that in english?

[slontis]

Please read the above comments. You need to write code, and you need to understand the restrictions of GCM before using it.

[markg85]

But why?

Having it in a command line tool helps a lot in lowering an already very high bar and makes it scriptable.

If I recall correctly, libressl does allow this in the cli tool. But mixing the 2 *ssl projects is super confusing so I'd rather not.

Requiring code only ups the threshold of attempting to use it. It still doesn't guarantee proper usage. Heck, I'd even argue that encrypting in code is far more likely to be done wrongly then doing it in cli. See, with a cli you - as the authors of said cli - have a way to guide the user to do the right thing. You lose that with code.

[paulidale]

It is impossible to know if the GCM decryption is correct until the very end. At that point it's not possible to tell the receiver to ignore everything processed thus far. It's possible to get a few cases correct but not all. What happens if the output is a pipe? Or if the output overwrites an existing file? Moreover, it is trivial to flip bits in the plaintext with GCM mode. If this isn't discovered until the very end after the payload has been mostly processed, it is a critical vulnerability.

I'll also note that GCM was designed for data in transit not data at rest. Using it for data at rest is a misuse IMO.

[markg85]

In my case i want to encrypt video data which is the exact situation where AES-GCM shines. It surprises me that the openssl opinion is so... how to put it politely.. intentionally overly constrained. Sure, it's all possible but your current approach makes it a lot more complex then it needs to be.

Are you sure a reconsideration of this (with libressl in mind where this is possible) isn't in order?

[paulidale]

Video data is arguably one of the worst cases, the data will almost certainly be used before the validity is checked after everything has been decrypted. This is a massive shortcoming in GCM -- you must decrypt everything before even a single byte can be consumed. Yes, I know it's really fast but it's completely insecure when used like this. I've been asked about using GCM sans integrity check sufficiently often that I came up with Pauli's Mutual Fund.

Your money is safe. All your data are encrypted using the latest government approved methods.

$ echo "Pauli is owed $    1000 dollars" | openssl enc -k fnord -e -aes-128-ctr -pbkdf2 >ciphertext
$ xxd ciphertext >data
$ cat data
00000000: 5361 6c74 6564 5f5f 0766 9237 007c 4b52  Salted__.f.7.|KR
00000010: ac81 4d00 79b2 1691 1f37 e1b8 fb26 203f  ..M.y....7...& ?
00000020: 5535 d358 be98 b049 9a93 3c02 ff51 20fb  U5.X...I..<..Q .
$ vi data
$ cat data
00000000: 5361 6c74 6564 5f5f 0766 9237 007c 4b52  Salted__.f.7.|KR
00000010: ac81 4d00 79b2 1691 1f37 e1b8 fb26 203f  ..M.y....7...& ?
00000020: 4c25 c359 be98 b049 9a93 3c02 ff51 20fb  U5.X...I..<..Q .
$ xxd -r data >ciphertext
$ openssl enc -k fnord -d -aes-128-ctr -pbkdf2
Pauli is owed $ 9000000 dollars

If you run the same commands, you will end up with different ciphertext. Don't worry, flip the same six bits as I did in the first four bytes of the final row of data. In my example these were originally 5535 d358 and I changed them to 4c25 c359.

This example is for CTR mode but GCM is identical in operation.

Hence, I doubt that the enc command will get AEAD encryption -- there are too many tricky cases that would need to be covered. However this question comes up reasonably often and the technical committee is aware of the desire for it. It is possible that a new command could be introduced to support AEAD modes.

I just had a look at the LibreSSL 3.5.3 source code, the enc command there knows absolutely nothing about AEAD ciphers -- it does the encryption and does nothing with the integrity tag. I.e. there is no integrity check. You'd be just as secure (i.e. completely insecure) and faster if you switched to CTR mode. I consider this misleading and and a major oversight from the OpenBSD folks who are usually very particular about forcing a secure outcome.

[markg85]

You are essentially saying that Netflix doesn't know what they are doing.
PrivateInternetAccess is wrong
The claims about it being faster then CTR are wrong.

What you find most when googling for AES gcm vs ctr is that GCM should be the superior one in every way. That a comparable security to GCM would be CTR+HMAC. The HMAC part is omitted in GCM because it's "build in".

Now i'm sure you are correct and with the best intentions! But what i find elsewhere does at the very least seem to contradict you.

What makes this security stuff so freaking complex is this very message. You can clearly see the "different thoughts the internet has" here. And all are equally passionate about their opinion.

I'm at a loss here as to what is secure now and what's not.

[t8m]

Security through crypto is an extremely complex topic and when reading the links above you're unfortunately overlooking the details. One of the most important of these details are that all the links above talk about using the AES-GCM as part of data-in-transit protection in well known and well secured protocols as TLS and SSH.

I can only repeat that openssl enc is a very simple tool that is not aimed at proper data-in-transit protection. It is a very much legacy tool that really should not be used for any serious purpose.

[markg85]

@t8m Thank you, that's the first thing that made sense to me.

So why is this not echoed anywhere else? You're the first one who clearly says that it's intended purpose is for in-transit data. To me that does sound like video files - and definitely video streaming - are perfect candidates. Which again totally contradicts what @paulidale said. Unless there was a hidden assumption there that i didn't catch?

This post for example bashes hard on AES-GCM but still claims it's far better then the rest. Quote:

AES-GCM is still miles above what most developers reach for when they want to encrypt (e.g. ECB mode or CBC mode).

So something what you folks say, despite probably being with the best intention and that's very well appreciated, just doesn't match with what I find on the web.

While the example from @paulidale was well appreciated and helps a lot in understanding things. I question it to be the case with GCM. The case he explained is exactly what HMAC should fix https://www.youtube.com/watch?v=wlSG3pEiQdc and what I see mentioned A LOT on other places is that AES-GCM equals AES-CTR+HMAC. Their reasoning then is that the HMAC part is "integrated" into GCM.

Now it can be true that most of those resources have it wrong and the openssl devs have it right. Or there is some nuance there which makes it nuanced. I don't know. Regardless, why not educate the people and blog about this with deep technical blogs on https://www.openssl.org/blog/ Apparently that is desperately needed otherwise we wouldn't be having this discussion now.

[paulidale]

I did write this above:

I'll also note that GCM was designed for data in transit not data at rest

@t8m is not contradicting what I said, he's reiterating it with some additional clarification.

My example works as is for GCM without the integrity tag or GCM where the data is consumed before checking the integrity. This is exactly the case you're looking at. Yes, a HMAC will fix this but only if it is checked before any of the data is consumed. For TLS the packets are smallish and this checking is done as part of the protocol. For large video files in isolation, it won't be.

Security is complex and the specifics of each situation matter. Often a lot.

[markg85]

Ahh, nice @paulidale :)
See, that's what i meant with nuances. As on a surface view it looked contradicting.

As for "my case". I obviously intent to do AES-GCM with the integrity being done on it too. I was totally oblivious to the fact that libressl apparently doesn't do this even though it (again from a surface view) looked like they did.

With all that confusion i can only emphasize this one point more: Please blog about this! Educate the world basically! Not many people are going to find this github issue or follow everything that's written here. But the little snippets of information sprinkled here and there are quite important!

Lastly, do you know of an example code that makes proper use of AES-GCM with integrity that i can look at for my encryption purposes? My eventual endgoal here is this. I want to encrypt large video files and have them be playable - as encrypted - via ffmpeg. In ffmpeg you already have the "crypto" protocol that does AES-CBC. I don't know if i can as this stuff is complex, but i want to add AES-GCM read (forget write for the moment) support there too.

Small but important note. We somehow got on the AES-CTR examples, but my current case is AES-CBC and I intent the future to be AES-GCM.

[paulidale]

Seek and ye shall find: https://github.com/openssl/openssl/blob/master/demos/cipher/aesgcm.c We've been working on our demo code. It's not there yet but there has been some significant progress over the past year or two & we're always willing to accept more samples from third parties.

CBC mode also has a weakness: it permits one subsequent block to be meaningfully corrupted. Not ideal but way better than GCM sans integrity. There are other modes that are more resilient but most have their own nuances. Cryptography has many pitfalls.

As for my example: CTR and GCM are essentially identical. Both have an incrementing counter for each block. GCM adds additional integrity computations (which must slow it down). They aren't quite identical but are close enough for that to not matter in my example. An aside: never, ever, ever, ever reusing the nonce. This totally breaks GCM mode.

[arasic]

@paulidale For your aside note (last phrase of your comment), what if I want to use AES-GCM encryption deterministically (where for any input m, an identical/deterministic output will be generated) with goal to be able to performing searching on encrypted ciphertexts. The reason for using AES-GCM is because of authentication functionality. An attacker that tries to alter ciphertexts would not be given the decrypted plaintext, but would simply fail without giving any further clues/hints.

Considering the 4 inputs

There are four inputs for authenticated encryption: 
* the secret key, 
* initialization vector (IV) (sometimes called a nonce†), 
* the plaintext itself, 
* and optional additional authentication data (AAD).

How insecure would it be to have a fixed IV per plaintext/message input? Do you have any other alternatives/suggestions on how to achieve that?

[admercs]

If users want it, why not support it and let them be the judge of whether or not it fits their use case?

🤔

[admercs]

From a user perspective, the technical merits of GCM vs CTR do not matter. What matters is having the option when it is needed.

Currently, user must install a separate system such as Dexios. Projects that are more receptive to user feedback tend to win in the end.

[t8m]

OpenSSL is not an end-user encryption tool. OpenSSL is a crypto algorithm and protocol library. It is not our intention to change this any time soon although that of course does not mean it will never happen.

If you need an end-user encryption tool, you absolutely must use some other software which might use OpenSSL as the back end for the crypto.

BTW, I did not study Dexios but I would expect it to use AEAD encryption properly - i.e. always check the integrity of the encrypted data before it is output to the user.

[arapov]

@admercs, @eenthropy, On behalf of OpenSSL, I extend our sincere apologies. We recognize that this situation doesn't align with the commitment we made to our community recently. Adjusting takes time and effort, and we appreciate your understanding. We genuinely value community input and feedback. In response, we've opened a ticket on our board to address this. We're actively exploring ways to enhance our library by developing practical tools. While we can't guarantee immediate action on this specific request, we welcome and encourage external contributions.

[markg85]

Thank you for that reply @arapov! It's surprising to me that some openssl maintainers are just violently against this request. And that stance has been the way till just a few days ago. I'm happy it's now at least a thing that looks to be acceptable.

It already seemed super detrimental to me to require everyone using GCM to develop their own ways. A common thing you hear in encryption world is "don't do it yourself! Reuse existing tooling and libraries from the folks who know what they's doing". That is openssl and you guys here for sure!

[kroeckx]

GCM has an authentication tag. In protocols like TLS, this tag is at the end of every TLS record. OpenSSL only decrypts whole TLS records, and so the tag is always checked before data is returned.

The problem with enc is that there is no standard for how to store meta data, like where is the IV, is it split in blocks of fixed size, and things like that. So the only option is that the tag is at the end of the file. If you want to support a safe mode of operation and only return decrypted data after the authentication tag has been verified, this means that you first need to decrypt everything, store it in memory, verify the authentication tag, and then return all the data.

The enc tool currently does not work this way, it will return data once it's decrypted, it does not internally buffer it until it can verify that authentication tag. This means it can return data that has been modified, and you will most likely already process it, which is most likely not what you want.

[markg85]

At the end of a file would be a nasty implementation and make GCM useless for streaming purposes. As you said, you need to decrypt everything before you can verify it.

Would it be possible to say that GCM requires (zero-padded) aligned block sizes where the block size included the tag? So for example to keep it short. Say a format like:

Block:
----------------------------------------------------------------------------------------------------------
| data + padding + fixed length tag | data + padding + fixed length tag | .....
----------------------------------------------------------------------------------------------------------

[kroeckx]

That's basically what TLS does, but the padding isn't needed for each block.

To be able to support such a thing, it's probably best that we have some sort of standardized file format. I'm not aware of any existing file format that we can use to do that.

The other option is that we pick a fixed block size, that you can overwrite with a parameter.

The reason a new file format is handy is that we could write meta data in it so it's easier to decrypt later. People run into problems when we change default. We can write things in it like the block size. If a password was supplied, we could write things like which algorithm was used, for instance pbkdf2 with HMAC-SHA256 and the iteration count.

[markg85]

What kind of file format are you talking about?
A just plain AES encrypted file also doesn't have a specific file format as far as i know.

I'd say that GCM shouldn't need a format either.
The encrypting would require parameters which would result x number of blocks.
De decryption would require "the right parameters" to be able to do something with these blocks (decrypt).

While a file format might be beneficial for just case of "encrypting files and storing that file as encrypted data", it's not directly helpful for the streaming case.

Look at HLS streaming for example with per chunk encryption keys.
Those chunks are part of a m3u "playlist" file. The parameters for those encrypted chunks are stored in the m3u file, not in the encrypted file itself.

With that all in mind, i'd suggest to not bother about a file format at all, right?

[paulidale]

The suggested blocks are effectively a file format.

There needs to be some metadata present and it likely requires more than just the tags.

[ArthurMelin]

Are there any plans to allow AEAD ciphers for non-streaming scenarios? The CLI can also write to a temporary file, then mv/rename the file to its final path only once the tag has been authenticated.

Regarding the question about the file format, maybe the tag can be passed as a separate command line argument, e.g. -in/out-auth-tag?