Encrypted data reveal the length of the clear text data

[xformerfhs]

Hello, everybody,

I post this issue after having contacted Julien Vehent. He suggested to open an issue here so it can be discussed. So I post it here in the hope to contribute to the usefulness and security of this great software.

Problem description

The length of the encrypted form of the data is always the same length as the clear text data. So it is possible to guess the data value from it’s length.

Let us take this yaml file as an example:

strings:
- a
- aB
- aBc
- aBcD
- aBcDe
numbers:
- 1
- 12
- 123
- 1.2
- 1.2300
booleans:
- true
- false
- y
- n
- on
- off

This is converted to something like this by sops:

field:
- ENC[AES256_GCM,data:Qw==,iv:3Db00Y2V3vELD+cxyeAzaI0WdZOOqw7P3SrIphYlkM8=,tag:Rm1hVD4J7Wcyteh2pUXqQw==,type:str]
- ENC[AES256_GCM,data:hlU=,iv:RStn69CRX7uddn7rKHu0YYYNvsSaxPp0bqgsuamdAIM=,tag:BjVgphl+yqd49lkGFvVtAw==,type:str]
- ENC[AES256_GCM,data:UXxu,iv:a4ufQdVF1dwH1m+wZRdd2tYIZaXKRj/QhzyJ1HDu7dg=,tag:xbUz2UxdVOV/GuDKkjTXdA==,type:str]
- ENC[AES256_GCM,data:HA1E1Q==,iv:6Y7pxNWY4jZn5OLfa4Sd/DQzo9ye2ubM1u2e5IQN9I0=,tag:a1WgmqUhJKvqo/qKbWvTiQ==,type:str]
- ENC[AES256_GCM,data:PYfdMqo=,iv:X+Z1DdUV7A+4p8Yj8zFYBhJtNJFLm1mqOmHevMcDxn0=,tag:l97d0/y8b/uBM50DLZ/TzA==,type:str]
numbers:
- ENC[AES256_GCM,data:IQ==,iv:AFshGBvOry/iUaPNm9ZgFrkb03AevzERJhEH8AiCfcQ=,tag:sILDbauS1RqIBCQM6XWArQ==,type:int]
- ENC[AES256_GCM,data:tSI=,iv:vvnAkSw69eQL64bmsZpjrCIh16qI13esc3tLsLGPlj0=,tag:Pf3Y/m3xP81fSVrcMez0eg==,type:int]
- ENC[AES256_GCM,data:rBeB,iv:xgzYmW9i6y61Udt226ooTx/vzcOQFtgIAVyomUNGix0=,tag:h8yHZasQGHa3AY4VG1J0ZA==,type:int]
- ENC[AES256_GCM,data:rNTf,iv:EzN8AQ4T63J0YjV+y2y6ViOe9M2KrERICpbEsTU/1io=,tag:Thftkoq//92XyzS3ZmKBgg==,type:float]
- ENC[AES256_GCM,data:Gvzw2A==,iv:OrdWXplruWs4BG2DduSL4Ld72Teao62WivMWISkG9+E=,tag:6XF2e3HNwyV3YB7AyPw2Rg==,type:float]
booleans:
- ENC[AES256_GCM,data:kU1bxA==,iv:h+B5QbhktBIP2gSoyPlYtFq+/LDlvRP5ndXuL8GN6xY=,tag:PIV5v63hDg4wPnEsFikeAA==,type:bool]
- ENC[AES256_GCM,data:yRRf3Kc=,iv:+kJKCNe2BHFU0zEK4IJoDUk+/VWXgiddyYxhFlqNnZA=,tag:dXw+8mSWKtTJd4F5VMHp/Q==,type:bool]
- ENC[AES256_GCM,data:kU1bxA==,iv:h+B5QbhktBIP2gSoyPlYtFq+/LDlvRP5ndXuL8GN6xY=,tag:PIV5v63hDg4wPnEsFikeAA==,type:bool]
- ENC[AES256_GCM,data:yRRf3Kc=,iv:+kJKCNe2BHFU0zEK4IJoDUk+/VWXgiddyYxhFlqNnZA=,tag:dXw+8mSWKtTJd4F5VMHp/Q==,type:bool]
- ENC[AES256_GCM,data:kU1bxA==,iv:h+B5QbhktBIP2gSoyPlYtFq+/LDlvRP5ndXuL8GN6xY=,tag:PIV5v63hDg4wPnEsFikeAA==,type:bool]
- ENC[AES256_GCM,data:yRRf3Kc=,iv:+kJKCNe2BHFU0zEK4IJoDUk+/VWXgiddyYxhFlqNnZA=,tag:dXw+8mSWKtTJd4F5VMHp/Q==,type:bool]

When we look at the encrypted data (the values after the data: tag) we see the following:

Type str:

Clear text	Clear text length	Encrypted bytes	Encrypted length
a	1	0x43	1
aB	2	0x8655	2
aBc	3	0x517c6e	3
aBcD	4	0x1c0d44d5	4
aBcDe	5	0x3d87dd32aa	5

Type int:

Clear text	Clear text length	Encrypted bytes	Encrypted length
1	1	0x21	1
12	2	0xb522	2
123	3	0xac1781	3

Type float:

Clear text	Clear text length	Encrypted bytes	Encrypted length
1.2	3	0xacd4df	3
1.2300	6 (true length is 4)	0x1afcf0d8	4

Type boolean:

Clear text	Clear text length	Encrypted bytes	Encrypted length
true	4	0x914d5bc4	4
false	5	0xc9145fdca7	5

There are two observations:

1. The data is "canonicalized" before being encrypted
   1. This is most easily seen for the boolean values where all variants are mapped to true and false
   2. All the other types are processed, as well, as trailing zeroes (for int and float) or blanks (for all types) are removed before the data is encrypted
2. The length of the encrypted bytes is always the same as the length of the "canonicalized" clear text

What is the implication?

An adversary does not know the content of a str, int or float but in the current implementation of sops he knows the clear text length from the encrypted lengths and this is a valuable information. If the clear text is short, like e.g. up to 5 characters, he can simply guess or brute force the clear text value. Having the length of the clear text limits the number of tries he has perform. If the length of the clear text data is long enough (whatever "long enough" means) this does not reduce the security as there are just to many possibilities to try.

Numbers tend to be short and so can easily be guessed by just knowing their length. So this is a problem for numbers and short strings.

A special case are boolean values. They can only have 2 different values, so it is trivial to "guess" the value. The adversary will have the correct value after at most two tries. All he needs to know is the type.

What to do about it?

I propose to conceal the true length of the data by prepending and appending random data to them before they are encrypted. There should be 0-m data prepended and 0-n data appended. The lengths of these "blindings" should be stored with the data.

Here is an example of how that could be done:

1. Clear text: data
2. Clear text as bytes: 0x64617461
3. Prepend e.g. 3 random bytes and append e.g. 4 random bytes: 0x168ffa64617461a29fbc5d
4. Append the lengths of the prepended and appended bytes: 0x168ffa64617461a29fbc5d0304

So the data that is finally encrypted is 0x168ffa64617461a29fbc5d0304, instead of 0x64617461.

On decryption the blinding lengths are read from the end of the decrypted bytes. With this information the blinding can be removed and the data restored.

This is just an example. There may be better ways to do this.

There should just be a method to conceal the length of the clear text.

Of course one needs some means to differentiate between values that use blinding and others that do not.

Regarding the boolean values I think it does not make sense to "blind" them like in the example above as there are only two possible clear text values. The only means to protect them would be to conceal the type and treat them in a special way.

[autrilla]

Thanks for reporting this. I agree this is an issue, but I think it's fairly small.

If the clear text is short, like e.g. up to 5 characters, he can simply guess or brute force the clear text value.

They'd need to know what the value is used for (realistically, this should be easy to figure out from the cleartext keys and file name). Even then, fixing exposing the payload length would be security by obscurity, and you probably want to have some other barrier to an attack other than obscurity.

Numbers tend to be short and so can easily be guessed by just knowing their length. So this is a problem for numbers and short strings.

Padding would only fix this in some cases. If your short number is something like replica_count: 3, the attacker also knows that it's not going to be a very large number based on the context. It seems rare to me that you'd use a low-entropy secret for something that you don't want brute-forced. Do you have a specific case of a short value whose length you'd want to keep secret?

So yes, it's not great that the payload length is exposed, but fixing it introduces other questions. How small does a payload need to be before we pad it? We'd definitely welcome a design and implementation for this problem that is backwards compatible.

[xformerfhs]

I am a security consultant and I reported this issue because I found several instances of this during reviews. The clients had used parameters with names like remoteuser or db_password and their lengths were between 3 and 7 characters. So, it is easy to see what the value is used for.

Knowing that an encrypted value of a parameter named db_password has a length of exactly 5 characters helps an attacker. He does not have to test all the other lengths when brute-forcing. Just the length he knows. So, if the company he is attacking is named ABC and the password is 6 characters he could guess the password is probably something like ABC123. This further reduces the search space.

I found that people used very weak credentials which were way too short and presumed that they were secure because it was encrypted. In these cases sops gave them a false sense of security.

If the data is long enough this is not a weakness. Just for the short ones.

Well, "security by obscurity" is not bad in itself. It is bad when I try to hide an implementation or when I try to "hide" data with an easily breakable algorithm. It is useful and it it is sensible to use it when one hides information from an attacker. So, yes, it obscures the lengths of the data and yes, in this case, this is good. He can no longer deduce the real length of the data. It does not weaken the security, at all. On the contrary, it strengthens security.

It probably would be the easiest way to pad all data, regardless of the length. There should be no "threshold". Just pad everything.

For a backwards compatible implementation one could put a parameter in the encrypted value: E.g. add a new parameter to the ENC field, like format and put some value in it that tells sops what the format is. If the format parameter is missing one can assume that it is a parameter without blinding.

Or one could prepend the data field with a special character, like e.g. # followed by a character that specifies the format version. So every data that begins with #0 is a blinded data value. Everything that does not start with #0 is not blinded.

Both of these methods leave room to expand them if the format should have to be changed again for some reason.

These are just two examples. There are probably other ways to implement it.

This adds complexity to the code and so there should be a good reason to implement it. As I said, I opened this issue because I saw these cases where the users used very weak credentials with easily guessable value. Maybe some people could add their experience here.

[autrilla]

I am a security consultant and I reported this issue because I found several instances of this during reviews. The clients had used parameters with names like remoteuser or db_password and their lengths were between 3 and 7 characters. So, it is easy to see what the value is used for.

Knowing that an encrypted value of a parameter named db_password has a length of exactly 5 characters helps an attacker. He does not have to test all the other lengths when brute-forcing. Just the length he knows. So, if the company he is attacking is named ABC and the password is 6 characters he could guess the password is probably something like ABC123. This further reduces the search space.

I found that people used very weak credentials which were way too short and presumed that they were secure because it was encrypted. In these cases sops gave them a false sense of security.

If the data is long enough this is not a weakness. Just for the short ones.

Well, "security by obscurity" is not bad in itself. It is bad when I try to hide an implementation or when I try to "hide" data with an easily breakable algorithm. It is useful and it it is sensible to use it when one hides information from an attacker. So, yes, it obscures the lengths of the data and yes, in this case, this is good. He can no longer deduce the real length of the data. It does not weaken the security, at all. On the contrary, it strengthens security.

Of course, I never implied security by obscurity reduced security. It's just that it's an extremely weak measure to rely on, and you should have other defences. I think we agree on this, and I imagine you're recommending your clients they use longer secrets in the first place. To me, that's the actual solution to this problem.

But yes, SOPS obscuring that security hole might be helpful. It depends on whether users notice that SOPS exposes the secret length, and as such, notice they should use longer secrets so the secret length is (largely) irrelevant to an attacker. If SOPS padded everything, maybe users wouldn't notice. From your report, it seems like users do not notice anyway.

It probably would be the easiest way to pad all data, regardless of the length. There should be no "threshold". Just pad everything.

Sounds good, but even then, we still have to decide how much entropy we want to add to the length. We can't just pad with a static length, because then the attacker can figure that out from the source code. We can't pad everything with a random number of bytes either: the attacker can infer the probabilities of each plaintext length, which IMO can be especially bad for short plaintexts in some situations. E.g. if we pad by 100 bytes +- 100 bytes, and I see a ciphertext that's 105 bytes, I can make very good assumptions that the plaintext was short. This issue doesn't exist if we pad all messages to e.g. 4096 bits (arbitrary value based on 256 bits of entropy and assuming data in SOPS is hex encoded). The attacker has no idea of the original size for plaintexts shorter than 4096 bits, and for lengths longer than that, the length information is IMO irrelevant.

For a backwards compatible implementation one could put a parameter in the encrypted value: E.g. add a new parameter to the ENC field, like format and put some value in it that tells sops what the format is. If the format parameter is missing one can assume that it is a parameter without blinding.

Or one could prepend the data field with a special character, like e.g. # followed by a character that specifies the format version. So every data that begins with #0 is a blinded data value. Everything that does not start with #0 is not blinded.

Both of these methods leave room to expand them if the format should have to be changed again for some reason.

These are just two examples. There are probably other ways to implement it.

This adds complexity to the code and so there should be a good reason to implement it. As I said, I opened this issue because I saw these cases where the users used very weak credentials with easily guessable value. Maybe some people could add their experience here.

[xformerfhs]

Hello, Adrian,

and thanks for your answer.

When I read it it reminded me that I forgot to mention an important point in my first post. The data plus padding should have a minimum length. I totally agree with you on that. In fact, in one of my projects, which is here on GitHub, as well, I used a combination of blinding and block padding with a minimum length of 2 AES blocks (i.e. 16 bytes).

Maybe, then it would be a good idea to just pad the data to a minimum length. Maybe it would be sufficient to pad it to a minimum of 16 or maybe 20 bytes. That is long enough to make any brute-forcing unusable. One just needs a way to know where the real data are. This could be either a length field or some indicator whether padding is present, or not in the clear text data. Or one always uses padding with at least one padding character. If padding is present one could use abritrary tail byte padding that has no "invalid padding".

What do you think about this?

[autrilla]

I think the indicator of whether there's padding or not doesn't need to be in the clear text data, since that detail is already public based on the SOPS version used. But the length of the padding definitely needs to be in there. I think using the first (or last, I don't have a preference here) byte of the cleartext for the padding length should be sufficient. We can't use arbitrary padding bytes, because SOPS makes no assumptions about the input, the user could provide any byte sequence.

While we're at this, we might as well also put the data type inside the encrypted blob, instead of outside. It might also be interesting to use something like protobuf (since we already have a dependency on it) to serialize the (padding_length, data_type, blob), rather than rolling our own serialization. And since we're going to have to change every single value in all SOPS files, it might also be a good chance to make the entire thing inside ENC[...] a protobuf-encoded value instead of having our own ad-hoc parsing.