Elytron replacement in Quarkus distribution of Keycloak

[Pepo48]

PoC: https://github.com/Pepo48/keycloak/tree/file-vault

OVERVIEW

The previous and now removed WildFly distribution provided a built-in vault provider that reads secrets from a keystore-backed Elytron credential store. This was used both by our Keycloak Vault implementation, as well as encrypting config values in standalone.xml. Since Keycloak is now based on Quarkus, the ElytronCS provider, which was implemented as a WildFly extension, can’t be supported anymore.

Naturally, we feel the responsibility to come up with an alternative solution and currently there seem to be two options that we can leverage - SmallRye Keystore and Keycloak Vault.

Adding an integration with upcoming SmallRye Keystore

smallrye/smallrye-config#833

Affected scope: Quarkus properties
Similar functionality in WildFly distribution: elytron-tool.sh script (to mask plaintext property values)

SmallRye Keystore offers to us two main features:

• Adding an additional Keycloak config file in Java Keystore format. That way any config value placed in this Keystore is encrypted. Behind the curtains a Java keystore is used as a config source.
• Encrypting plain text properties in .conf/.propreties files - this is the most requested feature by the community, SmallRye provides us with couple SecretKeysHandler implementations, in the PoC and the examples below we use Jasypt.

Resulting Java Keystore config file can then be created like:

...
keytool -importpass -alias db-password -keystore keystoreFile -storepass keystorepassword
...

Alternatively, encrypting plain text values using jasypt:

...
db-password=${jasypt::ENC(wqp8zDeiCQ5JaFvwDtoAcr2WMLdlD0rjwvo8Rh0thG5qyTQVGxwJjBIiW26y0dtU)}
...

Tests for demonstrating the proposed behavior can be found here.

Adding Keycloak (Java) Keystore Vault

Affected scope: Keycloak realm
Similar functionality in WildFly distribution: Plaintext vault, Elytron Credential Store

The idea was to use the existing Vault interface in order to implement Java Keystore Vault SPI. The current implementation of the provider expects imported password in the keystore, e.g. via following command:

keytool -importpass -alias test_alias -keystore keystore.p12 -storepass keystorepassword

This by default stores the password in a form of generic PBEKey (password based encryption) within SecretKeyEntry using PBEWithMD5AndDES as the algorithm.

Values in the Vault can then be accessed in Realm as (assuming using the REALM_UNDERSCORE_KEY key resolver): ${vault.realm-name_alias}.
E.g. given a Realm is named “my-realm” and an alias is named “test_alias”, the resulting placeholder is ${vault.my-realm_test_alias}.

The current implementation results into following behavior:

Tests for demonstrating the proposed behavior can be found here.

Proposed property nomenclature:

We intentionally decided to distinguish between the two sets of properties, simply because there are relevant use-cases, where a user may want to use both. For example, Keycloak provides a way to use Kubernetes secrets for multiple purposes within a Keycloak realm (see the Available integrations section). For these integrations though, the usage of (plain-text) file-based vault is required. At the same time, a user still may want to leverage the functionality that SmallRye Keystore offers. Our current perception is that these two approaches are not mutually exclusive and they can coexist together. Hence the following proposal:
SmallRye keystore

--config-file-encrypted keystoreFile
--config-file-encrypted-pass someCrazyPassword

Keycloak vault

--vault provider
--vault-keystore-file keystoreFile
--vault-keystore-pass someCrazyPassword
--vault-keystore-type pkcs12

Things to consider:

JKS and JCEKS keystore types are not recommended to use for quite some time[1][2][3]. Do we want to give users an opportunity to choose even less secure Keystore formats, or should we be more opinionated? PKCS#12 is the default recommended format since Java 9[4]. We should also consider FIPS support.

[1] https://www.ndss-symposium.org/wp-content/uploads/2018/02/ndss2018_02B-1_Focardi_paper.pdf
[2] https://neilmadden.blog/2017/11/17/java-keystores-the-gory-details/
[3] https://www.youtube.com/watch?v=viOds2uniC0&list=PLA-8aGQm6tkJgHnLcTyKo_4Qmhc6it5AF&index=1
[4] https://openjdk.org/jeps/229

[pedroigor]

JKS and JCEKS keystore types are not recommended to use for quite some time[1][2][3]. Do we want to give users an opportunity to choose even less secure Keystore formats, or should we be more opinionated? PKCS#12 is the default recommended format since Java 9[4]. We should also consider FIPS support.

I think we should support any type supported by smallrye-config but opinionated. Looking at their documentation looks like PKCS#12 is already the default type.

W.r.t. FIPS, if running in the approved mode you are only left with the BCFKS with the possibility to use PKCS#12 if running in non-approved mode.

In the future, we should also consider supporting PKCS#11 as per #17099.

[pedroigor]

--config-file-encrypted keystoreFile
--config-file-encrypted-pass someCrazyPassword

What about:

--config-keystore <path>
--config-keystore-password <password>

The motivation is that file indicates we are reading a regular file and once we support PKCS#11 there is no file involved but an HSM. Appending encrypted also does not make clear we are reading from a keystore.

[pedroigor]

--vault provider
--vault-keystore-file keystoreFile
--vault-keystore-pass someCrazyPassword
--vault-keystore-type pkcs12

Tha means we now have file and keystore as the possible values for vault option, right?

[abstractj]

@Pepo48 I was reading the proposal again, but it was not entirely clear to me. When using the options which Pedro mention (--vault-keystore...) does that mean that we will leverage from the SmallRey keystore to encrypt data?

Based on what @vmuzikar mentioned, I'm slightly concerned about providing using weak cryptographic algorithms like PBEWithMD5AndDES and probably I'm missing the point here. Even though I don't have any practical exploits to break it, MD5 and DES were already proven to be broken.

It would be possible to replace it by PBEWithHmacSHA256AndAES_128, or PBEWithHmacSHA256AndAES_256?

@pedroigor @vmuzikar, does it make any sense to you?

[vmuzikar]

@abstractj Thank you for pointing this out. I think it's a valid concern. In case of the Keycloak Keystore Vault (which is just an implementation of Keycloak SPI) we are not relying on SmallRye at all. It's plain and simple Java Keystore. And I think it's up to the Keystore which algorithm it uses, or in other words up to the user when creating the keystore. @Pepo48 Is that correct?

[Pepo48]

@abstractj thanks for pointing this out. I have to say that I'm not an security expert, I might be easily wrong. That was actually one of the expected outcomes of this discussion - make sure that the security/cryptographic part is done right.

The fact is that we already discussed this with @vmuzikar during the PoC phase. At first glance, I was concerned about the PBEWithMD5AndDES the same way.

Based on my research though, it's important to differentiate here between the algorithm used for storing the Secret Key Entry and algorithm used for protecting the password. As you can see in the PR[1], I'm already using PBEWithHMACSHA512AndAES_256 to retrieve the encrypted value (just to be completely honest I'm not sure whether it makes any difference here - during the testing I was able to retrieve the value basically with any of the supported PBE algos).

Some details are mentioned in the article Java KeyStores – the gory details[2] I attached in the proposal above and here's the post with an explanation that we discussed with Vašek earlier[3].

Based on my further investigation, Java keytool is pretty limited when it comes to PBE, which means you can't specify the algorithm used for storing the key - the object will always be stored as a generic PBEKey. This can be overcome though by generating the keystore programatically (which gives you more options), but I assume it's up to user at this point.

But again, I would appreciate if someone more experienced in this area could take a look at this.
Thanks!

[1] https://github.com/keycloak/keycloak/pull/19644/files#diff-b21d01880d89de8960bd458d8ab95018e70b4e44fe33b930b624e9903f1661eeR26
[2] https://neilmadden.blog/2017/11/17/java-keystores-the-gory-details/
[3] https://stackoverflow.com/a/71968527

[Pepo48]

Regarding the SmallRye Keystore, it's a separate part of the task. The PR I sent yesterday is purely about adding a Java keystore support as an implementation of the Vault SPI.

SmallRye Keystore is going to be integrated later (but we already know that it works thanks to the PoCs). And we can also divide it into two separate parts. The first provides a possibility to use Java keystore as a Config Source (Quarkus specific mechanism for managing properties) and when it comes to the Java keystore challenges actually they handled them similarly[1]. The second part is meant for masking plaintext values with the help of Secret Handlers (one of which is Jasypt Secret Handler) and we can really specify here more advanced algorithms (e. g. thanks to Jasypt's StandardPBEStringEncryptor[2][3]).

[1] https://github.com/smallrye/smallrye-config/pull/833/files#diff-1c2c8571283665cf773303b5fb3fe79c2c569d865763262819c0727fd29590b5R19
[2] http://www.jasypt.org/api/jasypt/1.8/org/jasypt/encryption/pbe/StandardPBEStringEncryptor.html
[3] https://github.com/smallrye/smallrye-config/pull/833/files#diff-c7ad23c26d495406b3790292905c75a5e56d5bf125ebd76e8382a978a6288440R18

[abstractj]

Thank you both for all the details. Great stuff.

[pedroigor]

${jasypt::ENC(wqp8zDeiCQ5JaFvwDtoAcr2WMLdlD0rjwvo8Rh0thG5qyTQVGxwJjBIiW26y0dtU)}

How users are going to generate the encrypted value? Do we want a command in our CLI or additional script?

Perhaps we want something like:

$ kc.sh tools encrypt <raw value>
${jasypt::ENC(wqp8zDeiCQ5JaFvwDtoAcr2WMLdlD0rjwvo8Rh0thG5qyTQVGxwJjBIiW26y0dtU)}

Not sure about the command position/name but I hope you get the idea. Behind the scenes, we encrypt the value and make it ready for use as an option value. We might also expose specific options to configure how to encrypt the value (e.g: alg).

Looks jasypt also has a CLI that we could rely on. See http://www.jasypt.org/cli.html. However, not sure if want to expose it to users and write docs about how to use it (but how to use our CL, only).

[vmuzikar]

Good point! Maybe question here is whether we even want/need jasypt support at this point. Maybe just the keystore would be enough. As you mentioned, if we want to do it properly, it requires a bit more than just enabling the support for it in config file.

[pedroigor]

I think we should enable it as it is a dependency already added by this work (smallrye-config-jasypt):

• It also provides a more complete solution and avoids users from figuring out by themselves how to encrypt values.
• It completely abstracts how values are encrypted whereas jasypt is only an implementation detail. If in the future we replace jasypt with something else it will be transparent to users
• Our documentation should be simpler and 100% Keycloak-based without mentioning any 3rd party solution.
• The proposed abstraction layer around jasypt should not be so complex to implement

[vmuzikar]

Ok, makes sense. 👍 Let's add a tool to the CLI for generating encrypted values.

[vmuzikar]

Talked today with @stianst around the jasypt support. We aligned on supporting just the Java Keystore for SmallRye Keystore, i.e. skip the jasypt support in the first iteration for now. Jasypt would be added in a future iteration. Does it work for you @Pepo48 @pedroigor?

[abstractj]

I'd go with the idea of using SmallRye as mentioned by Pedro, unless we have a strong argument for it. From my understanding, SmallRye with Jasypt provides support for AES/GCM, while Keycloak Vault supports PBEWithMD5AndDES, considered an insecure password-based key derivation function.

[vmuzikar]

This is actually two separate things and we need both of them. :) We need SmallRye Keystore (without Jasypt for now) to allow encrypting our dist/Quarkus config options, and then we need Keycloak Vault for encrypting sensitive Realm config options, like LDAP credentials. Does it make sense?

[abstractj]

Thanks for bringing some clarity to this @vmuzikar. I got confused with the sentence "currently there seem to be two options that we can leverage - SmallRye Keystore and Keycloak Vault."

[AhazIsraeli]

Hello,
I am using keycloak version 20.0.3.
I have tried using smallrye with the following but this was not working for me.
db-password=${jasypt::ENC(wqp8zDeiCQ5JaFvwDtoAcr2WMLdlD0rjwvo8Rh0thG5qyTQVGxwJjBIiW26y0dtU)}
I have tried to configure my own ConfigSourceInterceptor implementation to try and figure out what was going on, but after creating a jar file and placing it under the <keycloak_home>\providers folder I get an error when trying to apply kc.bat build as follows:
ERROR: Failed to run 'build' command.
ERROR: io.smallrye.config.ConfigSourceFactory: io.smallrye.config.PropertiesLocationConfigSourceFactory not a subtype
And this is before even knowing how to encrypt my own password.
Any help would be appreciated.
Best Regards,
Ahaz

[vmuzikar]

Hello,
thank you for your interest in the feature, but I'm afraid this is a design discussion implying it is not yet implemented in Keycloak. The target release for this is currently Keycloak 22.0.0.

[AhazIsraeli]

So, let me get that right.
For the time being, I have no option for encrypting my database password?

[vmuzikar]

I'm afraid that's correct. :-/ We messed up there a little bit and we are prioritizing a fix for 22.

[AhazIsraeli]

Many thanks. May I ask please what is the timeline for providing this fix? Or when 22 version will be available?
Best Regards,
Ahaz

[vmuzikar]

@AhazIsraeli Unfortunately, I can't share any exact release schedule at the moment but 22 should be released by the end of Q3. But that's just a current estimate, the actual date might be different in the end.

[Pepo48]

Hello everyone,

I went through the discussion, gathered the feedback and polished the final PR[1] accordingly.
The PR covers the Keycloak (Java) Keystore Vault section mentioned above. PTAL and let me know, thanks!

[1] #19644

[Pepo48]

Hi all,

here's the PR[1] that covers the remaining part - integration with the SmallRye Keystore, which allows us to use a Java keystore as a Quarkus config source as well as masking plaintext properties in config files. Feedback is very welcomed, so PTAL and let me know. Thanks!

[1] #20375