Inconsistent public keys loading order

[chrisFrodo]

Hello !

I'm using Apache with mod_auth_openidc and Keycloak. Both are configured to use « private_key_jwt » using a JWKS file exposed by the client.

My JWKS looks like this :

{"keys": [
    {"e": "AQAB", "kid": "key1", "kty": "RSA", "n": "lnmIE8ifjLZw6z4 ... YK9EA5nRIUeVqQ", "use": "sig"},
    {"e": "AQAB", "kid": "key2", "kty": "RSA", "n": "hPz_l096fpiBOnH ... mrR1USHEOw", "use": "enc", "alg": "RSA-OAEP", "enc": "A256GCM"}
    ]
}

My mod_auth_openidc.conf file has this two attributes to use signature and encryption :

OIDCPublicKeyFiles key1#/etc/apache2/key1.pem key2#/etc/apache2/key2.pem
OIDCPrivateKeyFiles key1#/etc/apache2/key1.key key2#/etc/apache2/key2.key

The problem is, sometimes (around 1 restart of Apache on 6) when i'm authenticating on my client, the following error shows in Apache logs :

oidc_util_json_string_print: oidc_util_check_json_error: response contained an "error" entry with value: ""unauthorized_client""
oidc_util_json_string_print: oidc_util_check_json_error: response contained an "error_description" entry with value: ""Unable to load public key""

And on Keycloak :

PublicKey wasn't found in the storage. Requested kid: 'key2' . Available kids: '[key1]'

To debug this, i made a local, full HTTP infrastructure and sniffed packets with Wireshark. Here's what i've observed :

In a working scenario :
1. Client sends JWS to Keycloak at « /token », JWS header contains the KID of the SIGNING key (« key1 » in my case)
2. Keycloak responds with an access token, refresh token and JWE id token (header contains the KID of the encryption key « key2 » in my case)

In a non-working scenario :
1. Client sends JWS to Keycloak at « /token », JWS header contains the KID of the ENCRYPTION key (« key2 » in my case)
2. Keycloak respond with an error : {"error":"unauthorized_client","error_description":"Unable to load public key"}

As mentionned in the thread https://groups.google.com/g/mod_auth_openidc/c/KQq0mOPONZE/m/ZlcCRtuuAAAJ, the first key of the attribute « OIDCPublicKeyFiles » should be the signing key, or, as this happen around 15% of the time when Apache restarts, i think that the module, sometimes, use the second key as the signing key and not the first one.

Later on, i've looked inside the source code. I've seen that keys are stored using hash tables from module « apr_hash ». It seems that the way entries are stored inside hash tables (index with the hashes) doesn't certify that the first entry added will still be the first entry later.

It means that when the module wants to sign, it uses « apr_hash_first() » which doesn't return the first key (signature) but the encryption key.

What do you think ?

Thanks !

[zandbelt]

I think your analysis is correct. The multiple key config feature was primarily designed for signing key rollover: you would send both public keys to the peer so that at some point in time you can seamlessly remove the older one.

It seems you're looking to use a different key for signing and encryption which is prevented by the lack of ordering, I will fix that.

[chrisFrodo]

Exactly ! Thanks.

[zandbelt]

would you be able to test 2019f54 ?

[chrisFrodo]

Hello,

Thank you for the fast fix. I've tested the commit exactly in the same environment, I've restarted Apache more than 20 times and i couldn't reproduce the bug. I think it has been fixed. Thanks again.