fix(certificate): hybrid dp cannot refresh certificate entity with vault reference

[windmgc]

Summary

This PR fixes a problem that a certificate entity with cert/keys stored in a vault-referenced type cannot be refreshed even if the vault reference secret value has been updated both in L1/L2 vault PDK.

The root cause of this problem is that in a dp node running hybrid mode, kong cache interfaces force the cache TTL to be infinite(and it also does not respect cache TTL configured in kong.conf), thus the certificate object cannot be updated unless a new configuration is received and cache purge being called.

The PR fixes the problem by caching the vault reference fields $ref and call kong.vault.update every time in get_certificate, and l1 serializer will keep the value serialized inside the certificate object.

This PR contains a test case that tests a certificate entity with a vault-referenced key that can be refreshed promptly, according to the TTL configured on the vault reference. The test case will fail consistently without the fix.

Checklist

• The Pull Request has tests
• A changelog file has been created under changelog/unreleased/kong or skip-changelog label added on PR if changelog is unnecessary. README.md
• There is a user-facing docs PR against https://github.com/Kong/docs.konghq.com - PUT DOCS PR HERE

Issue reference

FTI-5881

[bungle]

why wasn't this enough:
https://github.com/Kong/kong/blob/master/kong/runloop/certificate.lua#L217

I don't understand. Is it because of hit_level ~= 3, but it shoud be only 3 when caches have been flushed, and in that case DAO has updated it already?

[windmgc]

@bungle The value certificate returned by this line

kong/kong/runloop/certificate.lua

Line 211 in 1714e26

local certificate, err, hit_level = kong.core_cache:get(cache_key,

is something like

{cert = cdata<void *>: 0x0118f09730, key = cdata<void *>: 0x0118f0be20}

Without the $ref fields and plain cert/key fields. Because it has a l1 serializer configured here

kong/kong/runloop/certificate.lua

Line 170 in 1714e26

l1_serializer = parse_key_and_cert,

So we're actually caching these cdata objects inside the L1 cache.

Calling kong.vault.update on this table will have no real effect

On the other hand secrets may gets updated when certificate entity is still in cache(hit L1/L2), so certificate entity's TTL must follow vault's TTL in some way

[bungle]

certificate object cannot be refreshed unless a new configuration is received and cache purge being called.

The certificate entity itself is not refreshed unless there is new config. In which case it is read again from LMDB (because caches are purged)? But vault.update is always called before certificate is served.

The PR fixes the problem by overriding the TTL value returned by the certificate fetching callback function

This is where I have problem of understanding. Why would infinite value be a problem if caches are purged, and entity cannot change without caches being purged?

[bungle]

Because it has a l1 serializer configured here

Oh... that is the key! Thanks.

[bungle]

@windmgc check last commit here for alternate proposal (basically only changing certificate.lua):
#12870

My proposal:
d0deba8#diff-44f5bd52a34ad738a8e7a355f439efdb4cff0711743986d7ce52384e7d2cd1d2

[windmgc]

@bungle I've applied the changes in your proposal, please review again, thanks!

[bungle]

@kikito I added some backport labels to this.

[team-gateway-bot]

Successfully created cherry-pick PR for master:

• https://github.com/kong/kong-ee/pull/8851

[team-gateway-bot]

Successfully created backport PR for release/3.4.x:

• [backport -> release/3.4.x] fix(certificate): hybrid dp cannot refresh certificate entity with vault reference #12884

[team-gateway-bot]

Successfully created backport PR for release/3.5.x:

• [backport -> release/3.5.x] fix(certificate): hybrid dp cannot refresh certificate entity with vault reference #12885

[team-gateway-bot]

Successfully created backport PR for release/3.6.x:

• [backport -> release/3.6.x] fix(certificate): hybrid dp cannot refresh certificate entity with vault reference #12886