konnect update failures after realm added to key-auth plugin

[CharlieC3]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

When I add multiple consumers, each with their own key-auth credential, the KIC attempts to add an invalid realm: null property to the key-auth plugin which the local Kong database accepts, but Konnect rejects with the following error:

Failed pushing configuration to Konnect    {"error": "performing update for https://us.kic.api.konghq.com/kic/api/control-planes/<MY_CONTROL_PLANE_ID> failed: inserting key-auth into state: inserting credential 11d26180-a7c1-4146-a56b-594f6a40a798: entity already exists"}

When Konnect is not used, Kong accepts multiple consumers with key-auth with no issues, and they're all functional as expected. It's only when Konnect is in play that the KIC shows this error.

Expected Behavior

I would either expect:

1. If the local Kong control plane accepts the change, Konnect should too
2. Or the KIC to not add a null realm property to the key-auth plugin, avoiding the attempt to update the plugin in Konnect all together

Steps To Reproduce

1. Deploy KIC, control plane and data plane nodes to Kubernetes cluster
2. Connect KIC to Konnect
3. Deploy key-auth plugin
4. Deploy 1 consumer with credential for key-auth plugin (should succeed)
5. Deploy another consumer with their own unique credential for key-auth plugin (should fail)

Kong Ingress Controller version

3.1

Kubernetes version

1.28.7

Anything else?

I've run the KIC with trace logging enabled and I've dumped generated Kong configuration using this method.

The trace logs didn't reveal much I couldn't already tell from the info logs, but the generated kong configuration revealed something interesting.
The only diff between last_good.json and last_bad.json (reference above linked instructions) was the following:

This may suggest the KIC is trying to modify the key-auth plugin when the second consumer is added, and Konnect is perhaps rejecting this update while the local control plane is accepting it.

[randmonkey]

This is because KIC used the same value to redact the key field, so if there are more than one key_auth plugins in the cluster, they will have the same key field. Then Konnect will reject it because the key is unique for all key_auth plugins.
We have fixed it in #5964 and this is also backported to KIC 3.1. In the coming KIC 3.2 and patch release of 3.1 (will be released soon), this should be fixed.

[CharlieC3]

@randmonkey Thanks so much for the info! Glad to hear a fix is coming soon, as this issue is currently blocking my team from migrating to Konnect from pure OSS. Closing this issue.