KIC able to manage vault entities in Admin API

[mflendrich]

Is there an existing issue for this?

• I have searched the existing issues

Problem Statement

After #4558, KIC will be able to work with the default vault (configured via environment variables).

This issue is about extending vault support to also include multiple vaults managed by vault Admin API entities

Proposed Solution

• Introduce a CRD that will map to the vault Admin API entity
• Each vault has its own configuration (the vault config schema depends on the vault implementation - just like plugin schemas differ between plugins) - we'll need an approach similar to KongPlugin's configFrom
• CRD-defined vaults need to be usable within plugin configurations

Additional information

No response

Acceptance Criteria

• As a human operator I can define non-default vaults using a Kubernetes object. KIC configures that vault on the Gateway.
• As an API developer, I can reference a vault (CR-defined vault to be precise) value in a plugin config.

[rainest]

Do we know of a difference between envvar configuration and entity configuration? I guess the latter allows you to create multiple of the same type, but I don't know how common that is. At a basic level, you can configure the others without additional code.

Unlike plugins there are no custom vault providers, so we should define actual types for the config blobs--working with schema-less CRD fields is generally not fun. However, I don't know if there's a good way to map the separate sections all onto config in the Kong JSON--we probably would need to copy the CR object into some intermediate type that does use a schema-less field.

[pmalek]

This should include the RBAC rules being added to role in the chart:

- apiGroups:
  - configuration.konghq.com
  resources:
  - kongvaults
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - configuration.konghq.com
  resources:
  - kongvaults/status
  verbs:
  - get
  - patch
  - update

Currently nightly fails with ingress chart 0.10.1:

kong-controller-88b76dcb6-4xlx5 ingress-controller W0123 10:24:21.089708       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: failed to list *v1alpha1.KongVault: kongvaults.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-controller" cannot list resource "kongvaults" in API group "configuration.konghq.com" at the cluster scope
kong-controller-88b76dcb6-4xlx5 ingress-controller E0123 10:24:21.089780       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.0/tools/cache/reflector.go:229: Failed to watch *v1alpha1.KongVault: failed to list *v1alpha1.KongVault: kongvaults.configuration.konghq.com is forbidden: User "system:serviceaccount:kong:kong-controller" cannot list resource "kongvaults" in API group "configuration.konghq.com" at the cluster scope

[pmalek]

I've created #5471 and put it in 3.1 milestone to track the above.