Check secret count against KMS cache size

[ArchangelSDY]

Background

Self-managed clusters can have encryption-at-rest enabled and have KMS installed. However, do to design limitations in KMS v1, when secrets count exceeds KMS cache size, the performance of secret listing can be significantly impacted due to KeyVault throttling. We need a check to warn user about this risk.

Design

1. Check if we're on a master node of a self-managed cluster.
2. Check API server arguments to see if KMS is enabled: --encryption-provider-config
3. Check cache size in KMS config.

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - kms:
          name: myKmsPluginFoo
          endpoint: unix:///tmp/socketfile.sock
          cachesize: 100
          timeout: 3s
      - kms:
          name: myKmsPluginBar
          endpoint: unix:///tmp/socketfile.sock
          cachesize: 100
          timeout: 3s

4. List secrets and check the number. If it exceeds 80% for KMS cache size, emit a warning.

[bjqian]

Looks interesting.