WIP: CNTRLPLANE-3237: Introduce KMSProviderConfig#2163
WIP: CNTRLPLANE-3237: Introduce KMSProviderConfig#2163ardaguclu wants to merge 5 commits intoopenshift:masterfrom
Conversation
openshift-ci-robot
added
the
jira/valid-reference
openshift-ci
bot
added
the
do-not-merge/work-in-progress
|
@ardaguclu: This pull request references CNTRLPLANE-3237 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
WalkthroughMove KMS config from secret annotations into secret Data, add per-key KMS socket endpoints and KMS provider config types, propagate provider configs into generated encryption-config secrets, update controllers/tests to handle the new keys, and expand the mock KMS DaemonSet to per-key plugin containers. Changes
Sequence Diagram(s)sequenceDiagram
participant SC as StateController
participant KC as KeyController
participant EC as encryptionconfig.ToSecret
participant K8s as Kubernetes Secrets
participant KMS as KMS Plugin
SC->>KC: compute desired keys (per-key KMS mode)
KC->>SC: return per-key KMSEncryptionConfig + KMSProviderConfig
SC->>EC: call ToSecret(encryptionCfg, kmsProviderConfigs)
EC->>K8s: write encryption-config Secret Data (encryption payload + provider-config-<keyID>)
KMS->>K8s: KMS plugin listens on /var/run/kmsplugin/kms-<keyID>.sock (apiserver consumes socket)
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes 🚥 Pre-merge checks | ✅ 8 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (8 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: ardaguclu The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
@ardaguclu: This pull request references CNTRLPLANE-3237 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (4)
test/library/encryption/kms/assets/k8s_mock_kms_plugin_daemonset.yaml (2)
54-253: Consider templating to reduce duplication in the test asset.The 10 nearly-identical container definitions differ only in the container name and socket path number. While this is a test asset and correctness is not affected, this repetition is error-prone to maintain.
If the templating system supports it (Go templates are used based on
{{ .Image }}), consider generating the containers programmatically:{{- range $i := seq 1 10 }} - name: kms-plugin-{{ $i }} image: {{ $.Image }} ... args: - | rm -f /var/run/kmsplugin/kms-{{ $i }}.sock exec /usr/local/bin/mock-kms-plugin -listen-addr=unix:///var/run/kmsplugin/kms-{{ $i }}.sock -config-file-path=/etc/softhsm-config.json {{- end }}🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@test/library/encryption/kms/assets/k8s_mock_kms_plugin_daemonset.yaml` around lines 54 - 253, Replace the ten near-identical container blocks (names like kms-plugin-1..kms-plugin-10 and socket paths /var/run/kmsplugin/kms-#.sock) with a Go-template range loop that iterates from 1 to 10, uses the loop index for the container name and socket filename, and references the image as $.Image; keep the same securityContext, command, args (but with the index substituted), and volumeMounts (socket, softhsm-config with subPath, softhsm-tokens) inside the loop so the behavior remains identical while removing duplication.
57-58: Consider replacingprivileged: truewith specific capabilities or removing it if unnecessary.All 10 plugin containers and the init container use
privileged: true, but their operations—listening on Unix sockets, reading configuration files, and accessing mounted volumes—do not require privileged mode. For a test environment, replacing this with specific capabilities (e.g.,CAP_CHOWN) or a non-privileged security context would align with least-privilege principles. If socket creation and file access work without elevated privileges, remove thesecurityContextentirely.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@test/library/encryption/kms/assets/k8s_mock_kms_plugin_daemonset.yaml` around lines 57 - 58, The DaemonSet uses securityContext with privileged: true for the init container and all plugin containers; change this to follow least-privilege by removing the privileged setting or replacing it with narrow POSIX capabilities (e.g., add a securityContext.capabilities.add list with only needed capabilities like CAP_CHOWN) or a readOnlyRootFilesystem / runAsNonRoot setup if possible; update the initContainer and container specs (look for securityContext and privileged: true entries) to either remove the entire securityContext when not needed or explicitly list minimal capabilities and non-root settings so sockets and file access still work without full privileged mode.pkg/operator/encryption/testing/helpers.go (1)
24-29: Test constants duplicate production constants.These local constants duplicate values from
pkg/operator/encryption/secrets/types.go. While this avoids import cycles, it creates a maintenance burden if the values change.Consider adding a comment noting these must stay in sync with
secrets.EncryptionSecretKMSEncryptionConfigandsecrets.EncryptionSecretKMSProviderConfig.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@pkg/operator/encryption/testing/helpers.go` around lines 24 - 29, The test constants (encryptionSecretKeyDataForTest, encryptionSecretMigratedTimestampForTest, encryptionSecretMigratedResourcesForTest, encryptionSecretKMSEncryptionConfigForTest, encryptionSecretKMSProviderConfigForTest) duplicate production values; add a clear comment above these declarations stating they must remain in sync with the production symbols in pkg/operator/encryption/secrets (specifically secrets.EncryptionSecretKMSEncryptionConfig and secrets.EncryptionSecretKMSProviderConfig) to avoid drift and explain why the duplication exists (to avoid import cycles).pkg/operator/encryption/controllers/key_controller.go (1)
280-289: Hardcoded KMS provider configuration noted.The TODO comment indicates these values will be replaced by API when ready. For Tech Preview, this is acceptable, but consider:
- These hardcoded values will need to be parameterized before GA
- The image reference uses a personal/support account registry path which may not be suitable for production
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@pkg/operator/encryption/controllers/key_controller.go` around lines 280 - 289, The code currently assigns hardcoded KMS values in ks.KMSProviderConfig using state.KMSProviderConfig and state.VaultProviderConfig (fields Image, VaultAddress, VaultNamespace, TransitKey, TransitMount); replace these hardcoded literals with configurable sources (e.g., read from controller config, CR spec, environment variables or a ConfigMap/Secret) and fall back to safe defaults only for Tech Preview, ensure the image string is not a personal registry (make it configurable and validate it) and update/remove the TODO accordingly so the values are injected rather than baked into the key_controller.go logic.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@pkg/operator/encryption/controllers/key_controller.go`:
- Around line 280-289: The code currently assigns hardcoded KMS values in
ks.KMSProviderConfig using state.KMSProviderConfig and state.VaultProviderConfig
(fields Image, VaultAddress, VaultNamespace, TransitKey, TransitMount); replace
these hardcoded literals with configurable sources (e.g., read from controller
config, CR spec, environment variables or a ConfigMap/Secret) and fall back to
safe defaults only for Tech Preview, ensure the image string is not a personal
registry (make it configurable and validate it) and update/remove the TODO
accordingly so the values are injected rather than baked into the
key_controller.go logic.
In `@pkg/operator/encryption/testing/helpers.go`:
- Around line 24-29: The test constants (encryptionSecretKeyDataForTest,
encryptionSecretMigratedTimestampForTest,
encryptionSecretMigratedResourcesForTest,
encryptionSecretKMSEncryptionConfigForTest,
encryptionSecretKMSProviderConfigForTest) duplicate production values; add a
clear comment above these declarations stating they must remain in sync with the
production symbols in pkg/operator/encryption/secrets (specifically
secrets.EncryptionSecretKMSEncryptionConfig and
secrets.EncryptionSecretKMSProviderConfig) to avoid drift and explain why the
duplication exists (to avoid import cycles).
In `@test/library/encryption/kms/assets/k8s_mock_kms_plugin_daemonset.yaml`:
- Around line 54-253: Replace the ten near-identical container blocks (names
like kms-plugin-1..kms-plugin-10 and socket paths /var/run/kmsplugin/kms-#.sock)
with a Go-template range loop that iterates from 1 to 10, uses the loop index
for the container name and socket filename, and references the image as $.Image;
keep the same securityContext, command, args (but with the index substituted),
and volumeMounts (socket, softhsm-config with subPath, softhsm-tokens) inside
the loop so the behavior remains identical while removing duplication.
- Around line 57-58: The DaemonSet uses securityContext with privileged: true
for the init container and all plugin containers; change this to follow
least-privilege by removing the privileged setting or replacing it with narrow
POSIX capabilities (e.g., add a securityContext.capabilities.add list with only
needed capabilities like CAP_CHOWN) or a readOnlyRootFilesystem / runAsNonRoot
setup if possible; update the initContainer and container specs (look for
securityContext and privileged: true entries) to either remove the entire
securityContext when not needed or explicitly list minimal capabilities and
non-root settings so sockets and file access still work without full privileged
mode.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 74d7db0f-ae9d-4fe6-adb1-90b5bb8f25cc
📒 Files selected for processing (18)
pkg/operator/encryption/controllers/helpers_test.gopkg/operator/encryption/controllers/key_controller.gopkg/operator/encryption/controllers/key_controller_test.gopkg/operator/encryption/controllers/state_controller.gopkg/operator/encryption/controllers/state_controller_test.gopkg/operator/encryption/deployer/unionrevisionedpod_test.gopkg/operator/encryption/encryptionconfig/config.gopkg/operator/encryption/encryptionconfig/config_test.gopkg/operator/encryption/encryptionconfig/secret.gopkg/operator/encryption/observer/observe_encryption_config_test.gopkg/operator/encryption/secrets/secrets.gopkg/operator/encryption/secrets/secrets_test.gopkg/operator/encryption/secrets/types.gopkg/operator/encryption/state/types.gopkg/operator/encryption/statemachine/transition_test.gopkg/operator/encryption/testing/helpers.gotest/e2e-encryption/encryption_test.gotest/library/encryption/kms/assets/k8s_mock_kms_plugin_daemonset.yaml
ardaguclu
force-pushed
the
kms-v2-provider-config
branch
from
d9bafe1 to
585e22b
Compare
|
@ardaguclu: This pull request references CNTRLPLANE-3237 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@pkg/operator/encryption/secrets/secrets.go`:
- Around line 66-75: The new code drops the legacy annotation-backed KMS secret
read path, causing migration breakage; update the ToKeyState logic in secrets.go
so that after attempting to unmarshal EncryptionSecretKMSEncryptionConfig from
s.Data it falls back to checking the legacy annotation (the previous KMS
encryption config stored on the Secret's annotations), unmarshal that into
apiserverconfigv1.KMSConfiguration and set key.KMSEncryptionConfig, and only
return the "KMSEncryptionConfig can not be nil" error if both the Data field and
the legacy annotation are absent or both fail to parse; reference the
EncryptionSecretKMSEncryptionConfig symbol, s.Data / s.Annotations,
key.KMSEncryptionConfig and state.KeyState when making the change.
- Around line 135-149: FromKeyState currently allows serializing a key secret
with ks.Mode == state.KMS even when ks.KMSEncryptionConfig is nil, which breaks
round-trip with ToKeyState; update FromKeyState to validate and reject invalid
KMS states before writing: if ks.Mode == state.KMS and ks.KMSEncryptionConfig ==
nil return a clear error instead of producing a secret, and keep the existing
serialization of KMSEncryptionConfig and KMSProviderConfig
(EncryptionSecretKMSEncryptionConfig, EncryptionSecretKMSProviderConfig) only
when those fields are non-nil; reference the FromKeyState function, ks.Mode,
KMSEncryptionConfig, KMSProviderConfig, and the constants used for secret keys
when implementing the check and error return.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 446c51cf-2cc5-43fb-8ddc-4a15e21f75ff
📒 Files selected for processing (13)
pkg/operator/encryption/controllers/helpers_test.gopkg/operator/encryption/controllers/key_controller.gopkg/operator/encryption/controllers/key_controller_test.gopkg/operator/encryption/controllers/state_controller.gopkg/operator/encryption/controllers/state_controller_test.gopkg/operator/encryption/deployer/unionrevisionedpod_test.gopkg/operator/encryption/encryptionconfig/secret.gopkg/operator/encryption/observer/observe_encryption_config_test.gopkg/operator/encryption/secrets/secrets.gopkg/operator/encryption/secrets/types.gopkg/operator/encryption/state/types.gopkg/operator/encryption/testing/helpers.gotest/e2e-encryption/encryption_test.go
✅ Files skipped from review due to trivial changes (4)
- pkg/operator/encryption/deployer/unionrevisionedpod_test.go
- pkg/operator/encryption/controllers/helpers_test.go
- test/e2e-encryption/encryption_test.go
- pkg/operator/encryption/controllers/key_controller_test.go
🚧 Files skipped from review as they are similar to previous changes (6)
- pkg/operator/encryption/observer/observe_encryption_config_test.go
- pkg/operator/encryption/controllers/key_controller.go
- pkg/operator/encryption/encryptionconfig/secret.go
- pkg/operator/encryption/controllers/state_controller_test.go
- pkg/operator/encryption/state/types.go
- pkg/operator/encryption/testing/helpers.go
ardaguclu
force-pushed
the
kms-v2-provider-config
branch
from
585e22b to
e143c1e
Compare
|
@ardaguclu: This pull request references CNTRLPLANE-3237 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
There was a problem hiding this comment.
♻️ Duplicate comments (1)
pkg/operator/encryption/secrets/secrets.go (1)
135-149:⚠️ Potential issue | 🟠 MajorReject invalid KMS key states before writing the secret.
ToKeyStaterejects KMS secrets withoutKMSEncryptionConfig(lines 73-75), butFromKeyStatestill serializesks.Mode == state.KMSwith that field potentially unset. This makes the conversion non-round-trippable and lets callers persist a secret this package cannot read back.🛡️ Proposed fix
+ if ks.Mode == state.KMS && ks.KMSEncryptionConfig == nil { + return nil, fmt.Errorf("KMSEncryptionConfig cannot be nil when mode is KMS") + } + if ks.KMSEncryptionConfig != nil { kmsEncCfgJSON, err := json.Marshal(ks.KMSEncryptionConfig) if err != nil { return nil, err } s.Data[EncryptionSecretKMSEncryptionConfig] = kmsEncCfgJSON }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@pkg/operator/encryption/secrets/secrets.go` around lines 135 - 149, FromKeyState currently allows serializing ks with ks.Mode == state.KMS even when ks.KMSEncryptionConfig is nil, producing secrets that ToKeyState cannot read back; update FromKeyState to validate and reject invalid KMS states before writing the secret by returning an error when ks.Mode == state.KMS and ks.KMSEncryptionConfig == nil (and likewise validate any other required KMS fields such as KMSProviderConfig if your domain requires it) instead of proceeding to marshal and set EncryptionSecretKMSEncryptionConfig/EncryptionSecretKMSProviderConfig in s.Data.
🧹 Nitpick comments (1)
test/library/encryption/kms/assets/k8s_mock_kms_plugin_daemonset.yaml (1)
54-253: Consider templating the repeated container definitions.The 10 container blocks are nearly identical, differing only in the container name and socket index. While acceptable for a test asset, this creates maintenance burden if you need to change shared parameters (image, volumeMounts, securityContext).
Since this YAML already uses Go templating (
{{ .Index }},{{ .Image }}), you could reduce duplication by templating the containers as well:♻️ Proposed refactor using Go range template
containers: - - name: kms-plugin-1 - image: {{ .Image }} - imagePullPolicy: IfNotPresent - securityContext: - privileged: true - command: - - /bin/sh - - -c - args: - - | - rm -f /var/run/kmsplugin/kms-1.sock - exec /usr/local/bin/mock-kms-plugin -listen-addr=unix:///var/run/kmsplugin/kms-1.sock -config-file-path=/etc/softhsm-config.json - volumeMounts: - - name: socket - mountPath: /var/run/kmsplugin - - name: softhsm-config - mountPath: /etc/softhsm-config.json - subPath: softhsm-config.json - - name: softhsm-tokens - mountPath: /var/lib/softhsm/tokens - - name: kms-plugin-2 - ... (repeat for 3-10) +{{- range $i := list 1 2 3 4 5 6 7 8 9 10 }} + - name: kms-plugin-{{ $i }} + image: {{ $.Image }} + imagePullPolicy: IfNotPresent + securityContext: + privileged: true + command: + - /bin/sh + - -c + args: + - | + rm -f /var/run/kmsplugin/kms-{{ $i }}.sock + exec /usr/local/bin/mock-kms-plugin -listen-addr=unix:///var/run/kmsplugin/kms-{{ $i }}.sock -config-file-path=/etc/softhsm-config.json + volumeMounts: + - name: socket + mountPath: /var/run/kmsplugin + - name: softhsm-config + mountPath: /etc/softhsm-config.json + subPath: softhsm-config.json + - name: softhsm-tokens + mountPath: /var/lib/softhsm/tokens +{{- end }}🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@test/library/encryption/kms/assets/k8s_mock_kms_plugin_daemonset.yaml` around lines 54 - 253, Refactor the repeated kms-plugin-* container blocks into a Go template range to eliminate duplication: create a slice (e.g., .Plugins or .NumPlugins) and iterate (range) to render each container using a template body that sets name as "kms-plugin-{{index}}" and socket paths like /var/run/kmsplugin/kms-{{index}}. Keep shared fields (image / Image, imagePullPolicy, securityContext, command, args, volumeMounts, softhsm-config subPath) inside the templated block so one change updates all; update any references to container names or sockets (e.g., kms-plugin-1..kms-plugin-10 and -listen-addr=unix:///var/run/kmsplugin/kms-<index>.sock) to use the template index variable.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Duplicate comments:
In `@pkg/operator/encryption/secrets/secrets.go`:
- Around line 135-149: FromKeyState currently allows serializing ks with ks.Mode
== state.KMS even when ks.KMSEncryptionConfig is nil, producing secrets that
ToKeyState cannot read back; update FromKeyState to validate and reject invalid
KMS states before writing the secret by returning an error when ks.Mode ==
state.KMS and ks.KMSEncryptionConfig == nil (and likewise validate any other
required KMS fields such as KMSProviderConfig if your domain requires it)
instead of proceeding to marshal and set
EncryptionSecretKMSEncryptionConfig/EncryptionSecretKMSProviderConfig in s.Data.
---
Nitpick comments:
In `@test/library/encryption/kms/assets/k8s_mock_kms_plugin_daemonset.yaml`:
- Around line 54-253: Refactor the repeated kms-plugin-* container blocks into a
Go template range to eliminate duplication: create a slice (e.g., .Plugins or
.NumPlugins) and iterate (range) to render each container using a template body
that sets name as "kms-plugin-{{index}}" and socket paths like
/var/run/kmsplugin/kms-{{index}}. Keep shared fields (image / Image,
imagePullPolicy, securityContext, command, args, volumeMounts, softhsm-config
subPath) inside the templated block so one change updates all; update any
references to container names or sockets (e.g., kms-plugin-1..kms-plugin-10 and
-listen-addr=unix:///var/run/kmsplugin/kms-<index>.sock) to use the template
index variable.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 04563983-6d64-435c-b21a-0f1e31caf275
📒 Files selected for processing (18)
pkg/operator/encryption/controllers/helpers_test.gopkg/operator/encryption/controllers/key_controller.gopkg/operator/encryption/controllers/key_controller_test.gopkg/operator/encryption/controllers/state_controller.gopkg/operator/encryption/controllers/state_controller_test.gopkg/operator/encryption/deployer/unionrevisionedpod_test.gopkg/operator/encryption/encryptionconfig/config.gopkg/operator/encryption/encryptionconfig/config_test.gopkg/operator/encryption/encryptionconfig/secret.gopkg/operator/encryption/observer/observe_encryption_config_test.gopkg/operator/encryption/secrets/secrets.gopkg/operator/encryption/secrets/secrets_test.gopkg/operator/encryption/secrets/types.gopkg/operator/encryption/state/types.gopkg/operator/encryption/statemachine/transition_test.gopkg/operator/encryption/testing/helpers.gotest/e2e-encryption/encryption_test.gotest/library/encryption/kms/assets/k8s_mock_kms_plugin_daemonset.yaml
✅ Files skipped from review due to trivial changes (3)
- pkg/operator/encryption/controllers/helpers_test.go
- pkg/operator/encryption/deployer/unionrevisionedpod_test.go
- pkg/operator/encryption/statemachine/transition_test.go
🚧 Files skipped from review as they are similar to previous changes (10)
- pkg/operator/encryption/encryptionconfig/config_test.go
- pkg/operator/encryption/observer/observe_encryption_config_test.go
- pkg/operator/encryption/secrets/types.go
- pkg/operator/encryption/encryptionconfig/config.go
- pkg/operator/encryption/controllers/key_controller_test.go
- test/e2e-encryption/encryption_test.go
- pkg/operator/encryption/controllers/state_controller.go
- pkg/operator/encryption/encryptionconfig/secret.go
- pkg/operator/encryption/testing/helpers.go
- pkg/operator/encryption/controllers/key_controller.go
|
/retest |
|
@ardaguclu: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
2 participants
This is continuation of #2161 for the work that is described openshift/enhancements#1960
This PR introduces KMSProviderConfig which will be used as the internal representation of the API fields. This field will be serialized and carried from key_controller -> state_controller -> plugin lifecycle.
Summary by CodeRabbit
Bug Fixes
Features
Tests
Chores