Skip to content

fix(charts): replace copy-host-secret-artifact-s3 broken auto sync with more reliable manual sync#1215

Merged
davehadley merged 1 commit intomainfrom
drh/disable-automatic-syncing-of-s3-secret
Mar 24, 2026
Merged

fix(charts): replace copy-host-secret-artifact-s3 broken auto sync with more reliable manual sync#1215
davehadley merged 1 commit intomainfrom
drh/disable-automatic-syncing-of-s3-secret

Conversation

@davehadley
Copy link
Collaborator

No description provided.

@davehadley davehadley force-pushed the drh/disable-automatic-syncing-of-s3-secret branch from f9d1bdb to 8ac855c Compare March 23, 2026 13:38
@davehadley davehadley force-pushed the drh/disable-automatic-syncing-of-s3-secret branch from 8ac855c to a24a4e3 Compare March 23, 2026 13:52
@davehadley davehadley force-pushed the drh/disable-automatic-syncing-of-s3-secret branch from a24a4e3 to 59824b3 Compare March 23, 2026 14:35
@davehadley
Copy link
Collaborator Author

This PR provides a temporary solution for this issue:
https://diamondlightsource.slack.com/archives/C06N43M7JP3/p1762939672661569
https://kubernetes.slack.com/archives/CLGR9BJU9/p1762854995411249

Kyverno sync stops working when there are hundred or thousands of objects matching the update due to the update requests becoming too large for etcd and generating errors like:

2025-11-11T08:13:13Z ERR github.com/kyverno/kyverno/pkg/policy/policy_controller.go:431 > Unhandled Error error="failed to create UR on generating policy events copy-host-secret-artifact-s3: Request entity too large: limit is 3145728" logger=UnhandledError

This issue is confirmed to still exist in kyverno 1.17.1.

This PR keeps the automatic copy of the secret on namespace generation but disables sync. That means that when the s3 secret is changed, it will need to be manually synced by running:

kubectl create job --from=cronjob/copy-host-secret-artifact-s3 -nworkflows copy-host-secret-artifact-s3

We can revert to automatic syncing when this issue is fixed upstream in kyverno by batching large sync requests.

@davehadley davehadley marked this pull request as ready for review March 24, 2026 08:27
@davehadley davehadley merged commit e4b7ea7 into main Mar 24, 2026
49 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants