[Feature Request]Add support for secret annotation so only annotated secrets trigger rolling update

[johnphilipscs]

I am using your controller to trigger our app to perform a rolling update when cert-manager renews a certificate. It is my preference to restrict the trigger to only occur on the particular certificate secret and not on changes to other secrets.

[github-actions]

This issue has been inactive for 60 days. If the issue is still relevant please comment to re-activate the issue. If no action is taken within 7 days, the issue will be marked closed.

[johnphilipscs]

Still interested in this feature if anyone is working on this project.

[jabdoa2]

I guess ideally this annotations would be on the deployment side instead of the secret side. Otherwise, you might disable it for multiple deployments at once. We could create some kind of whitelist or blacklist annotation on the deployment. What do you think?

[jabdoa2]

@toelke do we want to support this?

Options:

1. Support wave.pusher.com/update-on-config-change: "only-annotated" on Deployments. Add wave.pusher.com/update-deployment-on-config-change: "true" to Secrets/CMs.
2. Add a flag to wave to change the default behavior. Add wave.pusher.com/update-deployment-on-config-change: "true" to Secrets/CMs.
3. Add wave.pusher.com/ignore: "true" to Secrets/CMs to selectively ignore Secrets/CMs. (Kind of different from the request)
4. Decide not to implement this.

[shine17]

@toelke do we want to support this?

Options:

1. Support wave.pusher.com/update-on-config-change: "only-annotated" on Deployments. Add wave.pusher.com/update-deployment-on-config-change: "true" to Secrets/CMs.
2. Add a flag to wave to change the default behavior. Add wave.pusher.com/update-deployment-on-config-change: "true" to Secrets/CMs.
3. Add wave.pusher.com/ignore: "true" to Secrets/CMs to selectively ignore Secrets/CMs. (Kind of different from the request)
4. Decide not to implement this.

1

[toelke]

I also prefer 1.

[erhhung]

I am encountering a need for this feature on a StatefulSet (from the kube-prometheus-stack deployment) where there are lots of ConfigMaps and Secrets mounted into various containers.

In my scenario, the app already provides dynamic config reloading capabilities to minimize pod restarts, but I have elected to add an oauth2-proxy container into the StatefulSets, that, unfortunately for the time being, does not support reloading mounted TLS secrets updated by cert-manager.

So I would like to trigger a rolling update only for specific TLS secrets while letting the existing config reloader handle updates to other resources.

Proposal

I propose that we maintain full backward compatibility with the existing implementation:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    wave.pusher.com/update-on-config-change: "true"
    wave.pusher.com/extra-configmaps: "some-namespace/my-configmap,configmap-in-same-namespace"
    wave.pusher.com/extra-secrets: "some-namespace/my-secret,some-other-namespace/foo"

And introduce similarly named annotations on the deployment side to include or exclude resources in the watchlists:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    wave.pusher.com/update-on-config-change: "true"
    wave.pusher.com/include-configmaps: "some-namespace/watch-this-configmap-only"
    wave.pusher.com/include-secrets: "some-namespace/watch-this-secret-only,watch-secret-in-same-namespace"
    wave.pusher.com/exclude-configmaps: "some-namespace/but-not-this-configmap"
    wave.pusher.com/exclude-secrets: "but-not-this-secret-in-same-namespace"

If the wave.pusher.com/update-on-config-change annotation is present and "true", start with all mounted ConfigMaps and Secrets in the Deployment, etc., (exclude ones referenced by extra-configmaps and extra-secrets annotations) — this is the default behavior.

1. If wave.pusher.com/include-configmaps annotation is present, ignore all mounted ConfigMaps and Secrets and watch the specified ones instead
2. If wave.pusher.com/include-secrets annotation is present, ignore all mounted ConfigMaps and Secrets and watch the specified ones instead
3. Add resources specified by wave.pusher.com/extra-configmaps and wave.pusher.com/extra-secrets annotations to the watchlists
4. If wave.pusher.com/exclude-configmaps annotation is present, remove specified ConfigMaps from the watchlist
5. If wave.pusher.com/exclude-secrets annotation is present, remove specified Secrets from the watchlist

The reason why having either include-configmaps or include-secrets annotation ignore both mounted ConfigMaps and Secrets is so that it wouldn't require having wave.pusher.com/include-configmaps: "" (if that is even a valid annotation) if all I wanted to monitor is listed only in wave.pusher.com/include-secrets.

The new annotations would follow the same [namespace/]<resource-name> CSV format as extra-configmaps and extra-secrets.

Then, in my use case, I would only need two annotations to achieve my desired behavior, e.g.:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations:
    wave.pusher.com/update-on-config-change: "true"
    wave.pusher.com/include-secrets: oauth2-proxy-tls