[BUG] Background scan configuration via the ConfigMap

[chipzoller]

Software version numbers
State the version numbers of applications involved in the bug.

• Kubernetes version: 1.21
• Kubernetes platform (if applicable; ex., EKS, GKE, OpenShift): K3d
• Kyverno version: 1.4.2

Describe the bug
The documentation states there is a flag called background-scan which can be added to the Kyverno ConfigMap with a value in the format of (int)(unit) which controls the background scan interval which defaults to one hour. Setting background-scan to 5m appears to do nothing. Even after a rollout restart of the Kyverno Deployment, the new background scan interval is not picked up.

To Reproduce
Steps to reproduce the behavior:

1. Edit the Kyverno ConfigMap.
2. Add background-scan: 5m
3. Wait 5 minutes
4. Check the Kyverno log to see when the next background scan appears. It's expected to see a message like the following but it never occurs:

I0829 12:43:51.910137       1 report.go:48] PolicyController/forceReconciliation "msg"="performing the background scan"  "scan interval"="5m0s"

Expected behavior
Kyverno picks up the new background scan interval immediately after the new value is set/updated and adopts the custom interval. Kyverno prints a logged message acknowledging a new background scan interval has been detected. Background scans happen at that interval with appropriate logged messages.

Additional context
Related to #2332

[anushkamittal20]

Hey @chipzoller
These are what you get by default every hour

I0922 11:34:57.690549       1 registration.go:191] Register/UpdateWebhookConfigurations "msg"="successfully updated validatingWebhookConfigurations"  "name"="kyverno-resource-validating-webhook-cfg"
2021/09/22 12:25:49 http: TLS handshake error from 10.244.0.1:57213: EOF
I0922 12:34:52.665075       1 report.go:48] PolicyController/forceReconciliation "msg"="performing the background scan"  "scan interval"="1h0m0s"

and to test the changes that we are trying to debug I tried to add background-scan: 5m in these three places, https://github.com/kyverno/kyverno/blob/main/definitions/k8s-resource/configmap.yaml, https://github.com/kyverno/kyverno/blob/main/definitions/install.yaml#L6604 and https://github.com/kyverno/kyverno/blob/main/definitions/install_debug.yaml#L6453 to add with the data. I hope this is right and yes I see that it doesn't show a message. I will be working on debugging this now.

[chipzoller]

As long as this is added in the Kyverno ConfigMap, it should be picked up but it's not. So you're adding it correctly.

[realshuting]

@chipzoller - were you configuring background-scan via the ConfigMap or the Kyverno args? I don't think we can do that via ConfigMap or do you see it somewhere in the doc?

[chipzoller]

From the docs:

Flags

The following flags are used to control the behavior of Kyverno and must be set in the Kyverno ConfigMap.

excludeGroupRole: excludeGroupRole role expected string with comma-separated group role. It will exclude all the group role from the user request. Default we are using system:serviceaccounts:kube-system,system:nodes,system:kube-scheduler.
excludeUsername: excludeUsername expected string with comma-separated kubernetes username. In generate request if user enable Synchronize in generate policy then only kyverno can update/delete generated resource but admin can exclude specific username who have access of delete/update generated resource.
filterK8sResources(deprecated): Kubernetes resources in the format “[kind,namespace,name]” where the policy is not evaluated by the admission webhook. For example –filterKind “[Deployment, kyverno, kyverno]” –filterKind “[Deployment, kyverno, kyverno],[Events, *, *]”.
gen-workers: the number of workers for processing generate policies concurrently. Default is set to 10.
background-scan: the interval (like 30s, 15m, 12h) for background processing. Default is set to 1h.
generateSuccessEvents: specifies whether (true/false) to generate success events. Default is set to “false”.

[realshuting]

I must have missed this statement when adding the doc "...must be set in the Kyverno ConfigMap". From PR #1730 I don't see we support this configuration via ConfigMap.

@anushkamittal20 - can you please confirm if this works when passing as a flag? If needed, we can open a feature request for the ConfigMap configuration.

[anushkamittal20]

Hey @realshuting @chipzoller when I add in

- args:
        - --background-scan=5m

I get the scans happening every 5 minutes

I0923 09:56:32.427231       1 registration.go:191] Register/UpdateWebhookConfigurations "msg"="successfully updated validatingWebhookConfigurations"  "name"="kyverno-resource-validating-webhook-cfg"
I0923 10:01:26.786608       1 report.go:48] PolicyController/forceReconciliation "msg"="performing the background scan"  "scan interval"="5m0s"
I0923 10:06:26.786513       1 report.go:48] PolicyController/forceReconciliation "msg"="performing the background scan"  "scan interval"="5m0s"

[realshuting]

Great, thanks @anushkamittal20!

@chipzoller - do you think we need to support this configuration via the ConfigMap? Otherwise we'll need to update the doc.

[chipzoller]

It sounds like we, at the least, need to fix the documentation. I personally think it would be good to support this in a ConfigMap though.

[realshuting]

It sounds like we, at the least, need to fix the documentation. I personally think it would be good to support this in a ConfigMap though.

Ok, marked as "good first issue"!

[chipzoller]

Making sure, with plans to deprecate the ConfigMap, we're still considering this use case.

[realshuting]

Yes it's still valid. The deprecation is only with resourceFilters because of the dynamic webhook changes. We no longer need it.

[realshuting]

In 1.8.x, backgroundScan is changed to a bool to disable/enable the background scan process. And it was removed from the Configmap Flags.