Skip to content

Commit

Permalink
Add Service call and versioning (#843)
Browse files Browse the repository at this point in the history 
* update policy settings

Signed-off-by: Chip Zoller <chipzoller@gmail.com>

* add service calls

Signed-off-by: Chip Zoller <chipzoller@gmail.com>

* update sample

Signed-off-by: Chip Zoller <chipzoller@gmail.com>

* update versioning strategy

Signed-off-by: Chip Zoller <chipzoller@gmail.com>

* more clarification of `--backgroundScanWorkers`

Signed-off-by: Chip Zoller <chipzoller@gmail.com>

* add trim_prefix()

Signed-off-by: Chip Zoller <chipzoller@gmail.com>

* add jmesPath to verifyImages imageExtractors

Signed-off-by: Chip Zoller <chipzoller@gmail.com>

* to_boolean

Signed-off-by: Chip Zoller <chipzoller@gmail.com>

* update items() filter with support for arrays

Signed-off-by: Chip Zoller <chipzoller@gmail.com>

* update x509_decode() to mention CSR support

Signed-off-by: Chip Zoller <chipzoller@gmail.com>

* add image_normalize()

Signed-off-by: Chip Zoller <chipzoller@gmail.com>

* add sum()

Signed-off-by: Chip Zoller <chipzoller@gmail.com>

* fix, remove duplicate sum()

Signed-off-by: Chip Zoller <chipzoller@gmail.com>

* add gitops notes for mutate

Signed-off-by: Chip Zoller <chipzoller@gmail.com>

* move and rework background scan page

Signed-off-by: Chip Zoller <chipzoller@gmail.com>

* reswizzle Writing Policies sections, names, and descriptions

Signed-off-by: Chip Zoller <chipzoller@gmail.com>

* additions to policy exceptions

Signed-off-by: Chip Zoller <chipzoller@gmail.com>

* extend reporting docs

Signed-off-by: Chip Zoller <chipzoller@gmail.com>

* add operations, fix samples

Signed-off-by: Chip Zoller <chipzoller@gmail.com>

* note on report chunking

Signed-off-by: Chip Zoller <chipzoller@gmail.com>

* update all JSON patch examples to use YAML and not JSON

Signed-off-by: Chip Zoller <chipzoller@gmail.com>

* add extra note on --autoUpdateWebhooks

Signed-off-by: Chip Zoller <chipzoller@gmail.com>

---------

Signed-off-by: Chip Zoller <chipzoller@gmail.com>
  • Loading branch information
@chipzoller
chipzoller committed May 14, 2023
1 parent 9bf720d commit b911373
Show file tree
Hide file tree
Showing 21 changed files with 867 additions and 596 deletions.
4 changes: 2 additions & 2 deletions content/en/docs/Installation/customization.md
View file
Expand Up @@ -232,11 +232,11 @@ The following flags can be used to control the advanced behavior of the various
2. `admissionReports` (AR): enables the AdmissionReport resource which is created from validate rules in `Audit` mode. Used to factor into a final PolicyReport. Default is `true`.
3. `allowInsecureRegistry` (ABR): allows Kyverno to work with insecure registries (i.e., bypassing certificate checks) either with [verifyImages](/docs/writing-policies/verify-images/) rules or [variables from image registries](/docs/writing-policies/external-data-sources/#variables-from-image-registries). Only for testing purposes. Not to be used in production situations.
4. `alsologtostderr` (ABCR): log to standard error as well as files (no effect when -logtostderr=true)
5. `autoUpdateWebhooks` (A): set this flag to `false` to disable auto-configuration of the webhook. With this feature disabled, Kyverno creates a default webhook configuration (which matches ALL resources), therefore, webhooks configuration via the ConfigMap will be ignored. However, the user still can modify it by patching the webhook resource manually. Default is `true`.
5. `autoUpdateWebhooks` (A): set this flag to `false` to disable auto-configuration of the webhook. Default is `true`. With this feature disabled, Kyverno creates a default webhook configuration (which matches ALL resources), therefore, webhooks configuration via the ConfigMap will be ignored. However, the user still can modify it by patching the webhook resource manually. Setting this flag to `false` after it has been set to `true` will retain existing webhooks and automatic updates will cease. All further changes will be manual in nature. If the webhook or webhook configuration resource is deleted, it will be replaced by one matching on a wildcard.
6. `backgroundServiceAccountName` (A): the name of the background controller's ServiceAccount name allowing the admission controller to disregard any AdmissionReview requests coming from Kyverno itself. This may need to be removed in situations where, for example, Kyverno needs to mutate a resource it just generated. Default is set to the ServiceAccount for the background controller.
7. `backgroundScan` (R): enables/disables background reporting scans. Has no effect on background reconciliation by the background controller. `true` by default.
8. `backgroundScanInterval` (R): sets the time interval when periodic background scans for reporting take place. Default is `1h`. Supports minute durations as well (e.g., `10m`).
9. `backgroundScanWorkers` (R): defines the number of internal worker threads to use when processing background scan reports. Default is `2`.
9. `backgroundScanWorkers` (R): defines the number of internal worker threads to use when processing background scan reports. Default is `2`. More workers means faster report processing at the cost of more resources consumed. Since the reports controller uses leader election, all reports processing will only be done by a single replica at a time.
10. `clientRateLimitBurst` (ABCR): configures the maximum burst for throttling. Uses the client default if zero. Default is `300`.
11. `clientRateLimitQPS` (ABCR): configures the maximum QPS to the API server from Kyverno. Uses the client default if zero. Default is `300`.
12. `disableMetrics` (ABCR): specifies whether to enable exposing the metrics. Default is `false`.
Expand Down
2 changes: 2 additions & 0 deletions content/en/docs/Installation/platform-notes.md
View file
Expand Up @@ -45,6 +45,8 @@ spec:
- Replace=true
```

For considerations when using Argo CD along with Kyverno mutate policies, see the documentation [here](/docs/writing-policies/mutate/#argocd).

#### Ownership Clashes

ArgoCD automatically sets the `app.kubernetes.io/instance` label and uses it to determine which resources form the app. The Kyverno Helm chart also sets this label for the same purposes. In order to resolve this conflict, configure ArgoCD to use a different tracking mechanism as described in the ArgoCD [documentation](https://argo-cd.readthedocs.io/en/latest/user-guide/resource_tracking/#additional-tracking-methods-via-an-annotation).
Expand Down
276 changes: 9 additions & 267 deletions content/en/docs/Policy Reports/_index.md
View file
Expand Up @@ -5,7 +5,9 @@ description: >
weight: 60
---

Policy reports are Kubernetes Custom Resources, generated and managed automatically by Kyverno, which contain the results of applying matching Kubernetes resources to Kyverno ClusterPolicy or Policy resources. They are created for `validate` and `verifyImages` rules when the policy in which they are contained is configured with `spec.validationFailureAction: Audit` or `spec.background: true` and a resource applies to one or more rules according to the policy definition. If resources violate multiple rules, there will be multiple entries. When resources are deleted, their entry will be removed from the report. Reports, therefore, always represent the current state of the cluster and do not record historical information. For example, if a validate policy in `Audit` mode exists containing a single rule which requires that all resources set the label `team` and a user creates a Pod which does not set the `team` label, Kyverno will allow the Pod's creation but record it as a `fail` result in a policy report due to the Pod being in violation of the policy and rule. Policies configured with `spec.validationFailureAction: Enforce` immediately block violating resources and therefore do not generate policy reports. Policy reports are an ideal way to observe the impact a Kyverno policy may have in a cluster without causing disruption. The insights gained from these policy reports may be used to provide valuable feedback to both users/developers so they may take appropriate action to bring offending resources into alignment, and to policy authors or cluster operators to help them refine policies prior to changing them to `Enforce` mode. Because reports are decoupled from policies, standard Kubernetes RBAC can then be applied to separate those who can see and manipulate policies from those who can view reports.
Policy reports are Kubernetes Custom Resources, generated and managed automatically by Kyverno, which contain the results of applying matching Kubernetes resources to Kyverno ClusterPolicy or Policy resources. They are created for `validate` and `verifyImages` rules when a resource is matched by one or more rules according to the policy definition. If resources violate multiple rules, there will be multiple entries. When resources are deleted, their entry will be removed from the report. Reports, therefore, always represent the current state of the cluster and do not record historical information.

For example, if a validate policy in `Audit` mode exists containing a single rule which requires that all resources set the label `team` and a user creates a Pod which does not set the `team` label, Kyverno will allow the Pod's creation but record it as a `fail` result in a policy report due to the Pod being in violation of the policy and rule. Policies configured with `spec.validationFailureAction: Enforce` immediately block violating resources and results will only be reported for `pass` evaluations. Policy reports are an ideal way to observe the impact a Kyverno policy may have in a cluster without causing disruption. The insights gained from these policy reports may be used to provide valuable feedback to both users/developers so they may take appropriate action to bring offending resources into alignment, and to policy authors or cluster operators to help them refine policies prior to changing them to `Enforce` mode. Because reports are decoupled from policies, standard Kubernetes RBAC can then be applied to separate those who can see and manipulate policies from those who can view reports.

Policy reports are created based on two different triggers: an admission event (a `CREATE`, `UPDATE`, or `DELETE` action performed against a resource) or the result of a background scan discovering existing resources. Policy reports, like Kyverno policies, have both Namespaced and cluster-scoped variants; a `PolicyReport` is a Namespaced resource while a `ClusterPolicyReport` is a cluster-scoped resource. However, unlike `Policy` and `ClusterPolicy` resources, the `PolicyReport` and `ClusterPolicyReport` resources contain results from resources which are at the same scope and _not_ what is determined by the Kyverno policy. For example, a `ClusterPolicy` (a cluster-scoped policy) contains a rule which matches on Pods (a Namespaced resource). Results generated from this policy and rule are written to a `PolicyReport` in the Namespace where the Pod exists.

Expand Down Expand Up @@ -53,17 +55,19 @@ Policy reports show policy results for current resources in the cluster only. Fo
Policy reports have a few configuration options available. For details, see the [container flags](/docs/installation/customization/#container-flags) section.

{{% alert title="Note" color="warning" %}}
Policy reports created from background scans are not subject to the configuration of either a [resource filter](/docs/installation/customization/#resource-filters) or a [Namespace selector](/docs/installation/customization/#namespace-selectors) defined in the [Kyverno ConfigMap](/docs/installation/customization/#configmap-keys). This is due to the fact that the overhead required to generate these reports is minimal and should have no performance impact.
Policy reports created from background scans are not subject to the configuration of a [Namespace selector](/docs/installation/customization/#namespace-selectors) defined in the [Kyverno ConfigMap](/docs/installation/customization/#configmap-keys).
{{% /alert %}}

Reports are automatically chunked in increments of 1,000 results. Once a single report reaches this number of results, a second report with an incremented resource name will be created.

## Report result logic

Entries in a policy report contain a `result` field which can be either `pass`, `skip`, `warn`, `error`, or `fail`.

| Result | Description |
|--------|------------------------------------------------------------------------------------------------------------------------------------|
| pass | The resource was applicable to a rule and the pattern passed evaluation. |
| skip | Preconditions were not satisfied (if applicable) in a rule and so further processing was not performed. |
| skip | Preconditions were not satisfied (if applicable) in a rule, or an applicable PolicyException exists and so further processing was not performed. |
| fail | The resource failed the pattern evaluation. |
| warn | The annotation `policies.kyverno.io/scored` has been set to `"false"` in the policy converting otherwise `fail` results to `warn`. |
| error | Variable substitution failed outside of preconditions and elsewhere in the rule (ex., in the pattern). |
Expand Down Expand Up @@ -144,270 +148,6 @@ timestamp:
seconds: 1.666095335e+09
```

## Example: Trigger a PolicyReport

By default, a `PolicyReport` object (Namespaced) is created in the same Namespace where resources apply to one or more Kyverno policies, be they `Policy` or `ClusterPolicy` policies.

A single Kyverno ClusterPolicy exists with a single rule which ensures Pods cannot mount Secrets as environment variables.

```yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: secrets-not-from-env-vars
spec:
background: true
validationFailureAction: Audit
rules:
- name: secrets-not-from-env-vars
match:
any:
- resources:
kinds:
- Pod
validate:
message: "Secrets must be mounted as volumes, not as environment variables."
pattern:
spec:
containers:
- name: "*"
=(env):
- =(valueFrom):
X(secretKeyRef): "null"
```

Creating a Pod in this Namespace which does not use any Secrets (and thereby does not violate the `secrets-not-from-env-vars` rule in the ClusterPolicy) will generate the first entry in the PolicyReport, but listed as a `PASS`.

```sh
$ kubectl run busybox --image busybox:1.28 -- sleep 9999
pod/busybox created

$ kubectl get po
NAME READY STATUS RESTARTS AGE
busybox 1/1 Running 0 66s

$ kubectl get polr
NAME PASS FAIL WARN ERROR SKIP AGE
cpol-secrets-not-from-env-vars 1 0 0 0 0 6s
```

Inspect the PolicyReport in the `default` Namespace to view its contents. Notice that the `busybox` Pod is listed as having passed.

```sh
$ kubectl get polr cpol-secrets-not-from-env-vars -o yaml

<snipped>
results:
- message: validation rule 'secrets-not-from-env-vars' passed.
policy: secrets-not-from-env-vars
resources:
- apiVersion: v1
kind: Pod
name: busybox
namespace: default
uid: 0dd94825-cc6e-435b-982b-fb76ac2fdc2a
result: pass
rule: secrets-not-from-env-vars
scored: true
source: kyverno
timestamp:
nanos: 0
seconds: 1666097147
summary:
error: 0
fail: 0
pass: 1
skip: 0
warn: 0
```

Create another Pod which violates the rule in the sample policy. Because the rule is written with `validationFailureAction: Audit`, resources are allowed to be created which violate the rule. If this occurs, another entry will be created in the PolicyReport which denotes this condition as a FAIL. By contrast, if `validationFailureAction: Enforce` and an offending resource was attempted creation, it would be immediately blocked and therefore would not generate another entry in the report.

```yaml
apiVersion: v1
kind: Pod
metadata:
name: secret-pod
spec:
containers:
- name: busybox
image: busybox:1.28
env:
- name: SECRET_STUFF
valueFrom:
secretKeyRef:
name: mysecret
key: mysecretname
```

Since the above Pod spec was allowed and it violated the rule, there should now be a failure entry in the PolicyReport in the `default` Namespace.

```sh
$ kubectl get polr polr-ns-default -o yaml

<snipped>
- message: 'validation error: Secrets must be mounted as volumes, not as environment
variables. rule secrets-not-from-env-vars failed at path /spec/containers/0/env/0/valueFrom/secretKeyRef/'
policy: secrets-not-from-env-vars
resources:
- apiVersion: v1
kind: Pod
name: secret-pod
namespace: default
uid: 72a7422c-fb6f-486f-b274-1ca0de55d49d
result: fail
rule: secrets-not-from-env-vars
scored: true
source: kyverno
timestamp:
nanos: 0
seconds: 1666098438
summary:
error: 0
fail: 1
pass: 1
skip: 0
warn: 0
```

Lastly, delete the Pod called `secret-pod` and once again check the PolicyReport object.

```sh
$ kubectl delete po secret-pod
pod "secret-pod" deleted

$ kubectl get polr cpol-secrets-not-from-env-vars
NAME PASS FAIL WARN ERROR SKIP AGE
cpol-secrets-not-from-env-vars 1 0 0 0 0 2m21s
```

Notice how the PolicyReport has removed the previously-failed entry when the violating Pod was deleted.

## Example: Trigger a ClusterPolicyReport

A ClusterPolicyReport is the same concept as a PolicyReport only it contains resources which are cluster scoped rather than Namespaced.

As an example, create the following sample ClusterPolicy containing a single rule which validates that all new Namespaces should contain the label called `thisshouldntexist` and have some value. Notice how `validationFailureAction: Audit` and `background: true` in this ClusterPolicy.

```yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-ns-labels
spec:
validationFailureAction: Audit
background: true
rules:
- name: check-for-labels-on-namespace
match:
any:
- resources:
kinds:
- Namespace
validate:
message: "The label `thisshouldntexist` is required."
pattern:
metadata:
labels:
thisshouldntexist: "?*"
```

After creating this sample ClusterPolicy, check for the existence of a ClusterPolicyReport object.

```sh
$ kubectl get cpolr
NAME PASS FAIL WARN ERROR SKIP AGE
cpol-require-ns-labels 0 5 0 0 0 76m
```

Notice that a ClusterPolicyReport named `cpol-require-ns-labels` exists with five failures.

The ClusterPolicyReport, when inspected, has the same structure as the PolicyReport object and contains entries in the `results` and `summary` objects with the outcomes of a policy audit.

```yaml
results:
- message: 'validation error: The label `thisshouldntexist` is required. rule check-for-labels-on-namespace
failed at path /metadata/labels/thisshouldntexist/'
policy: require-ns-labels
resources:
- apiVersion: v1
kind: Namespace
name: kube-node-lease
uid: 06e5056f-76a3-461a-8d45-2793b8bd5bbc
result: fail
rule: check-for-labels-on-namespace
scored: true
source: kyverno
timestamp:
nanos: 0
seconds: 1666098654
- message: 'validation error: The label `thisshouldntexist` is required. rule check-for-labels-on-namespace
failed at path /metadata/labels/thisshouldntexist/'
policy: require-ns-labels
resources:
- apiVersion: v1
kind: Namespace
name: default
uid: 4ffe22fd-0927-4ed1-8b04-50ca7ed58626
result: fail
rule: check-for-labels-on-namespace
scored: true
source: kyverno
timestamp:
nanos: 0
seconds: 1666098654
- message: 'validation error: The label `thisshouldntexist` is required. rule check-for-labels-on-namespace
failed at path /metadata/labels/thisshouldntexist/'
policy: require-ns-labels
resources:
- apiVersion: v1
kind: Namespace
name: kyverno
uid: 5d87cd66-ce30-4abc-b863-f3b97715a5f1
result: fail
rule: check-for-labels-on-namespace
scored: true
source: kyverno
timestamp:
nanos: 0
seconds: 1666098654
- message: 'validation error: The label `thisshouldntexist` is required. rule check-for-labels-on-namespace
failed at path /metadata/labels/thisshouldntexist/'
policy: require-ns-labels
resources:
- apiVersion: v1
kind: Namespace
name: kube-public
uid: c077ee71-435b-4921-9e05-8751fee71b64
result: fail
rule: check-for-labels-on-namespace
scored: true
source: kyverno
timestamp:
nanos: 0
seconds: 1666098654
- message: 'validation error: The label `thisshouldntexist` is required. rule check-for-labels-on-namespace
failed at path /metadata/labels/thisshouldntexist/'
policy: require-ns-labels
resources:
- apiVersion: v1
kind: Namespace
name: kube-system
uid: e63fabde-b572-4b07-b899-b2230f4eac69
result: fail
rule: check-for-labels-on-namespace
scored: true
source: kyverno
timestamp:
nanos: 0
seconds: 1666098654
summary:
error: 0
fail: 5
pass: 0
skip: 0
warn: 0
```

## Report internals

Expand Down Expand Up @@ -453,3 +193,5 @@ spec:
```

These intermediary resources have the same basic contents as a policy report and are used internally by Kyverno to build the final policy report. Kyverno will merge these results automatically into the appropriate policy report and there is no manual interaction typically required.

For more details on the internal reporting processes, see the developer docs [here](https://github.com/kyverno/kyverno/tree/main/docs/dev/reports).

0 comments on commit b911373

Please sign in to comment.