From b91137378760ec9bf2d7b16c722d2c287f93af3c Mon Sep 17 00:00:00 2001 From: Chip Zoller Date: Sun, 14 May 2023 17:48:58 -0400 Subject: [PATCH] Add Service call and versioning (#843) * update policy settings Signed-off-by: Chip Zoller * add service calls Signed-off-by: Chip Zoller * update sample Signed-off-by: Chip Zoller * update versioning strategy Signed-off-by: Chip Zoller * more clarification of `--backgroundScanWorkers` Signed-off-by: Chip Zoller * add trim_prefix() Signed-off-by: Chip Zoller * add jmesPath to verifyImages imageExtractors Signed-off-by: Chip Zoller * to_boolean Signed-off-by: Chip Zoller * update items() filter with support for arrays Signed-off-by: Chip Zoller * update x509_decode() to mention CSR support Signed-off-by: Chip Zoller * add image_normalize() Signed-off-by: Chip Zoller * add sum() Signed-off-by: Chip Zoller * fix, remove duplicate sum() Signed-off-by: Chip Zoller * add gitops notes for mutate Signed-off-by: Chip Zoller * move and rework background scan page Signed-off-by: Chip Zoller * reswizzle Writing Policies sections, names, and descriptions Signed-off-by: Chip Zoller * additions to policy exceptions Signed-off-by: Chip Zoller * extend reporting docs Signed-off-by: Chip Zoller * add operations, fix samples Signed-off-by: Chip Zoller * note on report chunking Signed-off-by: Chip Zoller * update all JSON patch examples to use YAML and not JSON Signed-off-by: Chip Zoller * add extra note on --autoUpdateWebhooks Signed-off-by: Chip Zoller --------- Signed-off-by: Chip Zoller --- content/en/docs/Installation/customization.md | 4 +- .../en/docs/Installation/platform-notes.md | 2 + content/en/docs/Policy Reports/_index.md | 276 +------------ content/en/docs/Policy Reports/background.md | 41 ++ content/en/docs/Policy Reports/examples.md | 271 +++++++++++++ content/en/docs/Releases/_index.md | 6 +- content/en/docs/Writing policies/autogen.md | 6 +- .../en/docs/Writing policies/background.md | 41 -- content/en/docs/Writing policies/cleanup.md | 7 +- .../en/docs/Writing policies/exceptions.md | 15 +- .../Writing policies/external-data-sources.md | 209 +++++++--- content/en/docs/Writing policies/generate.md | 7 +- content/en/docs/Writing policies/jmespath.md | 364 +++++++++++------- .../en/docs/Writing policies/match-exclude.md | 67 ++-- content/en/docs/Writing policies/mutate.md | 67 +++- .../docs/Writing policies/policy-settings.md | 17 +- .../en/docs/Writing policies/preconditions.md | 4 +- content/en/docs/Writing policies/tips.md | 5 +- content/en/docs/Writing policies/validate.md | 7 +- content/en/docs/Writing policies/variables.md | 4 +- .../en/docs/Writing policies/verify-images.md | 43 ++- 21 files changed, 867 insertions(+), 596 deletions(-) create mode 100644 content/en/docs/Policy Reports/background.md create mode 100644 content/en/docs/Policy Reports/examples.md delete mode 100644 content/en/docs/Writing policies/background.md diff --git a/content/en/docs/Installation/customization.md b/content/en/docs/Installation/customization.md index 424bb209c..4b11a0338 100644 --- a/content/en/docs/Installation/customization.md +++ b/content/en/docs/Installation/customization.md @@ -232,11 +232,11 @@ The following flags can be used to control the advanced behavior of the various 2. `admissionReports` (AR): enables the AdmissionReport resource which is created from validate rules in `Audit` mode. Used to factor into a final PolicyReport. Default is `true`. 3. `allowInsecureRegistry` (ABR): allows Kyverno to work with insecure registries (i.e., bypassing certificate checks) either with [verifyImages](/docs/writing-policies/verify-images/) rules or [variables from image registries](/docs/writing-policies/external-data-sources/#variables-from-image-registries). Only for testing purposes. Not to be used in production situations. 4. `alsologtostderr` (ABCR): log to standard error as well as files (no effect when -logtostderr=true) -5. `autoUpdateWebhooks` (A): set this flag to `false` to disable auto-configuration of the webhook. With this feature disabled, Kyverno creates a default webhook configuration (which matches ALL resources), therefore, webhooks configuration via the ConfigMap will be ignored. However, the user still can modify it by patching the webhook resource manually. Default is `true`. +5. `autoUpdateWebhooks` (A): set this flag to `false` to disable auto-configuration of the webhook. Default is `true`. With this feature disabled, Kyverno creates a default webhook configuration (which matches ALL resources), therefore, webhooks configuration via the ConfigMap will be ignored. However, the user still can modify it by patching the webhook resource manually. Setting this flag to `false` after it has been set to `true` will retain existing webhooks and automatic updates will cease. All further changes will be manual in nature. If the webhook or webhook configuration resource is deleted, it will be replaced by one matching on a wildcard. 6. `backgroundServiceAccountName` (A): the name of the background controller's ServiceAccount name allowing the admission controller to disregard any AdmissionReview requests coming from Kyverno itself. This may need to be removed in situations where, for example, Kyverno needs to mutate a resource it just generated. Default is set to the ServiceAccount for the background controller. 7. `backgroundScan` (R): enables/disables background reporting scans. Has no effect on background reconciliation by the background controller. `true` by default. 8. `backgroundScanInterval` (R): sets the time interval when periodic background scans for reporting take place. Default is `1h`. Supports minute durations as well (e.g., `10m`). -9. `backgroundScanWorkers` (R): defines the number of internal worker threads to use when processing background scan reports. Default is `2`. +9. `backgroundScanWorkers` (R): defines the number of internal worker threads to use when processing background scan reports. Default is `2`. More workers means faster report processing at the cost of more resources consumed. Since the reports controller uses leader election, all reports processing will only be done by a single replica at a time. 10. `clientRateLimitBurst` (ABCR): configures the maximum burst for throttling. Uses the client default if zero. Default is `300`. 11. `clientRateLimitQPS` (ABCR): configures the maximum QPS to the API server from Kyverno. Uses the client default if zero. Default is `300`. 12. `disableMetrics` (ABCR): specifies whether to enable exposing the metrics. Default is `false`. diff --git a/content/en/docs/Installation/platform-notes.md b/content/en/docs/Installation/platform-notes.md index 623243932..74aed372b 100644 --- a/content/en/docs/Installation/platform-notes.md +++ b/content/en/docs/Installation/platform-notes.md @@ -45,6 +45,8 @@ spec: - Replace=true ``` +For considerations when using Argo CD along with Kyverno mutate policies, see the documentation [here](/docs/writing-policies/mutate/#argocd). + #### Ownership Clashes ArgoCD automatically sets the `app.kubernetes.io/instance` label and uses it to determine which resources form the app. The Kyverno Helm chart also sets this label for the same purposes. In order to resolve this conflict, configure ArgoCD to use a different tracking mechanism as described in the ArgoCD [documentation](https://argo-cd.readthedocs.io/en/latest/user-guide/resource_tracking/#additional-tracking-methods-via-an-annotation). diff --git a/content/en/docs/Policy Reports/_index.md b/content/en/docs/Policy Reports/_index.md index 14a1dbbce..c0106c869 100644 --- a/content/en/docs/Policy Reports/_index.md +++ b/content/en/docs/Policy Reports/_index.md @@ -5,7 +5,9 @@ description: > weight: 60 --- -Policy reports are Kubernetes Custom Resources, generated and managed automatically by Kyverno, which contain the results of applying matching Kubernetes resources to Kyverno ClusterPolicy or Policy resources. They are created for `validate` and `verifyImages` rules when the policy in which they are contained is configured with `spec.validationFailureAction: Audit` or `spec.background: true` and a resource applies to one or more rules according to the policy definition. If resources violate multiple rules, there will be multiple entries. When resources are deleted, their entry will be removed from the report. Reports, therefore, always represent the current state of the cluster and do not record historical information. For example, if a validate policy in `Audit` mode exists containing a single rule which requires that all resources set the label `team` and a user creates a Pod which does not set the `team` label, Kyverno will allow the Pod's creation but record it as a `fail` result in a policy report due to the Pod being in violation of the policy and rule. Policies configured with `spec.validationFailureAction: Enforce` immediately block violating resources and therefore do not generate policy reports. Policy reports are an ideal way to observe the impact a Kyverno policy may have in a cluster without causing disruption. The insights gained from these policy reports may be used to provide valuable feedback to both users/developers so they may take appropriate action to bring offending resources into alignment, and to policy authors or cluster operators to help them refine policies prior to changing them to `Enforce` mode. Because reports are decoupled from policies, standard Kubernetes RBAC can then be applied to separate those who can see and manipulate policies from those who can view reports. +Policy reports are Kubernetes Custom Resources, generated and managed automatically by Kyverno, which contain the results of applying matching Kubernetes resources to Kyverno ClusterPolicy or Policy resources. They are created for `validate` and `verifyImages` rules when a resource is matched by one or more rules according to the policy definition. If resources violate multiple rules, there will be multiple entries. When resources are deleted, their entry will be removed from the report. Reports, therefore, always represent the current state of the cluster and do not record historical information. + +For example, if a validate policy in `Audit` mode exists containing a single rule which requires that all resources set the label `team` and a user creates a Pod which does not set the `team` label, Kyverno will allow the Pod's creation but record it as a `fail` result in a policy report due to the Pod being in violation of the policy and rule. Policies configured with `spec.validationFailureAction: Enforce` immediately block violating resources and results will only be reported for `pass` evaluations. Policy reports are an ideal way to observe the impact a Kyverno policy may have in a cluster without causing disruption. The insights gained from these policy reports may be used to provide valuable feedback to both users/developers so they may take appropriate action to bring offending resources into alignment, and to policy authors or cluster operators to help them refine policies prior to changing them to `Enforce` mode. Because reports are decoupled from policies, standard Kubernetes RBAC can then be applied to separate those who can see and manipulate policies from those who can view reports. Policy reports are created based on two different triggers: an admission event (a `CREATE`, `UPDATE`, or `DELETE` action performed against a resource) or the result of a background scan discovering existing resources. Policy reports, like Kyverno policies, have both Namespaced and cluster-scoped variants; a `PolicyReport` is a Namespaced resource while a `ClusterPolicyReport` is a cluster-scoped resource. However, unlike `Policy` and `ClusterPolicy` resources, the `PolicyReport` and `ClusterPolicyReport` resources contain results from resources which are at the same scope and _not_ what is determined by the Kyverno policy. For example, a `ClusterPolicy` (a cluster-scoped policy) contains a rule which matches on Pods (a Namespaced resource). Results generated from this policy and rule are written to a `PolicyReport` in the Namespace where the Pod exists. @@ -53,9 +55,11 @@ Policy reports show policy results for current resources in the cluster only. Fo Policy reports have a few configuration options available. For details, see the [container flags](/docs/installation/customization/#container-flags) section. {{% alert title="Note" color="warning" %}} -Policy reports created from background scans are not subject to the configuration of either a [resource filter](/docs/installation/customization/#resource-filters) or a [Namespace selector](/docs/installation/customization/#namespace-selectors) defined in the [Kyverno ConfigMap](/docs/installation/customization/#configmap-keys). This is due to the fact that the overhead required to generate these reports is minimal and should have no performance impact. +Policy reports created from background scans are not subject to the configuration of a [Namespace selector](/docs/installation/customization/#namespace-selectors) defined in the [Kyverno ConfigMap](/docs/installation/customization/#configmap-keys). {{% /alert %}} +Reports are automatically chunked in increments of 1,000 results. Once a single report reaches this number of results, a second report with an incremented resource name will be created. + ## Report result logic Entries in a policy report contain a `result` field which can be either `pass`, `skip`, `warn`, `error`, or `fail`. @@ -63,7 +67,7 @@ Entries in a policy report contain a `result` field which can be either `pass`, | Result | Description | |--------|------------------------------------------------------------------------------------------------------------------------------------| | pass | The resource was applicable to a rule and the pattern passed evaluation. | -| skip | Preconditions were not satisfied (if applicable) in a rule and so further processing was not performed. | +| skip | Preconditions were not satisfied (if applicable) in a rule, or an applicable PolicyException exists and so further processing was not performed. | | fail | The resource failed the pattern evaluation. | | warn | The annotation `policies.kyverno.io/scored` has been set to `"false"` in the policy converting otherwise `fail` results to `warn`. | | error | Variable substitution failed outside of preconditions and elsewhere in the rule (ex., in the pattern). | @@ -144,270 +148,6 @@ timestamp: seconds: 1.666095335e+09 ``` -## Example: Trigger a PolicyReport - -By default, a `PolicyReport` object (Namespaced) is created in the same Namespace where resources apply to one or more Kyverno policies, be they `Policy` or `ClusterPolicy` policies. - -A single Kyverno ClusterPolicy exists with a single rule which ensures Pods cannot mount Secrets as environment variables. - -```yaml -apiVersion: kyverno.io/v1 -kind: ClusterPolicy -metadata: - name: secrets-not-from-env-vars -spec: - background: true - validationFailureAction: Audit - rules: - - name: secrets-not-from-env-vars - match: - any: - - resources: - kinds: - - Pod - validate: - message: "Secrets must be mounted as volumes, not as environment variables." - pattern: - spec: - containers: - - name: "*" - =(env): - - =(valueFrom): - X(secretKeyRef): "null" -``` - -Creating a Pod in this Namespace which does not use any Secrets (and thereby does not violate the `secrets-not-from-env-vars` rule in the ClusterPolicy) will generate the first entry in the PolicyReport, but listed as a `PASS`. - -```sh -$ kubectl run busybox --image busybox:1.28 -- sleep 9999 -pod/busybox created - -$ kubectl get po -NAME READY STATUS RESTARTS AGE -busybox 1/1 Running 0 66s - -$ kubectl get polr -NAME PASS FAIL WARN ERROR SKIP AGE -cpol-secrets-not-from-env-vars 1 0 0 0 0 6s -``` - -Inspect the PolicyReport in the `default` Namespace to view its contents. Notice that the `busybox` Pod is listed as having passed. - -```sh -$ kubectl get polr cpol-secrets-not-from-env-vars -o yaml - - -results: -- message: validation rule 'secrets-not-from-env-vars' passed. - policy: secrets-not-from-env-vars - resources: - - apiVersion: v1 - kind: Pod - name: busybox - namespace: default - uid: 0dd94825-cc6e-435b-982b-fb76ac2fdc2a - result: pass - rule: secrets-not-from-env-vars - scored: true - source: kyverno - timestamp: - nanos: 0 - seconds: 1666097147 -summary: - error: 0 - fail: 0 - pass: 1 - skip: 0 - warn: 0 -``` - -Create another Pod which violates the rule in the sample policy. Because the rule is written with `validationFailureAction: Audit`, resources are allowed to be created which violate the rule. If this occurs, another entry will be created in the PolicyReport which denotes this condition as a FAIL. By contrast, if `validationFailureAction: Enforce` and an offending resource was attempted creation, it would be immediately blocked and therefore would not generate another entry in the report. - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: secret-pod -spec: - containers: - - name: busybox - image: busybox:1.28 - env: - - name: SECRET_STUFF - valueFrom: - secretKeyRef: - name: mysecret - key: mysecretname -``` - -Since the above Pod spec was allowed and it violated the rule, there should now be a failure entry in the PolicyReport in the `default` Namespace. - -```sh -$ kubectl get polr polr-ns-default -o yaml - - -- message: 'validation error: Secrets must be mounted as volumes, not as environment - variables. rule secrets-not-from-env-vars failed at path /spec/containers/0/env/0/valueFrom/secretKeyRef/' - policy: secrets-not-from-env-vars - resources: - - apiVersion: v1 - kind: Pod - name: secret-pod - namespace: default - uid: 72a7422c-fb6f-486f-b274-1ca0de55d49d - result: fail - rule: secrets-not-from-env-vars - scored: true - source: kyverno - timestamp: - nanos: 0 - seconds: 1666098438 -summary: - error: 0 - fail: 1 - pass: 1 - skip: 0 - warn: 0 -``` - -Lastly, delete the Pod called `secret-pod` and once again check the PolicyReport object. - -```sh -$ kubectl delete po secret-pod -pod "secret-pod" deleted - -$ kubectl get polr cpol-secrets-not-from-env-vars -NAME PASS FAIL WARN ERROR SKIP AGE -cpol-secrets-not-from-env-vars 1 0 0 0 0 2m21s -``` - -Notice how the PolicyReport has removed the previously-failed entry when the violating Pod was deleted. - -## Example: Trigger a ClusterPolicyReport - -A ClusterPolicyReport is the same concept as a PolicyReport only it contains resources which are cluster scoped rather than Namespaced. - -As an example, create the following sample ClusterPolicy containing a single rule which validates that all new Namespaces should contain the label called `thisshouldntexist` and have some value. Notice how `validationFailureAction: Audit` and `background: true` in this ClusterPolicy. - -```yaml -apiVersion: kyverno.io/v1 -kind: ClusterPolicy -metadata: - name: require-ns-labels -spec: - validationFailureAction: Audit - background: true - rules: - - name: check-for-labels-on-namespace - match: - any: - - resources: - kinds: - - Namespace - validate: - message: "The label `thisshouldntexist` is required." - pattern: - metadata: - labels: - thisshouldntexist: "?*" -``` - -After creating this sample ClusterPolicy, check for the existence of a ClusterPolicyReport object. - -```sh -$ kubectl get cpolr -NAME PASS FAIL WARN ERROR SKIP AGE -cpol-require-ns-labels 0 5 0 0 0 76m -``` - -Notice that a ClusterPolicyReport named `cpol-require-ns-labels` exists with five failures. - -The ClusterPolicyReport, when inspected, has the same structure as the PolicyReport object and contains entries in the `results` and `summary` objects with the outcomes of a policy audit. - -```yaml -results: -- message: 'validation error: The label `thisshouldntexist` is required. rule check-for-labels-on-namespace - failed at path /metadata/labels/thisshouldntexist/' - policy: require-ns-labels - resources: - - apiVersion: v1 - kind: Namespace - name: kube-node-lease - uid: 06e5056f-76a3-461a-8d45-2793b8bd5bbc - result: fail - rule: check-for-labels-on-namespace - scored: true - source: kyverno - timestamp: - nanos: 0 - seconds: 1666098654 -- message: 'validation error: The label `thisshouldntexist` is required. rule check-for-labels-on-namespace - failed at path /metadata/labels/thisshouldntexist/' - policy: require-ns-labels - resources: - - apiVersion: v1 - kind: Namespace - name: default - uid: 4ffe22fd-0927-4ed1-8b04-50ca7ed58626 - result: fail - rule: check-for-labels-on-namespace - scored: true - source: kyverno - timestamp: - nanos: 0 - seconds: 1666098654 -- message: 'validation error: The label `thisshouldntexist` is required. rule check-for-labels-on-namespace - failed at path /metadata/labels/thisshouldntexist/' - policy: require-ns-labels - resources: - - apiVersion: v1 - kind: Namespace - name: kyverno - uid: 5d87cd66-ce30-4abc-b863-f3b97715a5f1 - result: fail - rule: check-for-labels-on-namespace - scored: true - source: kyverno - timestamp: - nanos: 0 - seconds: 1666098654 -- message: 'validation error: The label `thisshouldntexist` is required. rule check-for-labels-on-namespace - failed at path /metadata/labels/thisshouldntexist/' - policy: require-ns-labels - resources: - - apiVersion: v1 - kind: Namespace - name: kube-public - uid: c077ee71-435b-4921-9e05-8751fee71b64 - result: fail - rule: check-for-labels-on-namespace - scored: true - source: kyverno - timestamp: - nanos: 0 - seconds: 1666098654 -- message: 'validation error: The label `thisshouldntexist` is required. rule check-for-labels-on-namespace - failed at path /metadata/labels/thisshouldntexist/' - policy: require-ns-labels - resources: - - apiVersion: v1 - kind: Namespace - name: kube-system - uid: e63fabde-b572-4b07-b899-b2230f4eac69 - result: fail - rule: check-for-labels-on-namespace - scored: true - source: kyverno - timestamp: - nanos: 0 - seconds: 1666098654 -summary: - error: 0 - fail: 5 - pass: 0 - skip: 0 - warn: 0 -``` ## Report internals @@ -453,3 +193,5 @@ spec: ``` These intermediary resources have the same basic contents as a policy report and are used internally by Kyverno to build the final policy report. Kyverno will merge these results automatically into the appropriate policy report and there is no manual interaction typically required. + +For more details on the internal reporting processes, see the developer docs [here](https://github.com/kyverno/kyverno/tree/main/docs/dev/reports). diff --git a/content/en/docs/Policy Reports/background.md b/content/en/docs/Policy Reports/background.md new file mode 100644 index 000000000..c6e586d49 --- /dev/null +++ b/content/en/docs/Policy Reports/background.md @@ -0,0 +1,41 @@ +--- +title: Background Scans +description: > + Periodically reapply policies to existing resources for reporting. +weight: 15 +--- + +Kyverno can validate existing resources in the cluster that may have been created before a policy was created. This can be useful when evaluating the potential effects some new policies will have on a cluster prior to changing them to `Enforce` mode. The application of policies to existing resources is referred to as **background scanning** and is enabled by default unless `spc.background` is set to `false` in a policy like shown below in the snippet. + +```yaml +spec: + background: false + rules: + - name: default-deny-ingress +``` + +{{% alert title="Note" color="info" %}} +Background scans are handled by the reports controller and not the background controller. +{{% /alert %}} + +Background scanning, enabled by default in a `Policy` or `ClusterPolicy` object with the `spec.background` field, allows Kyverno to periodically scan existing resources and find if they match any `validate` or `verifyImages` rules. If existing resources are found which would violate an existing policy, the background scan notes them in a `ClusterPolicyReport` or a `PolicyReport` object, depending on if the resource is namespaced or not. It does not block any existing resources that match a rule, even in `Enforce` mode. It has no effect on either `generate` or `mutate` rules for the purposes of reporting. + +Background scanning occurs on a periodic basis (one hour by default) and offers some configuration options via [container flags](/docs/installation/customization/#container-flags). + +When background scanning is enabled, regardless of whether the policy's `validationFailureAction` is set to `Enforce` or `Audit`, the results will be recorded in a report. To see the specifics of how reporting works with background scans, refer to the tables below. + +**Reporting behavior when `background: true`** + +| | New Resource | Existing Resource | +|----------------------------------|--------------|-------------------| +| `validationFailureAction: Enforce` | Pass only | Report | +| `validationFailureAction: Audit` | Report | Report | + +**Reporting behavior when `background: false`** + +| | New Resource | Existing Resource | +|----------------------------------|--------------|-------------------| +| `validationFailureAction: Enforce` | Pass only | None | +| `validationFailureAction: Audit` | Report | None | + +Also, policy rules that are written using either certain variables from [AdmissionReview](/docs/writing-policies/variables/#variables-from-admission-review-requests) request information (e.g. `request.userInfo`), or fields like Roles, ClusterRoles, and Subjects in `match` and `exclude` statements, cannot be applied to existing resources in the background scanning mode since that information must come from an AdmissionReview request and is not available if the resource exists. Hence, these rules must set `background` to `false` to disable background scanning. The exceptions to this are `request.object` and `request.namespace` variables as these will be translated from the current state of the resource. diff --git a/content/en/docs/Policy Reports/examples.md b/content/en/docs/Policy Reports/examples.md new file mode 100644 index 000000000..5e817c21c --- /dev/null +++ b/content/en/docs/Policy Reports/examples.md @@ -0,0 +1,271 @@ +--- +title: Example Scenarios +description: > + Follow along scenarios for creating and viewing your first policy reports. +weight: 25 +--- + +## Example: Trigger a PolicyReport + +By default, a `PolicyReport` object (Namespaced) is created in the same Namespace where resources apply to one or more Kyverno policies, be they `Policy` or `ClusterPolicy` policies. + +A single Kyverno ClusterPolicy exists with a single rule which ensures Pods cannot mount Secrets as environment variables. + +```yaml +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: secrets-not-from-env-vars +spec: + background: true + validationFailureAction: Audit + rules: + - name: secrets-not-from-env-vars + match: + any: + - resources: + kinds: + - Pod + validate: + message: "Secrets must be mounted as volumes, not as environment variables." + pattern: + spec: + containers: + - name: "*" + =(env): + - =(valueFrom): + X(secretKeyRef): "null" +``` + +Creating a Pod in this Namespace which does not use any Secrets (and thereby does not violate the `secrets-not-from-env-vars` rule in the ClusterPolicy) will generate the first entry in the PolicyReport, but listed as a `PASS`. + +```sh +$ kubectl run busybox --image busybox:1.28 -- sleep 9999 +pod/busybox created + +$ kubectl get po +NAME READY STATUS RESTARTS AGE +busybox 1/1 Running 0 66s + +$ kubectl get polr +NAME PASS FAIL WARN ERROR SKIP AGE +cpol-secrets-not-from-env-vars 1 0 0 0 0 6s +``` + +Inspect the PolicyReport in the `default` Namespace to view its contents. Notice that the `busybox` Pod is listed as having passed. + +```sh +$ kubectl get polr cpol-secrets-not-from-env-vars -o yaml + + +results: +- message: validation rule 'secrets-not-from-env-vars' passed. + policy: secrets-not-from-env-vars + resources: + - apiVersion: v1 + kind: Pod + name: busybox + namespace: default + uid: 0dd94825-cc6e-435b-982b-fb76ac2fdc2a + result: pass + rule: secrets-not-from-env-vars + scored: true + source: kyverno + timestamp: + nanos: 0 + seconds: 1666097147 +summary: + error: 0 + fail: 0 + pass: 1 + skip: 0 + warn: 0 +``` + +Create another Pod which violates the rule in the sample policy. Because the rule is written with `validationFailureAction: Audit`, resources are allowed to be created which violate the rule. If this occurs, another entry will be created in the PolicyReport which denotes this condition as a FAIL. By contrast, if `validationFailureAction: Enforce` and an offending resource was attempted creation, it would be immediately blocked and therefore would not generate another entry in the report. However, if the resource passed then a PASS result would be created in the report. + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: secret-pod +spec: + containers: + - name: busybox + image: busybox:1.28 + env: + - name: SECRET_STUFF + valueFrom: + secretKeyRef: + name: mysecret + key: mysecretname +``` + +Since the above Pod spec was allowed and it violated the rule, there should now be a failure entry in the PolicyReport in the `default` Namespace. + +```sh +$ kubectl get polr polr-ns-default -o yaml + + +- message: 'validation error: Secrets must be mounted as volumes, not as environment + variables. rule secrets-not-from-env-vars failed at path /spec/containers/0/env/0/valueFrom/secretKeyRef/' + policy: secrets-not-from-env-vars + resources: + - apiVersion: v1 + kind: Pod + name: secret-pod + namespace: default + uid: 72a7422c-fb6f-486f-b274-1ca0de55d49d + result: fail + rule: secrets-not-from-env-vars + scored: true + source: kyverno + timestamp: + nanos: 0 + seconds: 1666098438 +summary: + error: 0 + fail: 1 + pass: 1 + skip: 0 + warn: 0 +``` + +Lastly, delete the Pod called `secret-pod` and once again check the PolicyReport object. + +```sh +$ kubectl delete po secret-pod +pod "secret-pod" deleted + +$ kubectl get polr cpol-secrets-not-from-env-vars +NAME PASS FAIL WARN ERROR SKIP AGE +cpol-secrets-not-from-env-vars 1 0 0 0 0 2m21s +``` + +Notice how the PolicyReport has removed the previously-failed entry when the violating Pod was deleted. + +## Example: Trigger a ClusterPolicyReport + +A ClusterPolicyReport is the same concept as a PolicyReport only it contains resources which are cluster scoped rather than Namespaced. + +As an example, create the following sample ClusterPolicy containing a single rule which validates that all new Namespaces should contain the label called `thisshouldntexist` and have some value. Notice how `validationFailureAction: Audit` and `background: true` in this ClusterPolicy. + +```yaml +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: require-ns-labels +spec: + validationFailureAction: Audit + background: true + rules: + - name: check-for-labels-on-namespace + match: + any: + - resources: + kinds: + - Namespace + validate: + message: "The label `thisshouldntexist` is required." + pattern: + metadata: + labels: + thisshouldntexist: "?*" +``` + +After creating this sample ClusterPolicy, check for the existence of a ClusterPolicyReport object. + +```sh +$ kubectl get cpolr +NAME PASS FAIL WARN ERROR SKIP AGE +cpol-require-ns-labels 0 5 0 0 0 76m +``` + +Notice that a ClusterPolicyReport named `cpol-require-ns-labels` exists with five failures. The number of combined results here will depend on the number of Namespaces in your cluster. At the moment the policy was created, Kyverno began inspecting existing Namespaces to evaluate them against the policy. + +The ClusterPolicyReport, when inspected, has the same structure as the PolicyReport object and contains entries in the `results` and `summary` objects with the outcomes of a policy audit. + +```yaml +results: +- message: 'validation error: The label `thisshouldntexist` is required. rule check-for-labels-on-namespace + failed at path /metadata/labels/thisshouldntexist/' + policy: require-ns-labels + resources: + - apiVersion: v1 + kind: Namespace + name: kube-node-lease + uid: 06e5056f-76a3-461a-8d45-2793b8bd5bbc + result: fail + rule: check-for-labels-on-namespace + scored: true + source: kyverno + timestamp: + nanos: 0 + seconds: 1666098654 +- message: 'validation error: The label `thisshouldntexist` is required. rule check-for-labels-on-namespace + failed at path /metadata/labels/thisshouldntexist/' + policy: require-ns-labels + resources: + - apiVersion: v1 + kind: Namespace + name: default + uid: 4ffe22fd-0927-4ed1-8b04-50ca7ed58626 + result: fail + rule: check-for-labels-on-namespace + scored: true + source: kyverno + timestamp: + nanos: 0 + seconds: 1666098654 +- message: 'validation error: The label `thisshouldntexist` is required. rule check-for-labels-on-namespace + failed at path /metadata/labels/thisshouldntexist/' + policy: require-ns-labels + resources: + - apiVersion: v1 + kind: Namespace + name: kyverno + uid: 5d87cd66-ce30-4abc-b863-f3b97715a5f1 + result: fail + rule: check-for-labels-on-namespace + scored: true + source: kyverno + timestamp: + nanos: 0 + seconds: 1666098654 +- message: 'validation error: The label `thisshouldntexist` is required. rule check-for-labels-on-namespace + failed at path /metadata/labels/thisshouldntexist/' + policy: require-ns-labels + resources: + - apiVersion: v1 + kind: Namespace + name: kube-public + uid: c077ee71-435b-4921-9e05-8751fee71b64 + result: fail + rule: check-for-labels-on-namespace + scored: true + source: kyverno + timestamp: + nanos: 0 + seconds: 1666098654 +- message: 'validation error: The label `thisshouldntexist` is required. rule check-for-labels-on-namespace + failed at path /metadata/labels/thisshouldntexist/' + policy: require-ns-labels + resources: + - apiVersion: v1 + kind: Namespace + name: kube-system + uid: e63fabde-b572-4b07-b899-b2230f4eac69 + result: fail + rule: check-for-labels-on-namespace + scored: true + source: kyverno + timestamp: + nanos: 0 + seconds: 1666098654 +summary: + error: 0 + fail: 5 + pass: 0 + skip: 0 + warn: 0 +``` diff --git a/content/en/docs/Releases/_index.md b/content/en/docs/Releases/_index.md index 84db7cba2..e209fd206 100644 --- a/content/en/docs/Releases/_index.md +++ b/content/en/docs/Releases/_index.md @@ -38,9 +38,7 @@ Patch releases are for backwards-compatible bug fixes, very minor enhancements w Kyverno uses GitHub tags to manage versions. New releases and release candidates are published using the wildcard tag `v..*`. -The dev images are pushed with tag `..dev-N-`, where "N" is a two-digit number beginning at one for the major-minor combination and incremented by one on each subsequent tagged image. See [more](https://github.com/kyverno/kyverno/pkgs/container/kyverno/versions) examples. You can find published dev images for a specific commit via the GitHub workflow named [image](https://github.com/kyverno/kyverno/actions/workflows/image.yaml). For example, this [job](https://github.com/kyverno/kyverno/runs/4579160206?check_suite_focus=true) pushed images with tag `1.6-dev-7-gff99d92f` for PR [#2856](https://github.com/kyverno/kyverno/pull/2856). - -To test with the latest images for different release branches, the images are pushed with `.-dev-latest`. +Untagged pushes to both release branches and the `main` branch also result in images being built. For commits pushed to a release branch (a branch prefixed with `release-`), an image is built with a tag equaling the git SHA which produced the build. A rolling tag of the branch name will be re-applied to the latest image built from commits pushed to that branch. For example, the latest commit to the `release-1.10` branch is `19621b7a703935d6217a404ab5054687a84a3eda` results in an image built which is tagged with `19621b7a703935d6217a404ab5054687a84a3eda` and `release-1.10`. Commits to the `main` branch follow a similar versioning strategy except the rolling tag `latest` will be used instead. ### Issues @@ -56,7 +54,7 @@ Release branches and PRs are managed as follows: - Branches are created for each major or minor release. - The branch name will contain the version, for example `release-1.5`. - Patch releases are created from a release branch. -- For critical fixes that need to be included in a patch release, PRs should always be first merged to main and then cherry-picked to the release branch. The milestone label (for example `milestone-1.5`) must be added to PRs that are to be merged to a release branch, along with the `cherry-pick-required` label. When the PR is cherry picked (or merged), the `cherry-pick-completed` label is added. +- For critical fixes that need to be included in a patch release, PRs should always be first merged to `main` and then cherry-picked to the release branch. The milestone label (for example `milestone-1.5`) must be added to PRs that are to be merged to a release branch, along with the `cherry-pick-required` label. When the PR is cherry picked (or merged), the `cherry-pick-completed` label is added. - For complex changes, or when branches diverge significantly, separate PRs may be required for `main` and release branches. - During PR review, the Assignee selection is used to indicate the reviewer. diff --git a/content/en/docs/Writing policies/autogen.md b/content/en/docs/Writing policies/autogen.md index 125324966..fbf528262 100644 --- a/content/en/docs/Writing policies/autogen.md +++ b/content/en/docs/Writing policies/autogen.md @@ -1,8 +1,8 @@ --- -title: Auto-Gen Rules for Pod Controllers +title: Auto-Gen Rules description: > - Automatically generate rules for Pod controllers. -weight: 100 + Automatically generate rules for Pod controllers. +weight: 110 --- Pods are one of the most common object types in Kubernetes and as such are the focus of most types of validation rules. But creation of Pods directly is almost never done as it is considered an anti-pattern. Instead, Kubernetes has many higher-level controllers that directly or indirectly manage Pods, namely the Deployment, DaemonSet, StatefulSet, Job, and CronJob resources. Writing policy that targets Pods but must be written for every one of these controllers would be tedious and inefficient. Kyverno solves this issue by supporting automatic generation of policy rules for higher-level controllers from a rule written exclusively for a Pod. For rules which match on Pods in addition to other kinds, auto-generation is not activated. diff --git a/content/en/docs/Writing policies/background.md b/content/en/docs/Writing policies/background.md deleted file mode 100644 index 5511d71fe..000000000 --- a/content/en/docs/Writing policies/background.md +++ /dev/null @@ -1,41 +0,0 @@ ---- -title: Background Scans -description: > - Manage applying policies to existing resources in a cluster. -weight: 110 ---- - -Kyverno can validate existing resources in the cluster that may have been created before a policy was created. This can be useful when evaluating the potential effects new `validate` policies will have on a cluster prior to changing them to `Enforce` mode. The application of policies to existing resources is referred to as **background scanning** and is enabled by default unless `background` is set to `false` in a policy like shown below in the snippet. - -```yaml -spec: - background: false - rules: - - name: default-deny-ingress -``` - -Background scanning, enabled by default in a `Policy` or `ClusterPolicy` object with the `spec.background` field, allows Kyverno to scan existing resources and find if they match any `validate` or `verifyImages` rules. If existing resources are found which would violate an existing policy, the background scan notes them in a `ClusterPolicyReport` or a `PolicyReport` object, depending on if the resource is namespaced or not. It does not block any existing resources that match a rule, even in `Enforce` mode. Background scanning is an optional field and defaults to `true`, only taking effect on `validate` and `verifyImages` rules. It has no effect on either `generate` or `mutate` rules. - -Background scanning occurs on a continual basis and offers some configuration options via [container flags](/docs/installation/customization/#container-flags). - -{{% alert title="Note" color="info" %}} -Kyverno does not mutate existing resources to prevent inadvertent changes to workloads. Mutate and generate rules are not processed during background scans. -{{% /alert %}} - -When background scanning is enabled, regardless of whether the policy's `validationFailureAction` is set to `Enforce` or `Audit`, the results will be recorded in a report. To see the specifics of how reporting works with background scans, refer to the tables below. - -**Reporting behavior when `background: true`** - -| | New Resource | Existing Resource | -|----------------------------------|--------------|-------------------| -| `validationFailureAction: Enforce` | None | Report | -| `validationFailureAction: Audit` | Report | Report | - -**Reporting behavior when `background: false`** - -| | New Resource | Existing Resource | -|----------------------------------|--------------|-------------------| -| `validationFailureAction: Enforce` | None | None | -| `validationFailureAction: Audit` | Report | None | - -Also, policy rules that are written using either certain variables from [AdmissionReview](/docs/writing-policies/variables/#variables-from-admission-review-request-data) request information (e.g. `request.userInfo`), or fields like Roles, ClusterRoles, and Subjects in `match` and `exclude` statements, cannot be applied to existing resources in the background scanning mode since that information must come from an AdmissionReview request and is not available if the resource exists. Hence, these rules must set `background` to `false` to disable background scanning. The exceptions to this are `request.object` and `request.namespace` variables as these will be translated from the current state of the resource. diff --git a/content/en/docs/Writing policies/cleanup.md b/content/en/docs/Writing policies/cleanup.md index 570ec4736..da895a6b4 100644 --- a/content/en/docs/Writing policies/cleanup.md +++ b/content/en/docs/Writing policies/cleanup.md @@ -1,7 +1,8 @@ --- -title: Cleanup -description: Automate the resource cleanup process by using a CleanupPolicy. -weight: 67 +title: Cleanup Rules +description: > + Remove Kubernetes resources. +weight: 70 --- {{% alert title="Warning" color="warning" %}} diff --git a/content/en/docs/Writing policies/exceptions.md b/content/en/docs/Writing policies/exceptions.md index 5377d4725..65bf4963c 100644 --- a/content/en/docs/Writing policies/exceptions.md +++ b/content/en/docs/Writing policies/exceptions.md @@ -1,7 +1,8 @@ --- -title: Exceptions -description: Create an exception to an existing policy using a PolicyException. -weight: 65 +title: Policy Exceptions +description: > + Create an exception to an existing policy using a PolicyException. +weight: 80 --- {{% alert title="Warning" color="warning" %}} @@ -10,7 +11,9 @@ Policy exceptions are an **alpha** feature and requires setting certain [contain Although Kyverno policies contain multiple methods to provide fine-grained control as to which resources they act upon in the form of [`match`/`exclude` blocks](/docs/writing-policies/match-exclude/#match-statements), [preconditions](/docs/writing-policies/preconditions/) at multiple hierarchies, [anchors](/docs/writing-policies/validate/#anchors), and more, all these mechanisms have in common that the resources which they are intended to exclude must occur in the same rule definition. This may be limiting in situations where policies may not be directly editable, or doing so imposes an operational burden. -For example, in organizations where multiple teams must interact with the same cluster, a team responsible for policy authoring and administration may not be the same team responsible for submission of resources. In these cases, it can be advantageous to decouple the policy definition from certain exclusions. Additionally, there are often times where an organization or team must allow certain exceptions which would violate otherwise valid rules but on a one-time basis if the risks are known and acceptable. Imagine a validate policy exists in the cluster and in `Enforce` mode which mandates all Pods must not mount host namespaces. A separate team has a legitimate need to run a specific tool in this cluster for a limited time which violates this policy. Normally, the policy would block such a "bad" Pod if the policy was not previously altered in such a way to allow said Pod to run. Rather than making adjustments to the policy, an exception may be granted. Both of these examples are use cases for a **PolicyException** resource described below. +For example, in organizations where multiple teams must interact with the same cluster, a team responsible for policy authoring and administration may not be the same team responsible for submission of resources. In these cases, it can be advantageous to decouple the policy definition from certain exclusions. Additionally, there are often times where an organization or team must allow certain exceptions which would violate otherwise valid rules but on a one-time basis if the risks are known and acceptable. + +Imagine a validate policy exists in `Enforce` mode which mandates all Pods must not mount host namespaces. A separate team has a legitimate need to run a specific tool in this cluster for a limited time which violates this policy. Normally, the policy would block such a "bad" Pod if the policy was not previously altered in such a way to allow said Pod to run. Rather than making adjustments to the policy, an exception may be granted. Both of these examples are use cases for a **PolicyException** resource described below. A `PolicyException` is a Namespaced Custom Resource which allows a resource(s) to be allowed past a given policy and rule combination. It can be used to exempt any resource from any Kyverno rule type although it is primarily intended for use with validate rules. A PolicyException encapsulates the familiar `match`/`exclude` statements used in `Policy` and `ClusterPolicy` resources but adds an `exceptions{}` object to select the policy and rule name(s) used to form the exception. The logical flow of how a PolicyException works in tandem with a validate policy is depicted below. @@ -121,6 +124,10 @@ exceptions: - host-namespaces ``` +PolicyExceptions also support background scanning. An exception which defines `spec.background=true` will influence [Policy Reports](/docs/policy-reports/) when the exception is processed, allowing report results to change from a `Fail` to a `Skip` result. + +Wildcards (`"*"`) are supported in the value of the `ruleNames[]` field allowing exception from any/all rules in the policy without having to name them explicitly. + Since PolicyExceptions are just another Custom Resource, their use can and should be controlled by a number of different mechanisms to ensure their creation in a cluster is authorized including: * Kubernetes RBAC diff --git a/content/en/docs/Writing policies/external-data-sources.md b/content/en/docs/Writing policies/external-data-sources.md index 1c02a0273..af5cde2b1 100644 --- a/content/en/docs/Writing policies/external-data-sources.md +++ b/content/en/docs/Writing policies/external-data-sources.md @@ -1,27 +1,23 @@ --- title: External Data Sources description: > - Use data from ConfigMaps, the Kubernetes API server, and image registries in Kyverno policies. -weight: 80 + Fetch data from ConfigMaps, the Kubernetes API server, other cluster services, and image registries for use in Kyverno policies. +weight: 100 --- -The [Variables](/docs/writing-policies/variables/) section discusses how variables can help create smarter and reusable policy definitions and introduced the concept of a rule [`context`](/docs/writing-policies/variables/#variables-from-external-data-sources) that stores all variables. +The [Variables](/docs/writing-policies/variables/) section discusses how variables can help create smarter and reusable policy definitions and introduced the concept of a rule [context](/docs/writing-policies/variables/#variables-from-external-data-sources) that stores all variables. -This section provides details on using ConfigMaps, API calls, and image registries to reference external data as variables in policies. - -{{% alert title="Note" color="info" %}} -For improved security and performance, Kyverno is designed to not allow connections to systems other than the cluster Kubernetes API server and image registries. Use a separate controller to fetch data from any source and store it in a ConfigMap that can be efficiently used in a policy. This design enables separation of concerns and enforcement of security boundaries. -{{% /alert %}} +This section provides details on using ConfigMaps, API calls, service calls, and image registries to reference external data as variables in policies. ## Variables from ConfigMaps A [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) resource in Kubernetes is commonly used as a source of configuration details which can be consumed by applications. This data can be written in multiple formats, stored in a Namespace, and accessed easily. Kyverno supports using a ConfigMap as a data source for variables. When a policy referencing a ConfigMap resource is evaluated, the ConfigMap data is checked at that time ensuring that references to the ConfigMap are always dynamic. Should the ConfigMap be updated, subsequent policy lookups will pick up the latest data at that point. -In order to consume data from a ConfigMap in a rule, a `context` is required. For each rule you wish to consume data from a ConfigMap, you must define a `context`. The context data can then be referenced in the policy rule using JMESPath notation. +In order to consume data from a ConfigMap in a rule, a context is required. For each rule you wish to consume data from a ConfigMap, you must define a context. The context data can then be referenced in the policy rule using JMESPath notation. ### Looking up ConfigMap values -A ConfigMap that is defined in a rule's `context` can be referred to using its unique name within the context. ConfigMap values can be referenced using a JMESPath style expression. +A ConfigMap that is defined in a rule's context can be referenced using its unique name within the context. ConfigMap values can be referenced using a JMESPath style expression. ```sh {{ .data. }} @@ -39,7 +35,7 @@ data: env: production ``` -To refer to values from a ConfigMap inside a rule, define a `context` inside the rule with one or more ConfigMap declarations. Using the sample ConfigMap snippet referenced above, the below rule defines a `context` which references this specific ConfigMap by name. +To refer to values from a ConfigMap inside a rule, define a context inside the rule with one or more ConfigMap declarations. Using the sample ConfigMap snippet referenced above, the below rule defines a context which references this specific ConfigMap by name. ```yaml rules: @@ -57,7 +53,7 @@ rules: Based on the example above, we can now refer to a ConfigMap value using `{{dictionary.data.env}}`. The variable will be substituted with the value `production` during policy execution. -Put into context of a full `ClusterPolicy`, referencing a ConfigMap as a variable looks like the following. +Put into context of a full policy, referencing a ConfigMap as a variable looks like the following. ```yaml apiVersion: kyverno.io/v1 @@ -69,16 +65,16 @@ metadata: spec: rules: - name: example-configmap-lookup - context: - - name: dictionary - configMap: - name: some-config-map - namespace: some-namespace match: any: - resources: kinds: - Pod + context: + - name: dictionary + configMap: + name: some-config-map + namespace: some-namespace mutate: patchStrategicMerge: metadata: @@ -86,17 +82,17 @@ spec: my-environment-name: "{{dictionary.data.env}}" ``` -In the above ClusterPolicy, a mutate rule matches all incoming Pod resources and adds a label to them with the name of `my-environment-name`. Because we have defined a `context` which points to our earlier ConfigMap named `mycmap`, we can reference the value with the expression `{{dictionary.data.env}}`. A new Pod will then receive the label `my-environment-name=production`. +In the above ClusterPolicy, a mutate rule matches all incoming Pod resources and adds a label to them with the name of `my-environment-name`. Because we have defined a context which points to our earlier ConfigMap named `mycmap`, we can reference the value with the expression `{{dictionary.data.env}}`. A new Pod will then receive the label `my-environment-name=production`. {{% alert title="Note" color="info" %}} ConfigMap names and keys can contain characters that are not supported by [JMESPath](http://jmespath.org/), such as "-" (minus or dash) or "/" (slash). To evaluate these characters as literals, add double quotes to that part of the JMESPath expression as follows: ``` {{ "".data."" }} ``` -See the [JMESPath page](/docs/writing-policies/jmespath/#formatting) for more information on formatting concerns. +See the [JMESPath page](/docs/writing-policies/jmespath/#formatting) for more information on formatting. {{% /alert %}} -Kyverno also has the ability to cache ConfigMaps commonly used by policies to reduce the number of API calls made. This both decreases the load on the API server and increases the speed of policy evaluation. Assign the label `cache.kyverno.io/enabled: "true"` to any ConfigMap and Kyverno will automatically cache it. Policy decisions will fetch the data from cache rather than querying the API server. +Kyverno also has the ability to cache ConfigMaps commonly used by policies to reduce the number of API calls made. This both decreases the load on the API server and increases the speed of policy evaluation. Assign the label `cache.kyverno.io/enabled: "true"` to any ConfigMap and Kyverno will automatically cache it. Policy decisions will fetch the data from cache rather than querying the API server. This feature may be disabled through an optional [container flag](/docs/installation/customization/#container-flags) if desired. ### Handling ConfigMap Array Values @@ -116,7 +112,7 @@ data: allowed-roles: '["cluster-admin", "cluster-operator", "tenant-admin"]' ``` -Now that the array data is saved in the `allowed-roles` key, here is a sample ClusterPolicy containing a single `rule` that blocks a Deployment if the value of the annotation named `role` is not in the allowed list. Notice how the [`parse_json()` JMESPath filter](/docs/writing-policies/jmespath/#parse_json) is used to interpret the value of the ConfigMap's `allowed-roles` key into an array of strings. +Now that the array data is saved in the `allowed-roles` key, here is a sample policy which blocks a Deployment if the value of the annotation named `role` is not in the allowed list. Notice how the [`parse_json()` JMESPath filter](/docs/writing-policies/jmespath/#parse_json) is used to interpret the value of the ConfigMap's `allowed-roles` key into an array of strings. ```yaml apiVersion: kyverno.io/v1 @@ -154,7 +150,7 @@ This rule denies the request for a new Deployment if the annotation `role` is no You may also notice that this sample uses variables from both AdmissionReview and ConfigMap sources in a single rule. This combination can prove to be very powerful and flexible in crafting useful policies. {{% /alert %}} -Once creating this sample `ClusterPolicy`, attempt to create a new Deployment where the annotation `role=super-user` and test the result. +Once creating this sample policy, attempt to create a new Deployment where the annotation `role=super-user` and test the result. ```yaml apiVersion: apps/v1 @@ -204,10 +200,10 @@ Kubernetes is powered by a declarative API that allows querying and manipulating A Kyverno Kubernetes API call works just as with `kubectl` and other API clients, and can be tested using existing tools. -For example, here is a command line that uses `kubectl` to fetch the list of Pods in a Namespace and then pipes the output to `kyverno jp` which counts the number of Pods: +For example, this command uses `kubectl` to fetch the list of Pods in a Namespace and then pipes the output to `kyverno jp` which counts the number of Pods: ```sh -kubectl get --raw /api/v1/namespaces/kyverno/pods | kyverno jp "items | length(@)" +kubectl get --raw /api/v1/namespaces/kyverno/pods | kyverno jp query "items | length(@)" ``` {{% alert title="Tip" color="info" %}} @@ -226,6 +222,33 @@ rules: jmesPath: "items | length(@)" ``` +Calls to the Kubernetes API server may also perform `POST` operations in addition to `GET` which is the default method. The returned data from the API server can then be used for further policy decisions. + +For example, this snippet below shows making a call to the [SubjectAccessReview API](https://kubernetes.io/zh-cn/docs/reference/kubernetes-api/authorization-resources/subject-access-review-v1/) to determine if an actor is authorized. Performing a `POST` operation requires specifying the `method` field and the `data` object with the contents of the request given as a list of key/value pairs. + +```yaml +context: + - name: subjectaccessreview + apiCall: + urlPath: /apis/authorization.k8s.io/v1/subjectaccessreviews + method: POST + data: + - key: kind + value: SubjectAccessReview + - key: apiVersion + value: authorization.k8s.io/v1 + - key: spec + value: + resource: "namespace" + resourceAttributes: + namespace: "{{ request.namespace }}" + verb: "delete" + group: "" + user: "{{ request.userInfo.username }}" +``` + +The response from such a request will be the full JSON return and accessible under the variable `subjectaccessreview`. + ### URL Paths The Kubernetes API organizes resources under groups and versions. For example, the resource type `Deployment` is available in the API Group `apps` with a version `v1`. @@ -242,21 +265,25 @@ For Namespaced resources, to get a specific resource by name or to get all resou For historic resources, the Kubernetes Core API is available under `/api/v1`. For example, to query all Namespace resources the path `/api/v1/namespaces` is used. -The Kubernetes API groups are defined in the [API reference documentation for v1.22](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#-strong-api-groups-strong-) and can also be retrieved via the `kubectl api-resources` command shown below: +The Kubernetes API groups are defined in the [API reference documentation for v1.25](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.25/) and can also be retrieved via the `kubectl api-resources` command shown below: ```sh $ kubectl api-resources -NAME SHORTNAMES APIGROUP NAMESPACED KIND -bindings true Binding -componentstatuses cs false ComponentStatus -configmaps cm true ConfigMap -endpoints ep true Endpoints -events ev true Event -limitranges limits true LimitRange -namespaces ns false Namespace -nodes no false Node -persistentvolumeclaims pvc true PersistentVolumeClaim - +NAME SHORTNAMES APIVERSION NAMESPACED KIND +bindings v1 true Binding +componentstatuses cs v1 false ComponentStatus +configmaps cm v1 true ConfigMap +endpoints ep v1 true Endpoints +events ev v1 true Event +limitranges limits v1 true LimitRange +namespaces ns v1 false Namespace +nodes no v1 false Node +persistentvolumeclaims pvc v1 true PersistentVolumeClaim +persistentvolumes pv v1 false PersistentVolume +pods po v1 true Pod +podtemplates v1 true PodTemplate +replicationcontrollers rc v1 true ReplicationController +resourcequotas quota v1 true ResourceQuota ... ``` @@ -265,20 +292,17 @@ The `kubectl api-versions` command prints out the available versions for each AP ```sh $ kubectl api-versions admissionregistration.k8s.io/v1 -admissionregistration.k8s.io/v1beta1 +admissionregistration.k8s.io/v1alpha1 apiextensions.k8s.io/v1 -apiextensions.k8s.io/v1beta1 apiregistration.k8s.io/v1 -apiregistration.k8s.io/v1beta1 apps/v1 authentication.k8s.io/v1 -authentication.k8s.io/v1beta1 authorization.k8s.io/v1 -authorization.k8s.io/v1beta1 autoscaling/v1 -autoscaling/v2beta1 -autoscaling/v2beta2 +autoscaling/v2 batch/v1 +certificates.k8s.io/v1 +coordination.k8s.io/v1 ... ``` @@ -300,7 +324,7 @@ The API group is shown in the third column of the output. You can then use the g kubectl api-versions | grep apps ``` -The output of this will be `apps/v1`. Older versions of Kubernetes (prior to 1.18) will also show `apps/v1beta2`. +The output of this will be `apps/v1`. {{% /alert %}} @@ -370,6 +394,31 @@ context: jmesPath: "items[?spec.type == 'LoadBalancer'] | length(@)" ``` +Using query parameters has the added benefit that there will always be a response even if the query returns no results. This can be beneficial (and even necessary) in some cases where an API call to fetch an exact resource by name may fail because the resource does not exist. + +For example, if the `foo` Service does not exist, an API call to return that specific resource will fail. + +```sh +$ kubectl get --raw /api/v1/namespaces/default/services/foo +Error from server (NotFound): services "foo" not found +``` + +However, an API call with a query parameter against all Services will return successful but with an empty collection. + +```sh +$ kubectl get --raw /api/v1/namespaces/default/services?fieldSelector=metadata.name=foo | jq +{ + "kind": "ServiceList", + "apiVersion": "v1", + "metadata": { + "resourceVersion": "167567" + }, + "items": [] +} +``` + +Further information on handling collections is covered below. + ### Handling collections The API server response for a `HTTP GET` on a URL path that requests collections of resources will be an object with a list of items (resources). @@ -432,7 +481,7 @@ This will return a `NamespaceList` object with a property `items` that contains To process this data in JMESPath, reference the `items`. Here is an example which extracts a few metadata fields across all Namespace resources: ```sh -kubectl get --raw /api/v1/namespaces | kyverno jp "items[*].{name: metadata.name, creationTime: metadata.creationTimestamp}" +kubectl get --raw /api/v1/namespaces | kyverno jp query "items[*].{name: metadata.name, creationTime: metadata.creationTimestamp}" ``` This produces a new JSON list of objects with properties `name` and `creationTime`. @@ -453,14 +502,14 @@ This produces a new JSON list of objects with properties `name` and `creationTim To find an item in the list you can use JMESPath filters. For example, this command will match a Namespace by its name: ```sh -kubectl get --raw /api/v1/namespaces | kyverno jp "items[?metadata.name == 'default'].{uid: metadata.uid, creationTimestamp: metadata.creationTimestamp}" +kubectl get --raw /api/v1/namespaces | kyverno jp query "items[?metadata.name == 'default'].{uid: metadata.uid, creationTimestamp: metadata.creationTimestamp}" ``` -In addition to wildcards and filters, JMESPath has many additional powerful features including several useful functions. Be sure to go through the [JMESPath tutorial](https://jmespath.org/tutorial.html) and try the interactive examples in addition to the Kyverno JMESPath page [here](/docs/writing-policies/jmespath/). +In addition to wildcards and filters, JMESPath has many additional powerful features including several useful functions. Be sure to go through the [JMESPath tutorial](https://jmespath.org/tutorial.html) and try the interactive examples in addition to the Kyverno [JMESPath page](/docs/writing-policies/jmespath/). -### Sample Policy: Limit Services of type LoadBalancer in a Namespace +### Examples -Here is a complete sample policy that limits each namespace to a single service of type `LoadBalancer`. +Here is a complete sample policy that limits each Namespace to a single Service of type `LoadBalancer`. ```yaml apiVersion: kyverno.io/v1 @@ -476,16 +525,13 @@ spec: - resources: kinds: - Service + operations: + - CREATE context: - name: serviceCount apiCall: urlPath: "/api/v1/namespaces/{{ request.namespace }}/services" jmesPath: "items[?spec.type == 'LoadBalancer'] | length(@)" - preconditions: - any: - - key: "{{ request.operation }}" - operator: Equals - value: CREATE validate: message: "Only one LoadBalancer service is allowed per namespace" deny: @@ -496,7 +542,52 @@ spec: value: 1 ``` -This sample policy retrieves the list of Services in the Namespace and stores the count of type `LoadBalancer` in a variable called serviceCount. A `deny` rule is used to ensure that the count cannot exceed one. +This sample policy retrieves the list of Services in the Namespace and stores the count of type `LoadBalancer` in a variable called `serviceCount`. A `deny` rule is used to ensure that the count cannot exceed one. + +## Variables from Service Calls + +Similar to how Kyverno is able to call the Kubernetes API server to both `GET` and `POST` data for use in a context variable, Kyverno is also able to call any other service in the cluster. This feature is nested under the apiCall context type and builds upon it. See the section [above](#variables-from-kubernetes-api-server-calls) on Kubernetes API calls for more information. + +By using the `apiCall.service` object, a call may be made to another URL to retrieve and store data. The fields `caBundle` and `url` are used to specify the CA bundle and URL, respectively, for the call. The fields `apiCall.urlPath` and `apiCall.service.url` are mutually exclusive; a call can only be to either the Kubernetes API or some other service. At this time, authentication as part of these service calls is not supported. The response from a Service call must only be JSON. + +For example, the following policy makes a `POST` request to another Kubernetes Service accessible at `http://sample.kyverno-extension/check-namespace` and sends it the data `{"name":"foonamespace"}` when a ConfigMap is created in the `foonamespace` Namespace. The JSON result Kyverno receives is stored in the context called `result`. The value of `result` is JSON where the key `allowed` is either `true` or `false`. The request is blocked if the value is `false`. + +```yaml +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: check-namespaces +spec: + validationFailureAction: Enforce + rules: + - name: call-extension + match: + any: + - resources: + kinds: + - ConfigMap + context: + - name: result + apiCall: + method: POST + data: + - key: namespace + value: "{{request.namespace}}" + service: + url: http://sample.kyverno-extension/check-namespace + caBundle: |- + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- + validate: + message: "namespace {{request.namespace}} is not allowed" + deny: + conditions: + all: + - key: "{{ result.allowed }}" + operator: Equals + value: false +``` ## Variables from Image Registries @@ -608,11 +699,9 @@ spec: - resources: kinds: - Pod - preconditions: - all: - - key: "{{request.operation}}" - operator: NotEquals - value: DELETE + operations: + - CREATE + - UPDATE validate: message: "Images run as root are not allowed." foreach: diff --git a/content/en/docs/Writing policies/generate.md b/content/en/docs/Writing policies/generate.md index 8e5cb4e63..c9b51be98 100644 --- a/content/en/docs/Writing policies/generate.md +++ b/content/en/docs/Writing policies/generate.md @@ -1,7 +1,8 @@ --- -title: Generate Resources -description: Create additional resources based on resource creation or updates. -weight: 60 +title: Generate Rules +description: > + Create new Kubernetes resources. +weight: 50 --- A `generate` rule can be used to create additional resources when a new resource is created or when the source is updated. This is useful to create supporting resources, such as new RoleBindings or NetworkPolicies for a Namespace. diff --git a/content/en/docs/Writing policies/jmespath.md b/content/en/docs/Writing policies/jmespath.md index cfb5fd6ec..3dbfd264d 100644 --- a/content/en/docs/Writing policies/jmespath.md +++ b/content/en/docs/Writing policies/jmespath.md @@ -388,11 +388,9 @@ spec: - resources: kinds: - Pod - preconditions: - any: - - key: "{{ request.operation }}" - operator: In - value: ["CREATE","UPDATE"] + operations: + - CREATE + - UPDATE validate: message: "The total memory defined in requests and limits must not exceed 200Mi." foreach: @@ -408,98 +406,6 @@ spec:

-### Sum - -
Expand - -

- -The sum() filter is a customized version of the default filter present in [upstream JMESPath](https://jmespath.org/specification.html#sum) which brings, in addition to the support for scalars (integers), the ability to sum duration and quantities. sum() is similar to add() with the difference that sum() accepts an array as an input while add() does not. - -`sum()` is value-aware (based on the formatting used for the inputs) just as `add()` and is capable of adding list of numbers, quantities, and durations without any form of unit conversion. - -Arithmetic filters like `sum()` currently accept inputs in the following formats. - -* Number (ex., \`10\`) -* Quantity (ex., '10Mi') -* Duration (ex., '10h') - -Note that how the inputs are enclosed determines how Kyverno interprets their type. Numbers enclosed in back ticks are scalar values while quantities and durations are enclosed in single quotes thus treating them as strings. Using the correct enclosing character is important because, in Kubernetes "regular" numbers are treated implicitly as units of measure. The number written \`10\` is interpreted as an integer or "the number ten" whereas '10' is interpreted as a string or "ten bytes". See the [Formatting](#formatting) section above for more details. - -| Input 1 | Output | -|----------------------- |----------| -| Array/Number | Number | -| Array/Quantity/Number | Quantity | -| Array/Duration/Number | Duration | - -Some specific behaviors to note: - -* If a combination of duration ('1h') and number (\`5\`) are the inputs, the number will be interpreted as seconds resulting in a sum of `1h0m5s`. -* Because of durations being a string just like [resource quantities](https://kubernetes.io/docs/reference/kubernetes-api/common-definitions/quantity/), and the minutes unit of "m" also present in quantities interpreted as the "milli" prefix, there is no support for minutes. - -For example, given the following map below - -```json -{ - "numbers": ['2Ki','5Ki','8Ki'] -} -``` - -the `sum()` filter can sum up this array of entities and produce a result in the same type of entity (in this case, Quantity). - -```sh -$ echo '{"numbers": ['2Ki','5Ki','8Ki']}' | kyverno jp query "sum(numbers)" -#sum(numbers) -"15Ki" -``` - -**Example:** This policy denies a Pod if the aggregated storage quota is greater than the one defined in a ConfigMap.. - -```yaml -apiVersion: kyverno.io/v1 -kind: ClusterPolicy -metadata: - name: restrict-storage-quota -spec: - validationFailureAction: Enforce - background: true - rules: - - name: restrict-storage - match: - any: - - resources: - kinds: - - Volume - preconditions: - all: - - key: "{{ request.operation || 'BACKGROUND' }}" - operator: AnyIn - value: [ "CREATE", "UPDATE" ] - context: - - name: customer-resource-quota - configMap: - name: osc-resource-quota - namespace: "{{ request.namespace }}" - - name: storage-count - apiCall: - urlPath: "/apis/storage.api.onmetal.de/v1alpha1/namespaces/{{request.namespace }}/volumes/" - jmesPath: "items[*].metadata.labels.\"osc-storage-gb\" | sum([].to_number(@))" - validate: - message: "Only {{ \"customer-resource-quota\".data.\" -limit.storage.gb\" }} GB storage are allowed. Already consumed {{ -\"storage-count\" }} GB onmetal storage. Trying to allocate additional {{ -request.object.metadata.labels.\"osc-storage-gb\" }} Gi of storage. Please -contact the administrator to increase the quota." - deny: - conditions: - any: - - key: "{{add((request.object.metadata.labels.\"osc-storage-gb\").to_number(@),\"storage-count\") }}" - operator: GreaterThan - value: "{{ \"customer-resource-quota\".data.\"limit.storage.gb\"}}" -``` - - -

@@ -685,13 +591,9 @@ spec: - resources: kinds: - Pod - preconditions: - any: - - key: "{{ request.operation }}" - operator: In - value: - - CREATE - - UPDATE + operations: + - CREATE + - UPDATE validate: message: Limits may not exceed 2.5x the requests. foreach: @@ -753,12 +655,53 @@ spec:

+### Image_normalize + +
Expand +

+ +The `image_normalize()` filter is used to output the canonical image string as it is known internally to Kyverno. This filter will render the internal values for the default image registry and tag (`latest`) if they apply to a given image. It is useful particularly in mutate rules where the full image value is needed to decide whether to mutate it. This filter cannot be tested with the Kyverno CLI because the runtime context is only available in webhook mode. + +For example, assuming the value of the default registry is left at its defaults of `docker.io` and a Pod is sent to Kyverno which has a single container the value of its `image` field being only `nginx`, when run through the `image_normalize()` filter will come out `docker.io/nginx:latest`. + +| Input 1 | Output | +|--------------------|----------| +| String | String | + +**Example:** This policy will mutate the value of the `image` field in a Pod's `containers[]` array if, after normalization, it begins with `docker.io`. The registry will be replaced with `harbor.corp.org`. + +```yaml +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: image-normalize-demo +spec: + rules: + - name: replace-image-registry-pod-containers + match: + any: + - resources: + kinds: + - Pod + mutate: + foreach: + - list: "request.object.spec.containers" + patchStrategicMerge: + spec: + containers: + - name: "{{ element.name }}" + image: "{{ regex_replace_all('^docker.io/(.*)$', image_normalize('{{element.image}}'), 'harbor.corp.org/$1' )}}" +``` + +

+
+ ### Items
Expand

-The `items()` filter iterates on map keys (ex., annotations or labels) and converts them to an array of objects with key/value attributes with custom names. +The `items()` filter iterates on map keys (ex., annotations or labels) or arrays and converts them to an array of objects with key/value attributes with custom names. For example, given the following map below @@ -785,9 +728,32 @@ $ echo '{"team" : "apple" , "organization" : "banana" }' | k kyverno jp "items(@ ] ``` +It can also work on an input source given as an array of objects. + +```sh +$ echo '[{"team" : "apple"} , {"organization" : "banana"}]' | kyverno jp query "items(@, 'key', 'value')" +Reading from terminal input. +Enter input object and hit Ctrl+D. +# items(@, 'key', 'value') +[ + { + "key": 0, + "value": { + "team": "apple" + } + }, + { + "key": 1, + "value": { + "organization": "banana" + } + } +] +``` + | Input 1 | Input 2 | Input 3 | Output | |--------------------------|--------------------|------------|---------------| -| Map (Object) | String | String | Array/Object | +| Map (Object) or Array | String | String | Array/Object | Related filter to `items()` is its inverse, [`object_from_list()`](#object_from_list). @@ -818,7 +784,9 @@ spec: patchesJson6902: |- - path: "/spec/forProvider/tagging/tagSet/-1" op: add - value: {"key": "{{element.key}}", "value": "{{element.value}}"} + value: + key: "{{element.key}}" + value": "{{element.value}}" ``` Given a Namespace which looks like the following @@ -967,11 +935,8 @@ spec: - resources: kinds: - Deployment - preconditions: - any: - - key: "{{request.operation}}" - operator: Equals - value: CREATE + operations: + - CREATE context: - name: pdb_count apiCall: @@ -1029,13 +994,9 @@ spec: - resources: kinds: - Pod - preconditions: - any: - - key: "{{ request.operation }}" - operator: In - value: - - CREATE - - UPDATE + operations: + - CREATE + - UPDATE validate: message: Limits must be evenly divisible by the requests. foreach: @@ -1489,12 +1450,8 @@ spec: - resources: kinds: - Secret - preconditions: - all: - - key: "{{request.operation}}" - operator: In - value: - - CREATE + operations: + - CREATE context: - name: randomtest variable: @@ -1800,9 +1757,12 @@ spec: rules: - name: check-path match: - resources: - kinds: - - Ingress + any: + - resources: + kinds: + - Ingress + operations: + - CREATE context: # Looks up the Ingress paths across the whole cluster. - name: allpaths @@ -1814,10 +1774,6 @@ spec: apiCall: urlPath: "/apis/networking.k8s.io/v1/namespaces/{{request.object.metadata.namespace}}/ingresses" jmesPath: "items[].spec.rules[].http.paths[].path" - preconditions: - - key: "{{request.operation}}" - operator: Equals - value: "CREATE" validate: message: >- The root path /{{request.object.spec.rules[].http.paths[].path | to_string(@) | split(@, '/') | [1]}}/ exists @@ -1889,6 +1845,56 @@ spec:

+### Sum + +
Expand +

+ +The `sum()` filter takes an array of numbers, durations, or quantities and sums them together. This is a customized version of `sum()` found in the [upstream JMESPath specification](https://jmespath.org/specification.html#sum) but augmented to support inputs common to Kubernetes workloads, specifically durations and quantities. `sum()` is similar to `add()` with the difference that `sum()` accepts an array as an input while `add()` does not. Inputs must be of a homogenous type. For example, the query `echo '{"input":['2Ki','5Gi','8Mi']}' | kyverno jp query "sum(input)"` results in the value `"5251074Ki"`. The query `echo '{"input":['2h','50s','90s']}' | kyverno jp query "sum(input)"` results in the value `"2h2m20s"`. And the query `echo '{"input":[6,3,8]}' | kyverno jp query "sum(input)"` results in the value of `17`. + +Arithmetic filters like `sum()` currently accept inputs in the following formats. + +* Number (ex., \`10\`) +* Quantity (ex., '10Mi') +* Duration (ex., '10h') + +See the [Formatting](#formatting) section above for more details on how inputs are expected to be supplied. Durations cannot be expressed as minutes (i.e., `"5m"`) as the "m" unit is interpreted to be millicores. + +| Input 1 | Output | +|--------------------|----------| +| Array/Number | Number | +| Array/Quantity | Quantity | +| Array/Duration | Duration | + +**Example:** This policy sums the memory requests of all containers in a Pod and denies it if it exceeds 1 gibibyte (1Gi). + +```yaml +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: sum-demo +spec: + validationFailureAction: Enforce + rules: + - name: memory-requests-check + match: + any: + - resources: + kinds: + - Pod + validate: + message: The sum of all memory requests in a Pod cannot exceed 1 gibibyte. + deny: + conditions: + all: + - key: "{{ sum(request.object.spec.containers[].resources.requests.memory) }}" + operator: GreaterThan + value: 1Gi +``` + +

+
+ ### Time_add
Expand @@ -2467,6 +2473,47 @@ spec:

+### To_boolean + +
Expand +

+ +The `to_boolean()` filter converts a string to its boolean equivalent. Any strings which spell out "true" or "false" regardless of character case will be returned in boolean format. For example, the query `to_boolean('true')` will result in the output `true`. The query `to_boolean('FalsE')` will result in the output `false`. + +This filter can be helpful when needing to produce output for a field which only accepts boolean without requiring more complex string manipulation. + +| Input 1 | Output | +|--------------------|---------| +| String | Boolean | + +**Example:** This policy sets the `hostIPC` field of a Pod spec appropriately based on the value of a label (a string). Note that use of this filter may require setting the policy option `spec.schemaValidation` to `false` since there may be a type checking mismatch. + +```yaml +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: to-boolean-demo +spec: + schemaValidation: false + rules: + - name: canuseIPC + match: + any: + - resources: + kinds: + - Pod + selector: + matchLabels: + canuseIPC: "true" + mutate: + patchStrategicMerge: + spec: + hostIPC: "{{ to_boolean (request.object.metadata.labels.canuseIPC) }}" +``` + +

+
+ ### To_lower
Expand @@ -2577,6 +2624,57 @@ spec:

+### Trim_prefix + +
Expand +

+ +The `trim_prefix()` filter takes an input string and from it trims the beginning by the second string. For the trim to occur, the input string must begin with the trimmed string. This filter differs from `trim()` in that it only removes a string from the beginning of another. For example, the query `trim_prefix('docker://kubevirt/fedora-cloud-registry-disk-demo','docker://')` will result in the output of `kubevirt/fedora-cloud-registry-disk-demo`. + +The `trim_prefix()` filter can be useful to remove URIs found in container image values referenced by some custom resources, or anywhere else where a more strategic removal of a substring within a parent is required. + +| Input 1 | Input 2 | Output | +|--------------------|--------------------|---------| +| String | String | String | + +**Example:** This policy uses the `trim_prefix()` filter to remove `docker://` from the name of an image in a KubeVirt `DataVolume` custom resource through use of an image extractor. + +```yaml +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: verify-data-volume-image +spec: + background: false + validationFailureAction: Enforce + rules: + - name: verify-data-volume-image + match: + any: + - resources: + kinds: + - DataVolume + imageExtractors: + DataVolume: + - path: /spec/source/registry/url + jmesPath: "trim_prefix(@, 'docker://')" + verifyImages: + - imageReferences: + - "*" + mutateDigest: true + verifyDigest: true + attestors: + - entries: + - keys: + publicKeys: | + -----BEGIN PUBLIC KEY----- + ... + -----END PUBLIC KEY----- +``` + +

+
+ ### Truncate
Expand @@ -2618,7 +2716,7 @@ spec:
Expand

-The `x509_decode()` filter takes in a string which is a PEM-encoded X509 certificate, and outputs a JSON object with the decoded certificate details. It may often be required to first decode a base64-encoded string using [base64_decode()](#base64_decode). This filter can be used to check and validate attributes within a certificate such as subject, issuer, SAN fields, and expiration time. An example of such a decoded object may look like the following: +The `x509_decode()` filter takes in a string which is a PEM-encoded X509 certificate or certificate signing request (CSR), and outputs a JSON object with the decoded details. It may often be required to first decode a base64-encoded string using [base64_decode()](#base64_decode). This filter can be used to check and validate attributes within a certificate or a CSR such as subject, issuer, SAN fields, and expiration time. An example of a decoded certificate may look like the following: ```json { diff --git a/content/en/docs/Writing policies/match-exclude.md b/content/en/docs/Writing policies/match-exclude.md index 6cdc2aab5..13b63fea9 100644 --- a/content/en/docs/Writing policies/match-exclude.md +++ b/content/en/docs/Writing policies/match-exclude.md @@ -1,6 +1,7 @@ --- -title: Select Resources -description: Use `match` and `exclude` to filter and select resources. +title: Selecting Resources +description: > + Identifying and filtering resources for policy evaluation. weight: 20 --- @@ -15,7 +16,7 @@ The `match` and `exclude` clauses have the same structure and can each contain * The following resource filters can be specified under an `any` or `all` clause. -* `resources`: select resources by names, namespaces, kinds, label selectors, annotations, and namespace selectors. +* `resources`: select resources by names, namespaces, kinds, , operations, label selectors, annotations, and namespace selectors. * `subjects`: select users, user groups, and service accounts * `roles`: select namespaced roles * `clusterRoles`: select cluster wide roles @@ -166,11 +167,8 @@ spec: - resources: kinds: - "*" - preconditions: - any: - - key: "{{ request.operation }}" - operator: Equals - value: CREATE + operations: + - CREATE validate: message: "The label `app.kubernetes.io/name` is required." pattern: @@ -227,33 +225,32 @@ spec: - name: check-pod-controller-labels # Each rule matches specific resource described by "match" field. match: - resources: - kinds: # Required, list of kinds - - Deployment - - StatefulSet - # Optional resource names. Supports wildcards (* and ?) - names: - - "mongo*" - - "postgres*" - # Optional list of namespaces. Supports wildcards (* and ?) - namespaces: - - "dev*" - - test - # Optional label selectors. Values support wildcards (* and ?) - selector: - matchLabels: - app: mongodb - matchExpressions: - - {key: tier, operator: In, values: [database]} - # Optional users or service accounts to be matched - subjects: - - kind: User - name: mary@somecorp.com - # Optional roles to be matched - roles: - # Optional clusterroles to be matched - clusterRoles: - - cluster-admin + any: + - resources: + kinds: # Required, list of kinds + - Deployment + - StatefulSet + # Optional resource names. Supports wildcards (* and ?) + names: + - "mongo*" + - "postgres*" + # Optional list of namespaces. Supports wildcards (* and ?) + namespaces: + - "dev*" + - test + # Optional label selectors. Values support wildcards (* and ?) + selector: + matchLabels: + app: mongodb + matchExpressions: + - {key: tier, operator: In, values: [database]} + # Optional users or service accounts to be matched + subjects: + - kind: User + name: mary@somecorp.com + # Optional clusterroles to be matched + clusterRoles: + - cluster-admin ``` {{% alert title="Note" color="info" %}} diff --git a/content/en/docs/Writing policies/mutate.md b/content/en/docs/Writing policies/mutate.md index fc5fbc2ba..7c990e4e5 100644 --- a/content/en/docs/Writing policies/mutate.md +++ b/content/en/docs/Writing policies/mutate.md @@ -1,8 +1,8 @@ --- -title: Mutate Resources +title: Mutate Rules description: > Modify resource configurations. -weight: 30 +weight: 40 --- A `mutate` rule can be used to modify matching resources and is written as either a RFC 6902 JSON Patch or a strategic merge patch. @@ -100,7 +100,8 @@ spec: patchesJson6902: |- - path: "/data" op: add - value: {"ship.properties": "{\"type\": \"starship\", \"owner\": \"utany.corp\"}"} + value: + ship.properties: '{"type": "starship", "owner": "utany.corp"}' ``` This is an example of a patch that removes a label from a Secret: @@ -124,7 +125,7 @@ spec: op: remove ``` -This policy rule adds elements to a list. In this case, it adds a new busybox container and a command. Note that because the `path` statement is a precise schema element, this will only work on a direct Pod and not higher-level objects such as Deployments. +This policy rule adds elements to a list. In this case, it adds a new busybox container and a command. Note that because the `path` statement is a precise schema element, this will only work on a direct Pod and not higher-level objects such as Deployments. Testing the below policy requires setting `spec.automountServiceAccountToken: false` in a Pod. ```yaml apiVersion: kyverno.io/v1 @@ -143,7 +144,9 @@ spec: patchesJson6902: |- - op: add path: "/spec/containers/1" - value: {"name":"busybox","image":"busybox:latest"} + value: + name: busybox + image: busybox:latest - op: add path: "/spec/containers/1/command" value: @@ -161,7 +164,11 @@ mutate: patchesJson6902: |- - op: add path: "/spec/tolerations/-" - value: {"key":"networkzone","operator":"Equal","value":"dmz","effect":"NoSchedule"} + value: + key: networkzone + operator: Equal + value: dmz + effect: NoSchedule ``` JSON Patch uses [JSON Pointer](http://jsonpatch.com/#json-pointer) to reference keys, and keys with tilde (`~`) and forward slash (`/`) characters need to be escaped with `~0` and `~1`, respectively. For example, the following adds an annotation with the key of `config.linkerd.io/skip-outbound-ports` with the value of `"8200"`. @@ -711,18 +718,16 @@ spec: - resources: kinds: - Pod - preconditions: - any: - - key: "{{ request.operation }}" - operator: Equals - value: CREATE + operation: + - CREATE mutate: foreach: - list: "request.object.spec.containers" patchesJson6902: |- - path: /spec/containers/{{elementIndex}}/securityContext op: add - value: {"runAsNonRoot" : true} + value: + runAsNonRoot: true ``` For a complete example of the `patchStrategicMerge` method that mutates the image to prepend the address of a trusted registry, see below. @@ -741,13 +746,9 @@ spec: - resources: kinds: - Pod - preconditions: - all: - - key: "{{request.operation}}" - operator: AnyIn - value: - - CREATE - - UPDATE + operations: + - CREATE + - UPDATE mutate: foreach: - list: "request.object.spec.containers" @@ -818,3 +819,31 @@ spec: op: replace value: "{{ replace_all('{{element}}', '.old.com', '.new.com') }}" ``` + +## GitOps Considerations + +It is very common to use Kyverno in a GitOps approach where policies and/or other resources, which may be affected by those policies, are deployed via a state held in git by a separate tool such as Argo CD or Flux. In the case of mutations, the classic problem which arises is Kyverno (or some other tool) mutates a resource which was created by a GitOps tool. The mutation changes the resource in a way which causes divergence from the state held in git. When this divergence is detected by the GitOps controller, said controller attempts to reconcile the resource to bring it back into alignment with the desired state held in git. This process continues in an endless loop and creates "fighting" between the mutating admission controller and the GitOps controller. This type of fighting is problematic as it increases churn in the cluster, increases resource consumption by the respective tools, and can lead to disruption if it occurs on a large scale. + +While Kyverno can interoperate with GitOps solutions when mutate rules are used, there are a few considerations of which to be aware and some recommended configurations. + +### Flux + +[Flux](https://fluxcd.io/) uses server-side apply along with dry-run mode when calculating a diff to determine if the actual state has diverged from desired state. Because Kyverno supports mutations in dry-run mode, the resource returned to Flux in this mode already includes the result of any would-be mutations. As a result, Flux natively accommodates mutating admission controllers such as Kyverno usually without any modifications needed. It is not necessary to inform Flux of the nature and number of mutations to expect for a given resource. + +However, as dry-run mode causes mutation webhooks to be invoked just as if not in dry-run mode, this produces additional load on Kyverno as it must perform the same mutations every time a diff is calculated. The Flux synchronization interval should be checked and balanced to ensure only the minimal amount of processing overhead is introduced. + +### ArgoCD + +[Argo CD](https://argoproj.github.io/cd) does not currently support server-side apply dry-run mode in its diff calculations like [Flux](#flux) does. While this is currently a [roadmap item](https://github.com/argoproj/argo-cd/issues/11574), it means using Argo CD with Kyverno mutate rules requires some specific configurations. See the [platform notes](/docs/installation/platform-notes/#notes-for-argocd-users) page for general recommendations with Argo CD first. + +In order to use Argo CD with Kyverno, it will require configuring the `Application` custom resource with one or more `ignoreDifferences` entries to [instruct Argo CD](https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/) to ignore the mutations created by Kyverno. Some of these options include `jqPathExpressions`, `jsonPointers`, and `managedFieldsManagers`. For example, if a Kyverno mutate rule is expected to add a label `foo` to all Deployments, the Argo CD `Application` may need a section as follows. + +```yaml +ignoreDifferences: + - group: apps + kind: Deployment + jqPathExpressions: + - .metadata.labels.foo +``` + +It may also be helpful to configure a `managedFieldsManagers` with a value of `kyverno` in the list to instruct Argo CD to allow Kyverno to own the fields it is mutating. diff --git a/content/en/docs/Writing policies/policy-settings.md b/content/en/docs/Writing policies/policy-settings.md index 0b43d8be5..7b52249cf 100644 --- a/content/en/docs/Writing policies/policy-settings.md +++ b/content/en/docs/Writing policies/policy-settings.md @@ -1,22 +1,27 @@ --- title: Policy Settings -description: Common configuration for all rules in a policy. +description: > + Common configuration for all rules in a policy. weight: 10 --- A [policy](/docs/kyverno-policies/) contains one or more rules, and the following common settings which apply to all rules in the policy: -* **applyRules**: States how many of the rules in the parent policy should be applied to a matching resource. Values are `One` and `All` (default). If set to `One`, the first matching rule to be applied will stop further rules from being evaluated. +* **applyRules**: states how many of the rules in the parent policy should be applied to a matching resource. Values are `One` and `All` (default). If set to `One`, the first matching rule to be applied will stop further rules from being evaluated. -* **validationFailureAction**: controls if a validation policy rule failure should block the admission review request (`Enforce`) or allow (`Audit`) the admission review request and report the policy failure in a policy report. Defaults to `Audit`. +* **background**: controls scanning of existing resources to find potential violations and generating Policy Reports. See the documentation [here](/docs/writing-policies/background/). Default to "true". -* **validationFailureActionOverrides**: a ClusterPolicy attribute that specifies `validationFailureAction` Namespace-wise. It overrides `validationFailureAction` for the specified Namespaces. +* **failurePolicy**: defines the API server behavior if the webhook fails to respond. Allowed values are "Ignore" or "Fail". Defaults to "Fail". Additionally, if set to "Ignore" will allow failing calls to image registries to be ignored. This allows for rule types like verifyImages or others which use image data to not block if the registry is temporarily down, useful in situations where images already exist on the nodes. + +* **generateExisting**: applicable to generate rules only. Controls whether Kyverno should evaluate the policy the moment it is created. -* **background**: controls scanning of existing resources to find potential violations and generating Policy Reports. See the documentation [here](https://kyverno.io/docs/writing-policies/background/). Default to "true". +* **mutateExistingOnPolicyUpdate**: applicable to mutate rules which define targets. Controls whether Kyverno should evaluate the policy when it is updated. * **schemaValidation**: controls whether policy validation checks are applied. Defaults to "true". Kyverno will attempt to validate the schema of a policy and fail if it cannot determine it satisfies the OpenAPI schema definition for that resource. Can occur on either validate or mutate policies. Set to "false" to skip schema validation. -* **failurePolicy**: defines the API server behavior if the webhook fails to respond. Allowed values are "Ignore" or "Fail". Defaults to "Fail". Additionally, in 1.8.0, if set to "Ignore" will allow failing calls to image registries to be ignored. This allows for rule types like verifyImages or others which use image data to not block if the registry is temporarily down, useful in situations where images already exist on the nodes. +* **validationFailureAction**: controls if a validation policy rule failure should block the admission review request (`Enforce`) or allow (`Audit`) the admission review request and report the policy failure in a policy report. Defaults to `Audit`. + +* **validationFailureActionOverrides**: a ClusterPolicy attribute that specifies `validationFailureAction` Namespace-wise. It overrides `validationFailureAction` for the specified Namespaces. * **webhookTimeoutSeconds**: specifies the maximum time in seconds allowed to apply this policy. The default timeout is 10s. The value must be between 1 and 30 seconds. diff --git a/content/en/docs/Writing policies/preconditions.md b/content/en/docs/Writing policies/preconditions.md index 168ede917..949716401 100644 --- a/content/en/docs/Writing policies/preconditions.md +++ b/content/en/docs/Writing policies/preconditions.md @@ -1,8 +1,8 @@ --- title: Preconditions description: > - Control policy rule execution based on variables. -weight: 90 + Fine-grained control of policy rule execution based on variables and expressions. +weight: 120 --- Preconditions allow controlling policy rule execution by building expressions based on variable values. While `match` and `exclude` allow filtering requests based on resource and user information, `preconditions` can be used to define custom filters for more granular control of when a rule should be applied. diff --git a/content/en/docs/Writing policies/tips.md b/content/en/docs/Writing policies/tips.md index d99ad594b..ccabb41da 100644 --- a/content/en/docs/Writing policies/tips.md +++ b/content/en/docs/Writing policies/tips.md @@ -1,7 +1,8 @@ --- title: Tips & Tricks -description: Tips and tricks for writing more effective policy. -weight: 120 +description: > + Tips and tricks for writing more effective policy. +weight: 140 --- These are some tips and tricks you can use when putting together your Kyverno policies. diff --git a/content/en/docs/Writing policies/validate.md b/content/en/docs/Writing policies/validate.md index 4f6093df5..d53fd331b 100644 --- a/content/en/docs/Writing policies/validate.md +++ b/content/en/docs/Writing policies/validate.md @@ -1,7 +1,8 @@ --- -title: Validate Resources -description: Check resource configurations for policy compliance. -weight: 50 +title: Validate Rules +description: > + Check resources configurations for policy compliance. +weight: 30 --- Validation rules are probably the most common and practical types of rules you will be working with, and the main use case for admission controllers such as Kyverno. In a typical validation rule, one defines the mandatory properties with which a given resource should be created. When a new resource is created by a user or process, the properties of that resource are checked by Kyverno against the validate rule. If those properties are validated, meaning there is agreement, the resource is allowed to be created. If those properties are different, the creation is blocked. The behavior of how Kyverno responds to a failed validation check is determined by the `validationFailureAction` field. It can either be blocked (`Enforce`) or allowed yet recorded in a [policy report](/docs/policy-reports/) (`Audit`). Validation rules in `Audit` mode can also be used to get a report on matching resources which violate the rule(s), both upon initial creation and when Kyverno initiates periodic scans of Kubernetes resources. Resources in violation of an existing rule placed in `Audit` mode will also surface in an event on the resource in question. diff --git a/content/en/docs/Writing policies/variables.md b/content/en/docs/Writing policies/variables.md index 4c25df358..92daf9f74 100644 --- a/content/en/docs/Writing policies/variables.md +++ b/content/en/docs/Writing policies/variables.md @@ -1,8 +1,8 @@ --- title: Variables description: > - Data-driven policies for reuse and intelligent decision making -weight: 70 + Defining and using variables in policies from multiple sources. +weight: 90 --- Variables make policies smarter and reusable by enabling references to data in the policy definition, the [admission review request](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#webhook-request-and-response), and external data sources like ConfigMaps, the Kubernetes API Server, and even OCI image registries. diff --git a/content/en/docs/Writing policies/verify-images.md b/content/en/docs/Writing policies/verify-images.md index 7533f4420..628e7f6d6 100644 --- a/content/en/docs/Writing policies/verify-images.md +++ b/content/en/docs/Writing policies/verify-images.md @@ -1,7 +1,8 @@ --- -title: Verify Images -description: Check image signatures and add digests -weight: 40 +title: Verify Images Rules +description: > + Check container image signatures and attestations to ensure supply chain security. +weight: 60 --- {{% alert title="Warning" color="warning" %}} @@ -21,10 +22,10 @@ Each rule contains: * One or more image reference patterns to match * Common configuration attributes: - * required: enforces that all matching images are verified - * mutateDigest: converts tags to digests for matching images - * verifyDigest: enforces that digests are used for matching images - * repository: use a different repository for fetching signatures + * `required`: enforces that all matching images are verified + * `mutateDigest`: converts tags to digests for matching images + * `verifyDigest`: enforces that digests are used for matching images + * `repository`: use a different repository for fetching signatures * Zero or more __attestors__ which can be public keys, certificates, and keyless configuration attributes used to identify trust authorities * Zero or more [in-toto attestation](https://github.com/in-toto/attestation/blob/main/spec/README.md) __statements__ to be verified. If attestations are provided, at least one attestor is required. @@ -803,6 +804,34 @@ spec: -----END PUBLIC KEY----- ``` +For Custom Resources which reference container images in a non-standard way, an optional `jmesPath` field may be used to apply a filter to transform the value of the extracted field. For example, in the case of KubeVirt's `DataVolume` custom resource, the fielding referencing the image needing verification is located at `spec.source.registry.url` as seen below. + +```yaml +apiVersion: cdi.kubevirt.io/v1beta1 +kind: DataVolume +metadata: + name: registry-image-datavolume +spec: + source: + registry: + url: "docker://kubevirt/fedora-cloud-registry-disk-demo" + pvc: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 5Gi +``` + +The value of the field contains a prefix of `docker://` which must be removed first. Applying a JMESPath expression in an extractor along with a Kyverno custom filter such as [`trim_prefix()`](/docs/writing-policies/jmespath/#trim_prefix) can be used to provide the container image for Kyverno to verify. + +```yaml +imageExtractors: + DataVolume: + - path: /spec/source/registry/url + jmesPath: "trim_prefix(@, 'docker://')" +``` + ## Special Variables A pre-defined, reserved special variable named `image` is available for use only in verifyImages rules. The following fields are available under the `image` object and may be used in a rule to reference the named fields.