Store Kyverno policies as OCI artifacts #19

developer-guy · 2022-05-31T12:55:21Z

Signed-off-by: Batuhan Apaydın batuhan.apaydin@trendyol.com
Co-authored-by: Furkan Türkal furkan.turkal@trendyol.com

JimBugwadia

few comments to discuss! :-)

JimBugwadia · 2022-06-22T02:58:07Z

proposals/store_kyverno_policies_in_oci_registries.md

+To push a Kyverno policy to a registry, the command would be:
+
+```shell
+kyverno push –policy <policy> <img>


A suggestion would be to model this similar to Tekton bundles:

❯ tkn bundle --help Manage Tekton Bundles Usage: tkn bundle [flags] tkn bundle [command] Aliases: bundle, tkb, bundles Available Commands: list List and print a Tekton bundle's contents push Create or replace a Tekton bundle Flags: -h, --help help for bundle -C, --no-color disable coloring (default: false) Use "tkn bundle [command] --help" for more information about a command.

We can do this via commands:

kyverno oci list ghcr.io/org/policy-set1:v1

kyverno oci push ghcr.io/org/policy-set1:v1 <YAMLs>

I loved it. We can store more than one policy within the layers of an image quickly, would you mind adding this into a KDP @JimBugwadia 😍

I've updated the KDP according to your feedbacks

JimBugwadia · 2022-06-22T03:18:33Z

proposals/store_kyverno_policies_in_oci_registries.md

+To pull a policy from a registry, the command would be:
+
+```shell
+kyverno pull <img> –o <dir>


Instead of doing a pull via a CLI, why not allow policies and other resources to be retrieved and applied in-memory from an OCI registry?

We can manage which policies should be loaded via a new resource:

apiVersion: kyverno.io/v1 kind: PolicySets spec: - name: pod-security oci: ghcr.io/myorg/psp:1.0 - name: best-practices oci: ghcr.io/myorg/best-practices:2.1 - name: multi-tenancy oci: ghcr.io/myorg/multi-tenancy:1.0 policies: - add-network-policy - add-quota

Awesome ideas🤩 but this KDP's context is a bit limited to kyverno CLI, I'd better discuss this in another KDP because I thought that it is nice to experience OCI Artifacts in Kyverno CLI first, then apply the same pattern to Kyverno's core, and once we get there maybe we could open another KDP to further discuss how it can be usable for Kyverno's core, WDYT?

@developer-guy - we can certainly phase this out.

However, what in your opinion would be the use case / benefit for users to manage policies in an OCI registry via the CLI? What I was thinking is that the main value of storing in an OCI registry is to make the policies easier to apply. Perhaps there are other use cases that you had in mind?

As part of this, will we update the CLI apply and test commands to take an OCI reference instead of a Git URL or folder?

will we update the CLI apply and test commands to take an OCI reference instead of a Git URL or folder?

Exactly, we should expand the tests to use OCI references as an input in addition to Git URL or folder, does that make sense?

However, what in your opinion would be the use case / benefit for users to manage policies in an OCI registry via the CLI? What I was thinking is that the main value of storing in an OCI registry is to make the policies easier to apply. Perhaps there are other use cases that you had in mind?

My answer would be these two things: portability and security.

From the security perspective:

If I let people store Kyverno policies in OCI registries as OCI images, people can use existing Sigstore tooling to prove their authenticity by signing them, and we can verify that signature before applying them as the Flux team planned.

From the portability perspective:
People can quickly share policies with others or copy them between registries.

stefanprodan · 2022-08-15T10:49:43Z

FYI: Starting with Flux v0.32, Kyverno users can use OCI as the desired state for cluster policies.

Example workflow

Publish polices

git clone https://github.com/kyverno/policies.git
cd policies

flux push oci://ghcr.io/org/policies/cert-manager:v1.0.0 \
  --path="./cert-manager" \
  --source="$(git config --get remote.origin.url)" \
  --revision="$(git branch --show-current)/$(git rev-parse HEAD)"

Reconcile polices

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: cert-manager-policies
  namespace: flux-system
spec:
  interval: 5m
  url: oci://ghcr.io/org/policies/cert-manager
  ref:
    semver: "1.x" # automated upgrade to latest stable release
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: cert-manager-policies
  namespace: flux-system
spec:
  sourceRef:
    kind: OCIRepository
    name: cert-manager-policies
  interval: 60m
  retryInterval: 5m
  path: ./
  prune: true
  wait: true
  timeout: 2m
  patches: # enforce all policies
    - patch: |
        - op: add
          path: /spec/validationFailureAction
          value: enforce
      target:
        kind: ClusterPolicy

JimBugwadia · 2022-08-21T19:26:38Z

Here is the Flux RFC for the feature described above:

https://github.com/fluxcd/flux2/tree/main/rfcs/0003-kubernetes-oci

The requirements we will want to consider for Kyverno are:

Kyverno CLI support to create policy sets in an OCI registry
Kyverno CLI support to apply / test policy sets from an OCI registry
Kyverno Webhook support to apply policy sets from an OCI registry

Signed-off-by: Batuhan Apaydın <batuhan.apaydin@trendyol.com> Co-authored-by: Furkan Türkal <furkan.turkal@trendyol.com> Signed-off-by: Batuhan Apaydın <batuhan.apaydin@trendyol.com>

developer-guy · 2022-09-02T08:51:05Z

proposals/store_kyverno_policies_in_oci_registries.md

+To pull a policy from a registry, the command would be:
+
+```shell
+kyverno pull <img> –o <dir>


will we update the CLI apply and test commands to take an OCI reference instead of a Git URL or folder?

Exactly, we should expand the tests to use OCI references as an input in addition to Git URL or folder, does that make sense?

However, what in your opinion would be the use case / benefit for users to manage policies in an OCI registry via the CLI? What I was thinking is that the main value of storing in an OCI registry is to make the policies easier to apply. Perhaps there are other use cases that you had in mind?

My answer would be these two things: portability and security.

From the security perspective:

If I let people store Kyverno policies in OCI registries as OCI images, people can use existing Sigstore tooling to prove their authenticity by signing them, and we can verify that signature before applying them as the Flux team planned.

From the portability perspective:
People can quickly share policies with others or copy them between registries.

developer-guy · 2022-09-02T08:51:30Z

proposals/store_kyverno_policies_in_oci_registries.md

+To push a Kyverno policy to a registry, the command would be:
+
+```shell
+kyverno push –policy <policy> <img>


I've updated the KDP according to your feedbacks

developer-guy · 2022-09-02T08:53:20Z

proposals/store_kyverno_policies_in_oci_registries.md

+
+Two implementation points exist in the Kyverno space that can be used to support pulling/pushing policies based on OCI Artifacts specification: Kyverno itself and its Command Line Interface (CL)I. First, we will start by introducing new subcommands for pulling/pushing policies using Kyverno CLI and then to Kyverno itself.
+
+One of the initial steps that OCI artifact authors need to decide upon is the mediaTypes that we will use to store Kyverno policies. According to the OCI Artifact Authors specification format defined for new artifact types, for the Kyverno policy layer, which we use to keep the policy itself, it should be represented as application/vnd.cncf.kyverno.policy.layer.v1+yaml, and application/vnd.cncf.kyverno.config.v1+json which will be used to store the policy config.


I've also added new mediaTypes for addressing a bunch of other use cases, here is the complete list:

# for policies in YAML format application/vnd.cncf.kyverno.policy.layer.v1+yaml # for policies in JSON format application/vnd.cncf.kyverno.policy.layer.v1+json # for storing multiple policies in the same layer application/vnd.cncf.kyverno.policy.layer.v1.tar+gzip

@developer-guy - its not clear why we need new mediaTypes for Kyverno.

Is there an existing media type that can be cleanly and securely reused?

Is there an existing media type that can be cleanly and securely reused?

Of course, we can use the existing mediaTypes defined in the OCI image-spec.

its not clear why we need new mediaTypes for Kyverno.

The intent was behind defining unique mediaTypes well defined in the OCI Artifacts repository.

TLDR;
Defining a unique artifact allows various tools to know how to uniquely work with the type. It allows a registry to display the type and tooling, such as vulnerability scanners, a means to know if and how they should interact with the contents.

But we can continue with existing mediaTypes if this is the case, that would not be a problem. @JimBugwadia

developer-guy · 2022-09-02T08:54:12Z

kindly ping @chipzoller @JimBugwadia @samj1912

developer-guy · 2022-10-25T06:18:32Z

we can close this one as we merged the PR of the implementation of this KDP ☝️

/cc @eddycharly @chipzoller

JimBugwadia · 2022-09-07T22:57:18Z

proposals/store_kyverno_policies_in_oci_registries.md

+
+OCI Artifacts provides a [reference guide](https://github.com/opencontainers/artifacts/blob/main/artifact-authors.md) for authors looking to introduce a new artifact type. A unique artifact type is similar to defining a file extension. Defining a unique artifact allows various tools to work with the type uniquely. For example, it will enable a registry to display the type and tooling, such as vulnerability scanners, to know if and how they should interact with the contents. To learn more about defining your artifact type, please [see](https://github.com/opencontainers/artifacts/blob/main/artifact-authors.md#defining-a-unique-artifact-type).
+
+     “application/vnd.[org/company].[objectType].[optional sub type].config.[version]+json”


Is there an existing media type that can be reused for Kubernetes resources? Do we need a new media type registered for Kyverno policies?

JimBugwadia · 2022-09-07T22:58:48Z

proposals/store_kyverno_policies_in_oci_registries.md

+
+> Issue [#3154](https://github.com/kyverno/kyverno/issues/3154)
+
+Two implementation points exist in the Kyverno space that can be used to support pulling/pushing policies based on OCI Artifacts specification: Kyverno itself and its Command Line Interface (CL)I. First, we will start by introducing new subcommands for pulling/pushing policies using Kyverno CLI and then to Kyverno itself.


maybe state as:

the Kyverno webhook which acts as an admission controller and the Kyverno Command Line Interface (CLI).

JimBugwadia · 2022-09-07T23:01:28Z

proposals/store_kyverno_policies_in_oci_registries.md

+
+> Issue [#3154](https://github.com/kyverno/kyverno/issues/3154)
+
+Two implementation points exist in the Kyverno space that can be used to support pulling/pushing policies based on OCI Artifacts specification: Kyverno itself and its Command Line Interface (CL)I. First, we will start by introducing new subcommands for pulling/pushing policies using Kyverno CLI and then to Kyverno itself.


In the Webhook, we fetch and cache policies from the API server. To implement this to use an OCI registry, we will need:

A CRD or configuration that specifies which OCI artifacts (policies) should be fetched and cached

Some configuration options to check for updates etc.

The implementation can then load the same policy cache.

The background reporter module will also need to fetch policies.

JimBugwadia · 2022-10-25T10:54:56Z

@developer-guy - sounds good! We can create a new KDP for implementation in the webhook. Thanks for the great work on this!

JimBugwadia · 2022-10-25T10:55:02Z

Need one more maintainer to sign off on the KDP.

cc: @realshuting @chipzoller @eddycharly @prateekpandey14 @vyankyGH

realshuting

Great work @developer-guy !

chipzoller mentioned this pull request Jun 1, 2022

to be able to pull/push and sign/verify kyverno policies by storing them in OCI registries kyverno/kyverno#3154

Closed

chipzoller changed the title ~~feature(oci): new kdp added, store policies in oci registries~~ Store Kyverno policies as OCI artifacts Jun 1, 2022

developer-guy force-pushed the main branch from 22f7e71 to 3157ddb Compare June 14, 2022 13:10

JimBugwadia reviewed Jun 22, 2022

View reviewed changes

developer-guy force-pushed the main branch from 3157ddb to 4da8ea3 Compare September 2, 2022 08:53

feature(oci): new kdp added

d9fa433

Signed-off-by: Batuhan Apaydın <batuhan.apaydin@trendyol.com> Co-authored-by: Furkan Türkal <furkan.turkal@trendyol.com> Signed-off-by: Batuhan Apaydın <batuhan.apaydin@trendyol.com>

developer-guy force-pushed the main branch from 4da8ea3 to d9fa433 Compare September 2, 2022 08:53

developer-guy commented Sep 2, 2022

View reviewed changes

developer-guy requested a review from JimBugwadia September 2, 2022 08:53

developer-guy mentioned this pull request Oct 17, 2022

feat: oci pull/push support for policie(s) kyverno/kyverno#5026

Merged

9 tasks

JimBugwadia approved these changes Oct 25, 2022

View reviewed changes

realshuting approved these changes Oct 26, 2022

View reviewed changes

realshuting merged commit 2bc6c25 into kyverno:main Oct 26, 2022

developer-guy mentioned this pull request Nov 6, 2022

[Feature] register Kyverno policy mediaTypes to iana mediaTypes kyverno/kyverno#5237

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Store Kyverno policies as OCI artifacts #19

Store Kyverno policies as OCI artifacts #19

developer-guy commented May 31, 2022

JimBugwadia left a comment

JimBugwadia Jun 22, 2022 •

edited

Loading

developer-guy Jun 22, 2022

developer-guy Sep 2, 2022

JimBugwadia Jun 22, 2022

developer-guy Jun 22, 2022

JimBugwadia Jun 30, 2022 •

edited

Loading

developer-guy Sep 2, 2022

stefanprodan commented Aug 15, 2022 •

edited

Loading

JimBugwadia commented Aug 21, 2022 •

edited

Loading

developer-guy Sep 2, 2022

developer-guy Sep 2, 2022

developer-guy Sep 2, 2022

JimBugwadia Sep 7, 2022

developer-guy Sep 8, 2022

developer-guy commented Sep 2, 2022

developer-guy commented Oct 25, 2022

JimBugwadia Sep 7, 2022

JimBugwadia Sep 7, 2022

JimBugwadia Sep 7, 2022

JimBugwadia commented Oct 25, 2022

JimBugwadia commented Oct 25, 2022

realshuting left a comment


		Two implementation points exist in the Kyverno space that can be used to support pulling/pushing policies based on OCI Artifacts specification: Kyverno itself and its Command Line Interface (CL)I. First, we will start by introducing new subcommands for pulling/pushing policies using Kyverno CLI and then to Kyverno itself.

		One of the initial steps that OCI artifact authors need to decide upon is the mediaTypes that we will use to store Kyverno policies. According to the OCI Artifact Authors specification format defined for new artifact types, for the Kyverno policy layer, which we use to keep the policy itself, it should be represented as application/vnd.cncf.kyverno.policy.layer.v1+yaml, and application/vnd.cncf.kyverno.config.v1+json which will be used to store the policy config.


		OCI Artifacts provides a [reference guide](https://github.com/opencontainers/artifacts/blob/main/artifact-authors.md) for authors looking to introduce a new artifact type. A unique artifact type is similar to defining a file extension. Defining a unique artifact allows various tools to work with the type uniquely. For example, it will enable a registry to display the type and tooling, such as vulnerability scanners, to know if and how they should interact with the contents. To learn more about defining your artifact type, please [see](https://github.com/opencontainers/artifacts/blob/main/artifact-authors.md#defining-a-unique-artifact-type).

		“application/vnd.[org/company].[objectType].[optional sub type].config.[version]+json”


		> Issue [#3154](https://github.com/kyverno/kyverno/issues/3154)

		Two implementation points exist in the Kyverno space that can be used to support pulling/pushing policies based on OCI Artifacts specification: Kyverno itself and its Command Line Interface (CL)I. First, we will start by introducing new subcommands for pulling/pushing policies using Kyverno CLI and then to Kyverno itself.

Store Kyverno policies as OCI artifacts #19

Store Kyverno policies as OCI artifacts #19

Conversation

developer-guy commented May 31, 2022

JimBugwadia left a comment

Choose a reason for hiding this comment

JimBugwadia Jun 22, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JimBugwadia Jun 30, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stefanprodan commented Aug 15, 2022 • edited Loading

Example workflow

Publish polices

Reconcile polices

JimBugwadia commented Aug 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

developer-guy commented Sep 2, 2022

developer-guy commented Oct 25, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JimBugwadia commented Oct 25, 2022

JimBugwadia commented Oct 25, 2022

realshuting left a comment

Choose a reason for hiding this comment

JimBugwadia Jun 22, 2022 •

edited

Loading

JimBugwadia Jun 30, 2022 •

edited

Loading

stefanprodan commented Aug 15, 2022 •

edited

Loading

JimBugwadia commented Aug 21, 2022 •

edited

Loading