Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kubewarden policy to validate cosign images #1183

Closed
1 of 3 tasks
mudler opened this issue Mar 18, 2022 · 12 comments
Closed
1 of 3 tasks

Kubewarden policy to validate cosign images #1183

mudler opened this issue Mar 18, 2022 · 12 comments
Assignees
Labels
kind/enhancement New feature or request kind/security
Projects

Comments

@mudler
Copy link
Contributor

mudler commented Mar 18, 2022

In order to validate images used for upgrades in kubernetes, we can have a kubewarden policy to validate cosign images and prevent upgrades from being pulled and hitting nodes.

Ping Kubewarden team to see if they have already a policy for it and we can just re-use it (and document it)

Action Items

  • See if there is a Kubewarden policy for validating cosign-ed images and see if fits our case, or either create a new one.
  • QA and check if fits our scenarios
  • Update our documentation with a practical example of building an image, signing it, and upgrading it in kubernetes with the kubewarden policy. Showing off also how an unsigned one would fail miserably in upgrading is a plus!
@mudler mudler added the kind/enhancement New feature or request label Mar 18, 2022
@mudler mudler self-assigned this Mar 18, 2022
@mudler mudler added this to 💡 Untriaged in Releases Mar 18, 2022
@mudler mudler removed their assignment Mar 18, 2022
@mudler mudler moved this from 💡 Untriaged to 🗳️ To Do in Releases Mar 22, 2022
@mudler
Copy link
Contributor Author

mudler commented Mar 24, 2022

I've pinged the kubewarden team as I could not find any policy for cosign in https://hub.kubewarden.io/, if that's the case we should implement our own policy

@mudler
Copy link
Contributor Author

mudler commented Mar 24, 2022

I've checked in with @flavio and at the moment this doesn't seems to be technically possible with Kubewarden but it's in the roadmap.

Let's postpone this as there is no pressure for it right now, in the case we need a temporary solution to verify cosign images we can write a custom admission controller for the purpose or use https://artifacthub.io/packages/helm/sigstore/cosigned and replace it later with a kubewarden policy.

@mudler mudler added the status/blocked Issue depends on another one label Mar 24, 2022
@mudler mudler moved this from 🗳️ To Do to 💡 Untriaged in Releases Mar 24, 2022
@mudler mudler moved this from 💡 Untriaged to 🗳️ To Do in Releases Mar 25, 2022
@mudler mudler moved this from 🗳️ To Do to ❆ Icebox in Releases Mar 29, 2022
@mudler mudler removed the status/blocked Issue depends on another one label Jun 7, 2022
@mudler
Copy link
Contributor Author

mudler commented Jun 7, 2022

This seems to be possible now

@flavio
Copy link
Member

flavio commented Jun 7, 2022

This is the kubewarden policy that implements image verification: https://github.com/kubewarden/verify-image-signatures

Feel free to reach out if something is missing or if you want to propose improvements

@mudler mudler self-assigned this Jun 8, 2022
@mudler
Copy link
Contributor Author

mudler commented Jun 9, 2022

From a preliminary test the policy seems sufficient as it is gating upgrades. I'm checking out now if I can make an upgrade going through it and documenting all the steps. So far looks promising

@mudler
Copy link
Contributor Author

mudler commented Jun 15, 2022

Bug was found along the way. Blocked by kubewarden/verify-image-signatures#20

@mudler mudler added the status/blocked Issue depends on another one label Jun 15, 2022
@mudler mudler removed the status/blocked Issue depends on another one label Jul 4, 2022
@mudler
Copy link
Contributor Author

mudler commented Jul 4, 2022

Seems to be fixed upstream now, having a look at it again

@mudler
Copy link
Contributor Author

mudler commented Jul 6, 2022

With latest kubewarden I get a strange error on the validation policy pod:


 2022-07-06T09:08:50.479016Z ERROR validation{host="policy-server-default-8b84b74c8-2skhx" policy_id="namespaced-system-upgrade-gate-upgrades" kind="Pod" kind_group="" kind_version="v1" name="apply-os-upgrade-on-localhost-with-4d7cde11576e7ba6998d35-rdbpz" namespace="system-upgrade" operation="CREATE" request_uid="1c691265-11d1-4fb3-83fb-8d8c1b27874f" resource="pods" resource_group="" resource_version="v1" subresource=""}:policy_eval:validate{self=PolicyEvaluator {  id: "namespaced-system-upgrade-gate-upgrades", settings: {"modifyImagesWithDigest": Bool(true), "signatures": Array([Object({"image": String("quay.io/c3os/*"), "keyless": Array([Object({"issuer": String("https://token.actions.githubusercontent.com"), "subject": String("https://github.com/c3os-io/c3os/.github/workflows/image.yaml@refs/heads/master")})])})])} }}: policy_evaluator::runtimes::wapc: callback evaluation failed policy_id=1 binding="kubewarden" operation="v1/verify" error="Cannot pull quay.io/c3os/c3os:sha256-4a6f272b46db9a99998a5eeb335c061d106d129abe9194f3396641ef1c04260d.sig: error sending request for url (https://cdn02.quay.io/sha256/66/66e3dfcad3ef0692442eb17cd9eaccee2599bef6406ec3fe3a7711ad1381a26c?Expires=1657099125&Signature=P3oFjZ48~0Y8uR07LXYM9-ZjMSmxiNaecTTDLowy98jWCinvi5hs5-zBfVACAx9KVwwZoUgrfKyetmEerpliU4HsvObvuPDXOjFVlPRPE4TZ-DY4Y6SSg5gFJAkDcCFLirwkRM3dVw0d5s~5zU7~cn7AQ2~GMgsWoa0XRHhctJP4Ovv3F3UQSl19NQsCeKqbzz-pDWUgZCEtiavZru2dgrNfpj4EbkUWWgVfiUgIGbz2FS55csAcPol3Q3EzilBK7UxLoFOJVutDs5c7uX5HXYFoRCnLYwKKl4jgPCIFtI7nZZCjGTXlkcEKhi89FWLtHkRti1qZG57x~nf0eb5~lw__&Key-Pair-Id=APKAJ67PQLWGCSP66DGA): error trying to connect: dns error: failed to lookup address information: Try again" 

Weirdly enough, dns resolution seem to work just fine in the cluster.

When I describe the job that creates the pod, I see in the events:

  Warning  FailedCreate  10s (x2 over 20s)  job-controller  Error creating: admission webhook "namespaced-system-upgrade-gate-upgrades.kubewarden.admission" denied the request: Request rejected by policy namespaced-system-upgrade-gate-upgrades. The policy attempted to mutate the request, but it is currently configured to not allow mutation

Digging still 👀 ...

@viccuad
Copy link
Member

viccuad commented Jul 21, 2022

The DNS resolution failure seems weird to be fair. I wonder if it is a fluke.

We just released v0.1.5 of the policy. It has a new github_actions signature setting, that is more secure (prevents about attackers consuming GHA reusable workflows and passing signatures as them) and also simpler to use (blog post in kubewarden.io/blog)

The policy with this settings should work:

apiVersion: policies.kubewarden.io/v1alpha2
kind: ClusterAdmissionPolicy
metadata:
  name: verify-image-signatures-policy
spec:
  module: registry://ghcr.io/kubewarden/policies/verify-image-signatures:v0.1.5
  rules:
  - apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["pods"]
    operations:
    - CREATE
    - UPDATE
  mutating: true
  settings:
    signatures:
        - image: "quay.io/c3os/*"
          github_actions: 
              owner: "c3os-io"
              repo: "c3os" # optional

The second error, the mutation one, may have been because the ClusterAdmissionPolicy was missing .spec.mutating: true.
By default, the policy will validate the container tag and by default mutate the container definition to append the digest of the verified tag (digests are immutable, tags are mutable).

If one doesn't want the policy to mutate, they can set .spec.settings.modifyImagesWithDigest: false (default is true), and then it is irrelevant if the policy has .spec.mutating: true.

@viccuad
Copy link
Member

viccuad commented Jul 21, 2022

Just tried with a different policy, and I get the dns error too:

2022-07-21T09:43:13.952451Z ERROR policy_server: error while downloading policy clusterwide-psp-capabilities from registry://ghcr.io/kubewarden/policies/capabilities-psp:v0.1.9: the policy registry://ghcr.io/kubewarden/policies/capabilities-psp:v0.1.9 could not be downloaded due to error: error sending request for url (https://ghcr.io/v2/): error trying to connect: dns error: failed to lookup address information: Try again

I will look into it.

@viccuad
Copy link
Member

viccuad commented Jul 21, 2022

The DNS error seems to be here because I started the cluster with VPN enabled, and then took it off (thanks to @raulcabello for the hint). I wonder if that was it too in your side @mudler.

@frelon
Copy link
Contributor

frelon commented May 12, 2023

Elemental-toolkit does not provide images anymore, closing this.

@frelon frelon closed this as not planned Won't fix, can't repro, duplicate, stale May 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/enhancement New feature or request kind/security
Projects
No open projects
Releases
❆ Icebox
Development

No branches or pull requests

5 participants