[Bug] [CLI] Verify Image test not working for private registry

[KavyaShree25]

Kyverno CLI Version

1.12.4

Description

I m writing a test for verify image policy with private registry and defining context variable from configmap. But Im unable to run the tests. Find the steps below

Steps to reproduce

1. Created a verify image policy with private registry and a context defined
2. Created a workflow for running the unit tests
3. Steps in the workflow
   1. Login to private registry
   2. Installing cosign
   3. Installing kyverno cli
   4. Running the kyverno tests by defining the test as below
      1. kyverno-test.yaml file had the defined tests
      2. resource file had the pods to test
      3. variables.yaml file had the context variable substitution that is the public key

Expected behavior

The registry should be accessed and the policy should be tested

Screenshots

No response

Kyverno logs

cosign "msg"="image verification failed" "error"="no matching signatures:\nno valid tlog entries found unable to fetch Rekor public keys: updating local metadata and targets: error updating to TUF remote mirror: invalid key\nremote status:{\n\t\"mirror\": \"https://sigstore-tuf-root.storage.googleapis.com\",\n\t\"metadata\": {\n\t\t\"root.json\": {\n\t\t\t\"version\": 9,\n\t\t\t\"len\": 6766,\n\t\t\t\"expiration\": \"12 Sep 24 06:53 UTC\",\n\t\t\t\"error\": \"\"\n\t\t},\n\t\t\"snapshot.json\": {\n\t\t\t\"version\": 148,\n\t\t\t\"len\": 2300,\n\t\t\t\"expiration\": \"30 Jul 24 16:06 UTC\",\n\t\t\t\"error\": \"\"\n\t\t},\n\t\t\"targets.json\": {\n\t\t\t\"version\": 9,\n\t\t\t\"len\": 5478,\n\t\t\t\"expiration\": \"12 Sep 24 06:13 UTC\",\n\t\t\t\"error\": \"\"\n\t\t},\n\t\t\"timestamp.json\": {\n\t\t\t\"version\": 201,\n\t\t\t\"len\": 719,\n\t\t\t\"expiration\": \"16 Jul 24 16:06 UTC\",\n\t\t\t\"error\": \"\"\n\t\t}\n\t}\n}\n no valid tlog entries found unable to fetch Rekor public keys: updating local metadata and targets: error updating to TUF remote mirror: invalid key\nremote status:{\n\t\"mirror\": \"https://sigstore-tuf-root.storage.googleapis.com\",\n\t\"metadata\": {\n\t\t\"root.json\": {\n\t\t\t\"version\": 9,\n\t\t\t\"len\": 6766,\n\t\t\t\"expiration\": \"12 Sep 24 06:53 UTC\",\n\t\t\t\"error\": \"\"\n\t\t},\n\t\t\"snapshot.json\": {\n\t\t\t\"version\": 148,\n\t\t\t\"len\": 2300,\n\t\t\t\"expiration\": \"30 Jul 24 16:06 UTC\",\n\t\t\t\"error\": \"\"\n\t\t},\n\t\t\"targets.json\": {\n\t\t\t\"version\": 9,\n\t\t\t\"len\": 5478,\n\t\t\t\"expiration\": \"12 Sep 24 06:13 UTC\",\n\t\t\t\"error\": \"\"\n\t\t},\n\t\t\"timestamp.json\": {\n\t\t\t\"version\": 201,\n\t\t\t\"len\": 719,\n\t\t\t\"expiration\": \"16 Jul 24 16:06 UTC\",\n\t\t\t\"error\": \"\"\n\t\t}\n\t}\n}"

Slack discussion

No response

Troubleshooting

• I have read and followed the troubleshooting guide.
• I have searched other issues in this repository and mine is not recorded.

[welcome]

Thanks for opening your first issue here! Be sure to follow the issue template!

[KavyaShree25]

Team any update on the bug

[vishal-chdhry]

@KavyaShree25 can you share the policy used?

[vishal-chdhry]

This bug seems to be because moved to a new TUF trust root for Sigstore after v2.2.0 (sigstore/cosign#3614), but 1.12.4 uses v2.2.4 which is supported.

[vishal-chdhry]

could not replicate the issue on main

$ kyverno version
Version: (devel)
Time: 2024-07-12T08:42:47Z
Git commit ID: 484e9aab3e9e2bf53ec939f373a625f5fc799049

Test:

$ kyverno test ./.manifest -v 3 --registry
Loading test  ( .manifest/kyverno-test.yaml ) ...
  Loading values/variables ...
  Loading policies ...
  Loading resources ...
  Loading exceptions ...
  Applying 1 policy to 1 resource ...
2024-07-16T15:12:16+05:30       INFO    engine.verify   internal/imageverifier.go:279   cache entry not found   {"policy.name": "test-private", "policy.namespace": "", "policy.apply": "All", "new.kind": "Pod", "new.namespace": "default", "new.name": "test-private-pod", "rule.name": "test-private", "namespace": "", "policy": "test-private", "ruleName": "test-private", "imageRef": "ghcr.io/vishal-chdhry/private-test-image:latest"}
2024-07-16T15:12:16+05:30       INFO    engine.verify   internal/imageverifier.go:335   verifying image signatures      {"policy.name": "test-private", "policy.namespace": "", "policy.apply": "All", "new.kind": "Pod", "new.namespace": "default", "new.name": "test-private-pod", "rule.name": "test-private", "image": "ghcr.io/vishal-chdhry/private-test-image:latest", "attestors": 1, "attestations": 0}
2024-07-16T15:12:18+05:30       DEBUG   cosign  cosign/cosign.go:71     verified image  {"count": 1, "bundleVerified": true}
2024-07-16T15:12:18+05:30       INFO    engine.verify   internal/imageverifier.go:517   image attestors verification succeeded  {"policy.name": "test-private", "policy.namespace": "", "policy.apply": "All", "new.kind": "Pod", "new.namespace": "default", "new.name": "test-private-pod", "rule.name": "test-private", "verifiedCount": 1, "requiredCount": 1}
  Checking results ...

│────│──────────────│──────────────│──────────────────────│────────│────────│
│ ID │ POLICY       │ RULE         │ RESOURCE             │ RESULT │ REASON │
│────│──────────────│──────────────│──────────────────────│────────│────────│
│ 1  │ test-private │ test-private │ Pod/test-private-pod │ Pass   │ Ok     │
│ 2  │ test-private │ test-private │ Pod/test-private-pod │ Pass   │ Ok     │
│────│──────────────│──────────────│──────────────────────│────────│────────│


Test Summary: 2 tests passed and 0 tests failed

Policy:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: test-private
spec:
  background: false
  failurePolicy: Fail
  rules:
  - match:
      any:
      - resources:
          kinds:
          - Pod
    name: test-private
    verifyImages:
    - attestors:
      - entries:
        - keyless:
            subject: vishal.choudhary@nirmata.com
            issuer:  https://accounts.google.com
            rekor:
              url: https://rekor.sigstore.dev
      imageReferences:
      - "*"
  validationFailureAction: Enforce
  webhookTimeoutSeconds: 30

[kapistka]

We have the same problem.
Kyverno v1.12.4, airgapped environment

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  annotations:
    ###workaround https://github.com/kyverno/kyverno/issues/10144
    pod-policies.kyverno.io/autogen-controllers: DaemonSet,Job,ReplicaSet,ReplicationController
    ###
    policies.kyverno.io/category: Software Supply Chain Security
    policies.kyverno.io/minversion: 1.7.0
    policies.kyverno.io/severity: medium
    policies.kyverno.io/subject: Pod
    policies.kyverno.io/title: Verify Image
  labels:
    app.kubernetes.io/component: kyverno
    app.kubernetes.io/instance: kyverno-policies
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kyverno-policies
    app.kubernetes.io/part-of: kyverno-policies
    app.kubernetes.io/version: 3.2.3
    helm.sh/chart: kyverno-policies-3.2.3
  name: required-verify-image
spec:
  admission: true
  background: false
  failurePolicy: Ignore
  rules:
  - match:
      any:
      - resources:
          kinds:
          - Pod
    name: verify-image
    skipBackgroundRequests: true
    verifyImages:
    - attestors:
      - count: 1
        entries:
        - keys:
            publicKeys: |-
              -----BEGIN PUBLIC KEY-----
              ...
              -----END PUBLIC KEY-----
            rekor:
              ignoreTlog: true
            signatureAlgorithm: sha256
      imageReferences:
      - '*'
      imageRegistryCredentials:
        secrets:
        - regcred
      mutateDigest: false
      required: false
      useCache: true
      verifyDigest: false
  validationFailureAction: Audit
  webhookTimeoutSeconds: 10

and default values:

  tuf:
    enabled: false

We get
failed to verify image ---: .attestors[0].entries[0].keys: failed to load CTLogs public keys: updating local metadata and targets: error updating to TUF remote mirror: tuf: failed to download 10.root.json: Get "https://tuf-repo-cdn.sigstore.dev/10.root.json": dial tcp A.B.C.D:443: i/o timeout remote status:{ "mirror": "https://tuf-repo-cdn.sigstore.dev", "metadata": {} }

[kapistka]

I withdraw the question. It's the right thing for me to do:

              entries:
                - keys:
                    ctlog:
                      ignoreSCT: true
                      pubkey: >-
                        -----BEGIN PUBLIC KEY-----
                        -----END PUBLIC KEY-----
                    publicKeys: >-
                      -----BEGIN PUBLIC KEY-----
                      -----END PUBLIC KEY-----
                    rekor:
                      ignoreTlog: true
                      pubkey: >-
                        -----BEGIN PUBLIC KEY-----
                        -----END PUBLIC KEY-----
                    signatureAlgorithm: sha256

The only thing that is stated in the documentation is this: pubKey. But in the schema like this: pubkey

[chipzoller]

Would you please create an issue (or, even better, a PR) to the documentation at kyverno/website so we can fix?