k8schain: does not ignore missing pull secrets like Kubernetes does

[mattmoor]

Describe the bug

If an imagePullSecrets is missing the kubernetes keychain returns the error:

go-containerregistry/pkg/authn/kubernetes/keychain.go

Lines 67 to 70 in 02f47e1

	ps, err := client.CoreV1().Secrets(opt.Namespace).Get(ctx, name, metav1.GetOptions{})
	if err != nil {
	return nil, err
	}

However, in kubernetes these are logged and ignored (TIL!):

For necessary pull secrets, the behavior here is arguably desirable because you get a clear error message to surface to users.

However, if the goal is blind compatibility with Kubernetes semantics, then this manifests errors for unnecessary pull secrets where Kubernetes deploys things just fine!

To Reproduce

1. Create a pod that deploys just fine, and add a non-existent pull secret.
2. Enable sigstore/policy-controller on the namespace and try again
3. 💥

Expected behavior

A clear and concise description of what you expected to happen.

Additional context

I believe that this would affect Knative (cc @dprotaso @evankanderson), Tekton (cc @imjasonh, @vdemeester), I know it affects sigstore/policy-controller (cc @vaikas @hectorj2f). cc @sudo-bmitch @jonjohnsonjr @dlorenc FYI

I could go either way on this, so I figured I'd tag some stakeholders and solicit opinions. 🍿

[dprotaso]

Just as an FYI to others - k8schain is static. So if secrets are missing the resulting chain is a noop and you need to recreate the chain when retrying in order to refetch

[imjasonh]

Just as an FYI to others - k8schain is static. So if secrets are missing the resulting chain is a noop and you need to recreate the chain when retrying in order to refetch

This is true. If this is a problem for anybody we can definitely rework it so the secrets are refetched from a lister, or updated after some expiration. Open to ideas based on usage and needs.