ImagePullSecrets bypassed in multi-tenant clusters

[deepakdeore2004]

Spegel bypasses Kubernetes imagePullSecrets authentication after the first pull. If Namespace A has valid credentials and pulls private-registry.com/secret-app:latest, Spegel caches it. Then Namespace B (with no credentials or expired credentials) can successfully pull the same image from Spegel's P2P cache without any authentication check.

This breaks namespace isolation in multi-tenant clusters where different teams should have separate access to private registries.

Question: Is this a known design tradeoff? Should multi-tenant clusters avoid using Spegel, or is there a roadmap for namespace-aware authorization?

[phillebaba]

I have discussed this a few times in person with people but realize there is no written record of my thoughts.

This is a problem with or without Spegel, the problem is just exacerbated with Spegel. You have the same problem in a multi tenant cluster if tenants share nodes. All cached images on a node are available for other tenants to use if your pod ends up on the node. The solution proposed by some Kubernetes maintainers is to enforce pull policy Always when pull credentials are present. This forces Containerd to make a HEAD request to ensure that the credentials are valid before allowing the use of the cached image.

A major drawback of this solution is that it creates a hard dependency of the upstream registry, if the registry is down you will not be able to use the image even if it is cached. Another is that it is not compatible with Spegel, or any other registry mirror for that matter. Credentials are not passed in mirror requests, meaning the mirror can return a 200 without verifying the credentials.

If you are able to, I recommend using a admission webhook to enforce which registry or registry namespace can be used per namespace. If each tenant has a unique registry or a repository in a shared registry enforcing image references is easy. Just block pod creation using private images that are not allowed for that namespace.

Requests to Spegel to not include any information about Kubernetes namespaces, so we cant determine if an image should be allowed or not. Any solution would require giving Spegel access to the Kubernetes API which is out of scope for Spegel. As there are community solutions readily available, I consider this an enterprise feature which cant be solved in the OSS version.

[phillebaba]

I have updated the documentation with a section about multi-tenant credentials.
https://spegel.dev/docs/usage/multi-tenant-credentials/