Mutation to replace image registry

[ritazh]

    Another scenario  for a more dynamic mutation is replacing image registry per cluster env. 

For instance, you define kube manifests once, let’s say it uses image grafana/grafana .

Based on the kube cluster it deploys to, mutation webhook can change/prepend the image registry to use private one.

Alternatively user would deploy custom mutation webhook or worst case, clobber the deployment manifests for each env.

This is highly desirable and kyverno policy engine has lot of maturity on it.

Originally posted by @tahirraza in #1134 (comment)

[tahirraza]

Thank you for creating the issue. I'd be happy to do jump in as needed.

[tahirraza]

Slack thread: https://openpolicyagent.slack.com/archives/CDTN970AX/p1667097854495169

[davis-haba]

A change to support this is in flight: #2429

[tahirraza]

Above PR seems merged. What's ETA for this in the future release ?

[ctml91]

Is it possible to use AssignImage, to replace the registry using a condition? For example, I want to replace any container image using docker.io to internal-registry.io. With the current implementation, my understanding it's all or nothing so any pod & container that is matched would get replaced, so it is not possible to set this type of condition? I do not want to replace registries other than docker.io, and do not want to define a hundred AssignImage instances to select specific containers/pods.

It would be nice to have some condition tests, similar to Assigns pathTests except they would need to be tests on the image registry, repo, or tag.

[maxsmythe]

Testing the value-to-be-modified has safety implications WRT convergence, risking turning mutation into a Turing machine:

https://github.com/orgs/open-policy-agent/discussions/15

I agree that there is a usability limitation here, though.

One thought would be to allow filtering (or other tests) on the container name, which would be immutable. I'm not sure container name is always a good proxy/sorting function for which images need re-writing, but curious if it would map on to your use case?

Kubernetes is also looking at hosted mutation (similar to the CEL KEP for hosted validation). I'm curious to see what their conclusions are before investing too deeply into mutations, as that has the potential to significantly disrupt the use cases/value proposition of webhook-based mutation.

[ritazh]

One thought would be to allow filtering (or other tests) on the container name, which would be immutable.

Would this be a new field or reuse pathTests?

[maxsmythe]

An extension of the location spec. Something like spec.containers[name:~^regexPrefix.*].image (not saying exactly that syntax)

[sozercan]

@davis-haba are we okay to close this with #2429?

[ctml91]

I'm not sure container name is always a good proxy/sorting function for which images need re-writing, but curious if it would map on to your use case?

Unfortunately it would really only work for me based on the original image registry. As new workloads get deployed I'd have to keep creating new instances of AssignImage, I would end up with about 300 of these.

Good to know about the hosted mutation, thank you. if you know of any area it's actively being discussed I'd love to follow along.

[maxsmythe]

Let me ask what the status of the discussion is and where/how to engage.

[maxsmythe]

It sounds like the most likely channel is sig-api-machinery's google group:

https://groups.google.com/g/kubernetes-sig-api-machinery

There's some early thought being put into the shape of things and to better understand the problem.