Proxy any registry and do it automatically (using MutatingWebhookConfiguration)

[awoimbee]

Sorry for the size of this PR, I saw no other way to do this (and do it fast).

The goals of this PR:

• Fix the helm chart
• One click install of Trow with its admission webhooks through helm
• Add a mutation admission webhook so images in pods are automatically routed through Trow => Fixes mention imageswap-webhook in the documentation #322
• Ability to proxy any registry (Implement kubernetes.io/dockerconfigjson as configuration for proxy any number of registries #250) -> handle multiple types of authentication methods (Proxying support from extrality fork #279)

Other changes that happened:

• replace the validation configuration: the old way of configuring validation was way too complicated for me. There were 6 CLI args + the logic regarding if the image is "local" or not. => replaced by a .yaml that only configures the webhook (no side effects)
• move some logic from trow-server to modules: proxy_auth.rs, image.rs => it made writing tests so much easier
• use kube and k8s-openapi instead of custom types for AdmissionReview

Deprecation of ./quick-install/ and ./install/:
I want to greatly simplify the deployment by only providing an easy to use helm chart. These folders push the user to run many manual steps/scripts and they duplicate deployment config. I did not outright delete these folders because this PR has already too many changes, but it's planned.

Notes:

• I am running this Trow version in prod. Deployed via Helm, chart hosted at: https://github.com/awoimbee/helm-charts/releases/tag/trow-0.4.1
• The helm_chart_validation test fails because it uses the old docker container, which takes different CLI args from the new one. This PR is already big enough as is so I'm not gonna fix that.
• the webhook patch deployment thing is largely inspired by ingress-nginx

[mysticaltech]

Hello folks, this is super useful! Probably what we need for #339. @awoimbee Why isn't it merged yet?

[mysticaltech]

@amouat Please consider merging this PR, it would help us a lot.

[amouat]

@mysticaltech there's a few conflicts that need resolved, but they don't look difficult.

@awoimbee what's the outcome of this patch - does helm become the standard way to install Trow? I assume kustomize still works?

[awoimbee]

@amouat
Helm becomes the standard way, I have not touched kustomize, so it will likely fail due to the new validation webhook and the new CLI args.

The outcomes are listed in the PR description, but if I had to list them now (1 year later), it would be:

• EZ install via helm, the required TLS certs are auto generated.
• MutatingWebhook: I can deploy stuff to my cluster without making sure every image is pulled through Trow, Trow does it automagically with its MutatingWebhook.
• EZ validating webhook config: one can easily set a blacklist, a whitelist and the default behavior.

My setup

I've been using this in prod since the 22nd of June 2022.

The equivalent values.yaml to what I use would be:

image:
  tag: 22-06-2022-0
trow:
  domain: 127.0.0.1:12345
  validation:
    enableWebhook: true
    config:
      default: Deny
      allow:
      - 127.0.0.1:12345/
      - 602401143452.dkr.ecr
      - public.ecr.aws/
  proxyConfig:
    enableWebhook: true
    config:
    - { alias: docker, host: registry-1.docker.io }
    - { alias: quay, host: quay.io }
    - { alias: nvcr, host: nvcr.io }
    - { alias: ghcr, host: ghcr.io }
    - { alias: k8s, host: registry.k8s.io }
    - { alias: k8s-gcr, host: gcr.k8s.io }
    - { alias: gcr-k8s, host: k8s.gcr.io }
    - { alias: elastic, host: docker.elastic.co }
#   - [...]
    - { alias: ecr, host: xxxxxxxxxxxx.dkr.ecr.us-west-1.amazonaws.com, username: AWS }
service:
  type: NodePort
  nodePort: 12345
resources:
  requests:
    cpu: 100m
    memory: 128Mi
volumeClaim:
  resource:
    requests:
      storage: 30Gi

The final describe pods looks like (I have shortened and redacted it):

Name:                 trow-0
Priority Class Name:  system-cluster-critical
Status:               Running
Controlled By:  StatefulSet/trow
Init Containers:
  trow-webhook-cert-translate:
    Command:
      sh
      -c
      cp /etc/trow/webhook-cert-ecc/cert /etc/trow/webhook-cert && \
      openssl pkcs8 -topk8 -nocrypt -in /etc/trow/webhook-cert-ecc/key -out /etc/trow/webhook-cert/key
    Mounts:
      /etc/trow/webhook-cert from webhook-cert-translated (rw)
      /etc/trow/webhook-cert-ecc from webhook-cert-ecc (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xxxxx (ro)
Containers:
  trow:
    Image:         awoimbee/trow:22-06-2022-0
    Args:
      --no-tls
      -n
      127.0.0.1:12345
      --proxy-registry-config-file=/etc/trow/proxy-registries.yaml
      --image-validation-config-file=/etc/trow/validation.yaml
    Mounts:
      /data from data-vol (rw)
      /etc/trow/proxy-registries.yaml from trow-proxy-cfg (ro,path="proxy-registries.yaml")
      /etc/trow/validation.yaml from trow-validation-cfg (ro,path="validation.yaml")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xxxxx (ro)
  trow-webhook:
    Image:         awoimbee/trow:22-06-2022-0
    Args:
      --cert=/etc/trow/webhook-cert/cert
      --key=/etc/trow/webhook-cert/key
      -n
      127.0.0.1:12345
      --proxy-registry-config-file=/etc/trow/proxy-registries.yaml
      --image-validation-config-file=/etc/trow/validation.yaml
    Mounts:
      /data from data-vol (rw)
      /etc/trow/proxy-registries.yaml from trow-proxy-cfg (ro,path="proxy-registries.yaml")
      /etc/trow/validation.yaml from trow-validation-cfg (ro,path="validation.yaml")
      /etc/trow/webhook-cert from webhook-cert-translated (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xxxxx (ro)
Volumes:
  data-vol:
  webhook-cert-ecc:
  webhook-cert-translated:
  trow-proxy-cfg:
  trow-validation-cfg:
  kube-api-access-xxxxx:

[amouat]

Basically I like this a lot, but I'd also like kustomize to keep working. If you're still using this and are able to maintain (the helm side of things) we should merge.

Unfortunately I don't have time to do much work on this any more since changing jobs.

[mysticaltech]

Folks, our Kube-Hetzner project is growing fast, and many Hetzner Cloud nodes are blacklisted by gcr and everything that uses it underneath like registry.k8s.io, and ghcr.io. But in a single cluster, most nodes are ok, so this is an IDEAL solution.

So, we will recommend this and even ship it as an add-on, doing our fair share to provide users with the best possible experience.

I think it has amazing potential! If you folks combine forces it would be of great help to a lot of people.

[amouat]

@mysticaltech please drop me an e-mail to adrian.mouat at gmail and we can talk about options.

[awoimbee]

Basically I like this a lot, but I'd also like kustomize to keep working. If you're still using this and are able to maintain (the helm side of things) we should merge.

I can maintain the helm chart, but not the quick-install and kustomize.
Keeping only the helm deployment would free this repo from more than 1700 lines of yaml, docs and scripts.

The helm chart from this PR is available for testing @ https://awoimbee.github.io/helm-charts, I'm using v0.4.1.

[mysticaltech]

@amouat Thanks, but please know that we are fully open-source and doing voluntary community work. It was a tool I originally created for myself, but it picked up some pace.

Personally, we will only need to helm option, so really appreciate you sharing this @awoimbee! 🙏

[awoimbee]

If you're still using this and are able to maintain (the helm side of things) we should merge.

We should merge @amouat.

[mysticaltech]

Wonderful!