[helm] Fix reverse proxy for builtin registry

[corneliusludmann]

With the new proxy implementation with Caddy, the access to the built-in registry is over Caddy which serves the registry with the main proxy certificates instead of the registry certificates (SSL termination). That has the following implications:

• Caddy accesses the registry via https://registry.default.svc.cluster.local. That means the self-signed cert of the registry needs this domain instead of https://registry.my-gitpod.example.com.
• Caddy needs to trust the self-singed certs of the registry (needs to know the CA).
• The dind of the image builder does not need the registry certs since the certs that Caddy uses are (need to be) trusted certs and not self-signed certs (as long as bypassProxy is not set).

I tested this on k3s with this setting and verified that the preview env still works.

Alternatives: The alternative would be to change the behavior of Caddy so that the proxy does not terminate SSL but does a raw redirect to the registry as it was in nginx (when this is possible with Caddy).

See also #4710

/cc @solomonope due to self-hosted
/cc @aledbf since you implemented the Caddy config

[csweichel]

Also assigning @aledbf here due to upcoming changes on self-hosted internal cert management

[corneliusludmann]

@wulfthimm: The (self-signed) cert of the registry has always the internal domain registry.default.svc.cluster.local as domain, never the public registry.my-gitpod.example.com domain. Don't know how the cert-manager works but could this lead to problems with the cert-manager configuration here? Do we need to drop this?

[aledbf]

@corneliusludmann I have a draft PR that improves the self-signed certificate generation and do not uses proxy to access the docker-registry. Please check #4592

[corneliusludmann]

I have a draft PR that improves the self-signed certificate generation and do not uses proxy to access the docker-registry. Please check #4592

Thanks for the hint, @aledbf! Big PR, couldn't go through everything. 😇 Does it mean that your PR solves the issues I posted in our slack channel self-hosted a few days ago [here and here (internal)]? Is there an ETA for your PR?

[aledbf]

Does it mean that your PR solves the issues I posted in our slack channel self-hosted a few days ago

Yes. I need to finish the PR before next Monday.

[aledbf]

@corneliusludmann I updated the PR and split the commits to be more clear about what is being changed

[geropl]

/werft run

👍 started the job as gitpod-build-clu-fix-registry-reverse-proxy.1

[geropl]

👍

Quick reasoning summary:

pull

• each registry-facade installs a re-direct in it's local /etc/hosts: reg.${domain}:port -> registry-facade node port
• this works because we issue publicly verifiable certs (Let's Encrypt) which include reg.

push:

• prod/staging/devstaging: imagebuilder get's the properly configured GCP creds and the GCP registry has publicly verifiable certs
• builtin: our chart has an exception that makes the registry available at registry.domain
• the builtin case is broken because we never test it (neither staging nor devstaging)
• the fix works as advertised

[geropl]

/werft run with-clean-slate-deployment

👍 started the job as gitpod-build-clu-fix-registry-reverse-proxy.2