[single-cluster/eks] Support for `self-signed` certificates

[Pothulapati]

Bug description

The new single cluster eks reference guide does not seem to support self-signed certificates.
The image build fails with the following error:

cannot pull image: rpc error: code = Unknown desc = failed to pull and unpack image "reg.be9c7-aws.tests.gitpod-self-hosted.com:20000/remote/33cbe75c-6aba-4bca-b64e-bbaaee0cf2be:latest": failed to resolve reference "reg.be9c7-aws.tests.gitpod-self-hosted.com:20000/remote/33cbe75c-6aba-4bca-b64e-bbaaee0cf2be:latest": failed to do request: Head https://reg.be9c7-aws.tests.gitpod-self-hosted.com:20000/v2/remote/33cbe75c-6aba-4bca-b64e-bbaaee0cf2be/manifests/latest: x509: certificate signed by unknown authority

This seems to because of the containerd on the node is not configured to trust the self-signed gitpod registry. This causes the image pulls to fail.

Support for self-signed worked in the gitpod-eks-guide because of us using custom built AMI images, that seems to configure containerd in a specific way to reload certs as per

https://github.com/gitpod-io/gitpod-eks-guide/blob/d2b5fca8db3e20997f08cdeefc4bea4ff938dfe6/ami/files/gitpod/containerd.toml#L26-L28

Steps to reproduce

This was found through the new self-signed test being written #12910

Run

werft run github -f -s .werft/installer-tests.ts -j .werft/eks-installer-tests.yaml -a debug=true -a self-signed=true -a skipTests=true -a preview=true

from the PR workspace, and see that the build fails. Manually creating the EKS environment, and passing the certs should also give the same.

[mrzarquon]

Found this comment again - i think the fix for EKS would be we update our infra terraform scripts to do post_bootstrap_user_data and just add

cat << CONFIG >> /etc/containerd/config.toml

[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d"

CONFIG

service containerd restart

That we do in our ALB guide. It doesn't hurt for none self signed instances, but just doing it by default will ensure it's included if other custom ssl elements are added later on.

[adrienthebo]

Added this to the single cluster reference architecture GA release to ensure it's tracked, but we may shift this ticket to the beta.

[Pothulapati]

Started working on it! My current plan is to make it the default (like @mrzarquon suggested) unless something comes up.