WIP: cert-manager & external-dns example

[ShankyJS]

What this PR does / why we need it:

This PR aims to add an useful example to install cert-manager using Helm Module as the cert-manager integration from Garden is going to be deprecated in 0.13.

This example also includes some really useful tools like external-dns and uses both tools to deploy a simple React application and expose it securely with HTTPs.

This is WIP

Which issue(s) this PR fixes:

Fixes #3521

Special notes for your reviewer:

[worldofgeese]

Looking 👁️

[worldofgeese]

I think it would be best if the README was a docs page in our docs directory, likely taking the place of the cert-manager integration page under the Advanced section with a mention under https://docs.garden.io/kubernetes-plugins/remote-k8s/ingress-and-dns somewhere inside this paragraph block (reproduced below):

If you would like to manage TLS for development environments, we recommend using your cloud provider's certificate management service in combination with a load balancer. You can find the documentation for AWS here and for GCP here.

All of the code could still target examples/ but I would like to see at least a functional skeleton produced in-line to the docs page so a reader can read and reproduce down the page without a lot of task-switching. One technical content author I appreciate is Michael Herman of TestDriven.io who wrote this beautiful piece of clear writing on dockerizing a Flask example app.

My approach is to build-up on the page to avoid overwhelm and to introduce discrete chunks of meaning gradually.

[worldofgeese]

Great first PR with a whole lotta content! I have not tested your code yet but there should be enough here for you to work with before I run through a test tomorrow.

Things to keep in mind:

• Markdown lint wherever possible until 0 errors are returned
  
  • I am a strong believer in using whatever editor you like: every editor should have Markdown linting plugins available.
• Always add alt-text for the visually impaired or blind.
• Prefer in-line code samples over images to increase readability except in such cases where it is unreasonable or detrimental.
• Prefer Markdown equivalent over HTML except in cases where Markdown does not support what you wish to convey.

I've also gone through and clarified language, fixed spelling, and brought brevity and an active voice where it was needed. Happy to do these as code suggestions over direct commits in the future, if that's your preference.

I'm considering resurrecting a technical content writing style guide for Garden.

[worldofgeese]

Using the latest edge-bonsai garden, I received errors deploying the internal Docker registry. As it's being deprecated I recommend me move away from examples that use it.

To mitigate, add the following keys and their values:

deploymentRegistry:
      hostname: "${var.registryHostname}"
      namespace: "${var.registryNamespace}"
    imagePullSecrets:
      - name: regcred
        namespace: default   # <--- the name and namespace of a valid Kubernetes imagePullSecret

variables:
  # change these values to your own
  registryHostname: docker.io
  registryNamespace: worldofgeese

Ensure you've created the image pull secret in your cluster first:

kubectl create secret generic regcred \
    --from-file=.dockerconfigjson=$HOME/.docker/config.json \
    --type=kubernetes.io/dockerconfigjson

ExternalDNS was also unable to create entries for my domain, despite first exporting my CF_API_KEY:

> kubectl logs po/external-dns-589f9df5b5-lk6m8 -n external-dns
time="2023-03-08T15:21:12Z" level=info msg="config: {APIServerURL: KubeConfig: RequestTimeout:30s DefaultTargets:[] ContourLoadBalancerService:heptio-contour/contour GlooNamespace:gloo-system SkipperRouteGroupVersion:zalando.org/v1 Sources:[service ingress] Namespace: AnnotationFilter: LabelFilter: FQDNTemplate: CombineFQDNAndAnnotation:false IgnoreHostnameAnnotation:false IgnoreIngressTLSSpec:false IgnoreIngressRulesSpec:false GatewayNamespace: GatewayLabelFilter: Compatibility: PublishInternal:false PublishHostIP:false AlwaysPublishNotReadyAddresses:false ConnectorSourceServer:localhost:8080 Provider:cloudflare GoogleProject: GoogleBatchChangeSize:1000 GoogleBatchChangeInterval:1s GoogleZoneVisibility: DomainFilter:[mead.video] ExcludeDomains:[] RegexDomainFilter: RegexDomainExclusion: ZoneNameFilter:[] ZoneIDFilter:[] TargetNetFilter:[] ExcludeTargetNets:[] AlibabaCloudConfigFile:/etc/kubernetes/alibaba-cloud.json AlibabaCloudZoneType: AWSZoneType: AWSZoneTagFilter:[] AWSAssumeRole: AWSAssumeRoleExternalID: AWSBatchChangeSize:1000 AWSBatchChangeInterval:1s AWSEvaluateTargetHealth:true AWSAPIRetries:3 AWSPreferCNAME:false AWSZoneCacheDuration:0s AWSSDServiceCleanup:false AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: AzureSubscriptionID: AzureUserAssignedIdentityClientID: BluecatDNSConfiguration: BluecatConfigFile:/etc/kubernetes/bluecat.json BluecatDNSView: BluecatGatewayHost: BluecatRootZone: BluecatDNSServerName: BluecatDNSDeployType:no-deploy BluecatSkipTLSVerify:false CloudflareProxied:false CloudflareZonesPerPage:50 CoreDNSPrefix:/skydns/ RcodezeroTXTEncrypt:false AkamaiServiceConsumerDomain: AkamaiClientToken: AkamaiClientSecret: AkamaiAccessToken: AkamaiEdgercPath: AkamaiEdgercSection: InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin InfobloxWapiPassword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true InfobloxView: InfobloxMaxResults:0 InfobloxFQDNRegEx: InfobloxCreatePTR:false InfobloxCacheDuration:0 DynCustomerName: DynUsername: DynPassword: DynMinTTLSeconds:0 OCIConfigFile:/etc/kubernetes/oci.yaml InMemoryZones:[] OVHEndpoint:ovh-eu OVHApiRateLimit:20 PDNSServer:http://localhost:8081 PDNSAPIKey: PDNSTLSEnabled:false TLSCA: TLSClientCert: TLSClientCertKey: Policy:sync Registry:txt TXTOwnerID:default TXTPrefix: TXTSuffix: Interval:1m0s MinEventSyncInterval:5s Once:false DryRun:false UpdateEvents:false LogFormat:text MetricsAddress::7979 LogLevel:info TXTCacheInterval:0s TXTWildcardReplacement: ExoscaleEndpoint:https://api.exoscale.ch/dns ExoscaleAPIKey: ExoscaleAPISecret: CRDSourceAPIVersion:externaldns.k8s.io/v1alpha1 CRDSourceKind:DNSEndpoint ServiceTypeFilter:[] CFAPIEndpoint: CFUsername: CFPassword: RFC2136Host: RFC2136Port:0 RFC2136Zone: RFC2136Insecure:false RFC2136GSSTSIG:false RFC2136KerberosRealm: RFC2136KerberosUsername: RFC2136KerberosPassword: RFC2136TSIGKeyName: RFC2136TSIGSecret: RFC2136TSIGSecretAlg: RFC2136TAXFR:false RFC2136MinTTL:0s RFC2136BatchChangeSize:50 NS1Endpoint: NS1IgnoreSSL:false NS1MinTTLSeconds:0 TransIPAccountName: TransIPPrivateKeyFile: DigitalOceanAPIPageSize:50 ManagedDNSRecordTypes:[A CNAME] GoDaddyAPIKey: GoDaddySecretKey: GoDaddyTTL:0 GoDaddyOTE:false OCPRouterName: IBMCloudProxied:false IBMCloudConfigFile:/etc/kubernetes/ibmcloud.json TencentCloudConfigFile:/etc/kubernetes/tencent-cloud.json TencentCloudZoneType: PiholeServer: PiholePassword: PiholeTLSInsecureSkipVerify:false PluralCluster: PluralProvider:}"
time="2023-03-08T15:21:12Z" level=info msg="Instantiating new Kubernetes client"
time="2023-03-08T15:21:12Z" level=info msg="Using inCluster-config based on serviceaccount-token"
time="2023-03-08T15:21:12Z" level=info msg="Created Kubernetes client https://10.100.0.1:443"
time="2023-03-08T15:21:13Z" level=error msg="Invalid request headers (6003)"
time="2023-03-08T15:22:13Z" level=error msg="Invalid request headers (6003)"
time="2023-03-08T15:23:13Z" level=error msg="Invalid request headers (6003)"
time="2023-03-08T15:24:13Z" level=error msg="Invalid request headers (6003)"
time="2023-03-08T15:25:14Z" level=error msg="Invalid request headers (6003)"
time="2023-03-08T15:26:15Z" level=error msg="Invalid request headers (6003)"
time="2023-03-08T15:27:15Z" level=error msg="Invalid request headers (6003)"
time="2023-03-08T15:28:16Z" level=error msg="Invalid request headers (6003)"
time="2023-03-08T15:29:17Z" level=error msg="Invalid request headers (6003)"
time="2023-03-08T15:30:18Z" level=error msg="Invalid request headers (6003)"
time="2023-03-08T15:31:18Z" level=error msg="Invalid request headers (6003)"
time="2023-03-08T15:32:19Z" level=error msg="Invalid request headers (6003)"
time="2023-03-08T15:33:19Z" level=error msg="Invalid request headers (6003)"
time="2023-03-08T15:34:20Z" level=error msg="Invalid request headers (6003)"

Searching this error led me to cert-manager/cert-manager#3021

[worldofgeese]

All changed variables and their values:

providers:
  - name: kubernetes
    context: tao@garden.io@ridiculous-creature-1678286886.eu-central-1.eksctl.io
    deploymentRegistry:
      hostname: "${var.registryHostname}"
      namespace: "${var.registryNamespace}"
    imagePullSecrets:
      - name: regcred
        namespace: default   # <--- the name and namespace of a valid Kubernetes imagePullSecret

variables:
  GENERATE_PROD_CERTS: false # Set this to true if you want to generate production certificates
  CF_DOMAIN: [ mead.video ] # Has to be an array, even if you only have one domain
  CF_EMAIL: cloudflare@tao.mozmail.com # This is the email you use to login to Cloudflare
  registryHostname: docker.io
  registryNamespace: worldofgeese

I also ran export CF_API_TOKEN=$secrettoken before deploying.

[stefreak]

I think the ingress and DNS guide should still introduce you to multiple ways of of how to set up DNS and the ingresses. Also, IMO it should still recommend users to use the cloud provider's solutions for certificate management and load balancing. For most teams, the additional complexity of maintaining cert-manager and external-dns is not worth it.

I'm very hesitant of making the whole remote Kubernetes "Set Up Ingress, TLS and DNS" guide that opinionated towards using cert-manager.

Maybe the cert-manager + ext-dns guide even move into a separate tutorial section, and mentioned in the generic "ingress and DNS" guide, next to the other available options.

That being said, I really see that you have put a lot of effort into the example and the guide. They're really detailed and will be a great resource for teams who need this kind of setup! 🏡 🌷

[worldofgeese]

Suggested change

	To showcase our certificates we decided to use a sample react-application, the code for that React Application is hosted in our examples folder in the main `Garden` repository [tls-and-dns](https://github.com/garden-io/garden/tree/main/examples/tls-and-dns/frontend).
	To showcase our certificates, we use a sample React application hosted in our examples folder in the main `Garden` repository [tls-and-dns](https://github.com/garden-io/garden/tree/main/examples/tls-and-dns/frontend).

[worldofgeese]

Suggested change

	Our website is now up, but without a certificate; because ExternalDNS already provisioned our DNS records, but we still haven't generated any Certificate for our frontend application.
	Our website is now up but without a certificate because we still haven't generated our Certificate resource.

[worldofgeese]

Suggested change

	Now that we have ExternalDNS and Cert-manager charts configured, we still have to configure our Cert-manager Custom Resources.
	Now that we have ExternalDNS and cert-manager deployed, we can move on to configuring cert-manager.

[ShankyJS]

Closing in favor of #3988