From 7342285583046c9b848a41b1ae85d5414e6cbd43 Mon Sep 17 00:00:00 2001 From: joaquin Date: Sat, 14 May 2022 08:49:01 -0700 Subject: [PATCH 1/3] gke refresh --- docs/tutorials/gke.md | 703 ++++++++++++++++++------------------------ 1 file changed, 303 insertions(+), 400 deletions(-) diff --git a/docs/tutorials/gke.md b/docs/tutorials/gke.md index 28419ad05e..80e26fbef1 100644 --- a/docs/tutorials/gke.md +++ b/docs/tutorials/gke.md @@ -1,394 +1,275 @@ # Setting up ExternalDNS on Google Kubernetes Engine -This tutorial describes how to setup ExternalDNS for usage within a GKE cluster. Make sure to use **>=0.4** version of ExternalDNS for this tutorial +This tutorial describes how to setup ExternalDNS for usage within a [GKE](https://cloud.google.com/kubernetes-engine) ([Google Kuberentes Engine](https://cloud.google.com/kubernetes-engine)) cluster. Make sure to use **>=0.11.0** version of ExternalDNS for this tutorial -## Set up your environment +## Single project test scenario using access scopes -Setup your environment to work with Google Cloud Platform. Fill in your values as needed, e.g. target project. +*If you prefer to try-out ExternalDNS in one of the existing environments you can skip this step* -```console -$ gcloud config set project "zalando-external-dns-test" -$ gcloud config set compute/region "europe-west1" -$ gcloud config set compute/zone "europe-west1-d" -``` +The following instructions use [access scopes](https://cloud.google.com/compute/docs/access/service-accounts#accesscopesiam) to provide ExternalDNS with the permissions it needs to manage DNS records within a single [project](https://cloud.google.com/docs/overview#projects), the organizing entity to allocate resources. -## GKE Node Scopes +Note that since these permissions are associated with the instance, all pods in the cluster will also have these permissions. As such, this approach is not suitable for anything but testing environments. -*If you prefer to try-out ExternalDNS in one of the existing environments you can skip this step* +This solution will only work when both CloudDNS and GKE are provisioned in the same project. If the CloudDNS zone is in a different project, this solution will not work. -The following instructions use instance scopes to provide ExternalDNS with the -permissions it needs to manage DNS records. Note that since these permissions -are associated with the instance, all pods in the cluster will also have these -permissions. As such, this approach is not suitable for anything but testing -environments. +### Configure Project Environment -Create a GKE cluster. +Setup your environment to work with Google Cloud Platform. Fill in your variables as needed, e.g. target project. -```console -$ gcloud container clusters create "external-dns" \ - --num-nodes 1 \ - --scopes "https://www.googleapis.com/auth/ndev.clouddns.readwrite" +```bash +# set variables to the appropriate desired values +PROJECT_ID="my-external-dns-test" +REGION="europe-west1" +ZONE="europe-west1-d" +ClOUD_BILLING_ACCOUNT="" +# set default settings for project +gcloud config set project $PROJECT_ID +gcloud config set compute/region $REGION +gcloud config set compute/zone $ZONE +# enable billing and APIs if not done already +gcloud beta billing projects link $PROJECT_ID \ + --billing-account $BILLING_ACCOUNT +gcloud services enable "dns.googleapis.com" +gcloud services enable "container.googleapis.com" +``` + +### Create GKE Cluster + +```bash +gcloud container clusters create $GKE_CLUSTER_NAME \ + --num-nodes 1 \ + --scopes "https://www.googleapis.com/auth/ndev.clouddns.readwrite" ``` -Create a DNS zone which will contain the managed DNS records. +**WARNING**: Note that this cluster will use the default [compute engine GSA](https://cloud.google.com/compute/docs/access/service-accounts#default_service_account) that contians the overly permissive project editor (`roles/editor`) role. So essentially, anything on the cluster could potentially grant escalated privileges. Also, as mentioned earlier, the access scope `ndev.clouddns.readwrite` will allow anything running on the cluster to have read/write permissions on all Cloud DNS zones within the same project. + +### Cloud DNS Zone -```console -$ gcloud dns managed-zones create "external-dns-test-gcp-zalan-do" \ - --dns-name "external-dns-test.gcp.zalan.do." \ - --description "Automatically managed zone by kubernetes.io/external-dns" +Create a DNS zone which will contain the managed DNS records. If using your own domain that was registered with a third-party domain registrar, you should point your domain's name servers to the values under the `nameServers` key. Please consult your registrar's documentation on how to do that. This tutorial will use example domain of `example.com`. + +```bash +gcloud dns managed-zones create "example-com" --dns-name "example.com." \ + --description "Automatically managed zone by kubernetes.io/external-dns" ``` Make a note of the nameservers that were assigned to your new zone. -```console -$ gcloud dns record-sets list \ - --zone "external-dns-test-gcp-zalan-do" \ - --name "external-dns-test.gcp.zalan.do." \ - --type NS -NAME TYPE TTL DATA -external-dns-test.gcp.zalan.do. NS 21600 ns-cloud-e1.googledomains.com.,ns-cloud-e2.googledomains.com.,ns-cloud-e3.googledomains.com.,ns-cloud-e4.googledomains.com. +```bash +gcloud dns record-sets list \ + --zone "example-com" --name "example.com." --type NS ``` -In this case it's `ns-cloud-{e1-e4}.googledomains.com.` but your's could slightly differ, e.g. `{a1-a4}`, `{b1-b4}` etc. - -Tell the parent zone where to find the DNS records for this zone by adding the corresponding NS records there. Assuming the parent zone is "gcp-zalan-do" and the domain is "gcp.zalan.do" and that it's also hosted at Google we would do the following. +Outputs: -```console -$ gcloud dns record-sets transaction start --zone "gcp-zalan-do" -$ gcloud dns record-sets transaction add ns-cloud-e{1..4}.googledomains.com. \ - --name "external-dns-test.gcp.zalan.do." --ttl 300 --type NS --zone "gcp-zalan-do" -$ gcloud dns record-sets transaction execute --zone "gcp-zalan-do" +``` +NAME TYPE TTL DATA +example.com. NS 21600 ns-cloud-e1.googledomains.com.,ns-cloud-e2.googledomains.com.,ns-cloud-e3.googledomains.com.,ns-cloud-e4.googledomains.com. ``` -### Deploy ExternalDNS - -Then apply the following manifests file to deploy ExternalDNS. +In this case it's `ns-cloud-{e1-e4}.googledomains.com.` but your's could slightly differ, e.g. `{a1-a4}`, `{b1-b4}` etc. -```yaml -apiVersion: v1 -kind: ServiceAccount -metadata: - name: external-dns ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRole -metadata: - name: external-dns -rules: -- apiGroups: [""] - resources: ["services","endpoints","pods"] - verbs: ["get","watch","list"] -- apiGroups: ["extensions","networking.k8s.io"] - resources: ["ingresses"] - verbs: ["get","watch","list"] -- apiGroups: [""] - resources: ["nodes"] - verbs: ["get", "watch", "list"] ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRoleBinding -metadata: - name: external-dns-viewer -roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: external-dns -subjects: -- kind: ServiceAccount - name: external-dns - namespace: default ---- -apiVersion: apps/v1 -kind: Deployment -metadata: - name: external-dns -spec: - strategy: - type: Recreate - selector: - matchLabels: - app: external-dns - template: - metadata: - labels: - app: external-dns - spec: - serviceAccountName: external-dns - containers: - - name: external-dns - image: k8s.gcr.io/external-dns/external-dns:v0.8.0 - args: - - --source=service - - --source=ingress - - --domain-filter=external-dns-test.gcp.zalan.do # will make ExternalDNS see only the hosted zones matching provided domain, omit to process all available hosted zones - - --provider=google -# - --google-project=zalando-external-dns-test # Use this to specify a project different from the one external-dns is running inside - - --google-zone-visibility=private # Use this to filter to only zones with this visibility. Set to either 'public' or 'private'. Omitting will match public and private zones - - --policy=upsert-only # would prevent ExternalDNS from deleting any records, omit to enable full synchronization - - --registry=txt - - --txt-owner-id=my-identifier -``` +## Cross project access scenario using Google Service Account -Use `--dry-run` if you want to be extra careful on the first run. Note, that you will not see any records created when you are running in dry-run mode. You can, however, inspect the logs and watch what would have been done. +More often, following best practices in regards to security and operations, Cloud DNS zones will be managed in a separate project from the Kubernetes cluster. This section shows how setup ExternalDNS to access Cloud DNS from a different project. These steps will also work for single project scenarios as well. -### Verify ExternalDNS works +ExternalDNS will need permissions to make changes to the Cloud DNS zone. There are three ways to configure the access needed: -Create the following sample application to test that ExternalDNS works. +* [Worker Node Service Account](#worker-node-service-account) +* [Static Credentials](#static-credentials) +* [Work Load Identity](#work-load-identity) -```yaml -apiVersion: v1 -kind: Service -metadata: - name: nginx - annotations: - external-dns.alpha.kubernetes.io/hostname: nginx.external-dns-test.gcp.zalan.do. -spec: - type: LoadBalancer - ports: - - port: 80 - targetPort: 80 - selector: - app: nginx +### Setup Cloud DNS and GKE ---- +Below are examples on how you can configure Cloud DNS and GKE in separate projects, and then use one of the three methods to grant access to ExternalDNS. Replace the environment variables to values that make sense in your environment. -apiVersion: apps/v1 -kind: Deployment -metadata: - name: nginx -spec: - selector: - matchLabels: - app: nginx - template: - metadata: - labels: - app: nginx - spec: - containers: - - image: nginx - name: nginx - ports: - - containerPort: 80 -``` +#### Configure Projects -After roughly two minutes check that a corresponding DNS record for your service was created. +For this process, create projects with the appropriate APIs enabled. -```console -$ gcloud dns record-sets list \ - --zone "external-dns-test-gcp-zalan-do" \ - --name "nginx.external-dns-test.gcp.zalan.do." +```bash +# set variables to appropriate desired values +GKE_PROJECT_ID="my-workload-project" +DNS_PROJECT_ID="my-cloud-dns-project" +ClOUD_BILLING_ACCOUNT="" +# enable billing and APIs for DNS project if not done already +gcloud config set project $DNS_PROJECT_ID +gcloud beta billing projects link $CLOUD_DNS_PROJECT \ + --billing-account $ClOUD_BILLING_ACCOUNT +gcloud services enable "dns.googleapis.com" +# enable billing and APIs for GKE project if not done already +gcloud config set project $GKE_PROJECT_ID +gcloud beta billing projects link $CLOUD_DNS_PROJECT \ + --billing-account $ClOUD_BILLING_ACCOUNT +gcloud services enable "container.googleapis.com" +``` + +#### Provisioning Cloud DNS + +Create a Cloud DNS zone in the designated DNS project. -NAME TYPE TTL DATA -nginx.external-dns-test.gcp.zalan.do. A 300 104.155.60.49 -nginx.external-dns-test.gcp.zalan.do. TXT 300 "heritage=external-dns,external-dns/owner=my-identifier" +```bash +gcloud dns managed-zones create "example-com" --project $DNS_PROJECT_ID \ + --description "example.com" --dns-name="example.com." --visibility=public ``` -Note created TXT record alongside A record. TXT record signifies that the corresponding A record is managed by ExternalDNS. This makes ExternalDNS safe for running in environments where there are other records managed via other means. +If using your own domain that was registered with a third-party domain registrar, you should point your domain's name servers to the values under the `nameServers` key. Please consult your registrar's documentation on how to do that. The example domain of `example.com` will be used for this tutorial. -Let's check that we can resolve this DNS name. We'll ask the nameservers assigned to your zone first. +#### Provisioning a GKE cluster for cross project access -```console -$ dig +short @ns-cloud-e1.googledomains.com. nginx.external-dns-test.gcp.zalan.do. -104.155.60.49 -``` +Create a GSA (Google Service Account) and grant it the [minimal set of privileges required](https://cloud.google.com/kubernetes-engine/docs/how-to/hardening-your-cluster#use_least_privilege_sa) for GKE nodes: -Given you hooked up your DNS zone with its parent zone you can use `curl` to access your site. +```bash +GKE_CLUSTER_NAME="my-external-dns-cluster" +GKE_REGION="us-central1" +GKE_SA_NAME="worker-nodes-sa" +GKE_SA_EMAIL="$GKE_SA_NAME@${GKE_PROJECT_ID}.iam.gserviceaccount.com" + +ROLES=( + roles/logging.logWriter + roles/monitoring.metricWriter + roles/monitoring.viewer + roles/stackdriver.resourceMetadata.writer +) -```console -$ curl nginx.external-dns-test.gcp.zalan.do - - - -Welcome to nginx! -... - - -... - - +gcloud iam service-accounts create $GKE_SA_NAME \ + --display-name $GKE_SA_NAME --project $GKE_PROJECT_ID + +# assign google service account to roles in GKE project +for ROLE in ${ROLES[*]}; do + gcloud projects add-iam-policy-binding $GKE_PROJECT_ID \ + --member "serviceAccount:$GKE_SA_EMAIL" \ + --role $ROLE +done ``` -Let's check that Ingress works as well. Create the following Ingress. +Create a cluster using this service account and enable [workload identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity): -```yaml -apiVersion: networking.k8s.io/v1 -kind: Ingress -metadata: - name: nginx -spec: - rules: - - host: via-ingress.external-dns-test.gcp.zalan.do - http: - paths: - - backend: - service: - name: nginx - port: - number: 80 - pathType: Prefix +```bash +gcloud container clusters create $GKE_CLUSTER_NAME \ + --project $GKE_PROJECT_ID --region $GKE_REGION --num-nodes 1 \ + --service-account "$GKE_SA_EMAIL" \ + --workload-pool "$GKE_PROJECT_ID.svc.id.goog" ``` -Again, after roughly two minutes check that a corresponding DNS record for your Ingress was created. +### Worker Node Service Account method -```console -$ gcloud dns record-sets list \ - --zone "external-dns-test-gcp-zalan-do" \ - --name "via-ingress.external-dns-test.gcp.zalan.do." \ +In this method, the GSA (Google Service Account) that is associated with GKE worker nodes will be configured to have access to Cloud DNS. -NAME TYPE TTL DATA -via-ingress.external-dns-test.gcp.zalan.do. A 300 130.211.46.224 -via-ingress.external-dns-test.gcp.zalan.do. TXT 300 "heritage=external-dns,external-dns/owner=my-identifier" -``` +**WARNING**: This will grant access to modify the Cloud DNS zone records for all containers running on cluster, not just ExternalDNS, so use this option with caution. This is not recommended for production environments. -Let's check that we can resolve this DNS name as well. +```bash +GKE_SA_EMAIL="$GKE_SA_NAME@${GKE_PROJECT_ID}.iam.gserviceaccount.com" -```console -dig +short @ns-cloud-e1.googledomains.com. via-ingress.external-dns-test.gcp.zalan.do. -130.211.46.224 +# assign google service account to dns.admin role in the cloud dns project +gcloud projects add-iam-policy-binding $DNS_PROJECT_ID \ + --member serviceAccount:$GKE_SA_EMAIL \ + --role roles/dns.admin ``` -Try with `curl` as well. +After this, follow the steps in [Deploy ExternalDNS](#deploy-externaldns). Make sure to set the `--google-project` flag to match the Cloud DNS project name. -```console -$ curl via-ingress.external-dns-test.gcp.zalan.do - - - -Welcome to nginx! -... - - -... - - -``` +### Static Credentials -### Clean up +In this scenario, a new GSA (Google Service Account) is created that has access to the CloudDNS zone. The credentials for this GSA are saved and installed as a Kubernetes secret that will be used by ExternalDNS. -Make sure to delete all Service and Ingress objects before terminating the cluster so all load balancers get cleaned up correctly. +This allows only containers that have access to the secret, such as ExternalDNS to update records on the Cloud DNS Zone. -```console -$ kubectl delete service nginx -$ kubectl delete ingress nginx -``` +#### Create GSA for use with static credentials -Give ExternalDNS some time to clean up the DNS records for you. Then delete the managed zone and cluster. +```bash +DNS_SA_NAME="external-dns-sa" +DNS_SA_EMAIL="$DNS_SA_NAME@${GKE_PROJECT_ID}.iam.gserviceaccount.com" -```console -$ gcloud dns managed-zones delete "external-dns-test-gcp-zalan-do" -$ gcloud container clusters delete "external-dns" +# create GSA used to access the Cloud DNS zone +gcloud iam service-accounts create $DNS_SA_NAME --display-name $DNS_SA_NAME + +# assign google service account to dns.admin role in cloud-dns project +gcloud projects add-iam-policy-binding $DNS_PROJECT_ID \ + --member serviceAccount:$DNS_SA_EMAIL --role "roles/dns.admin" ``` -Also delete the NS records for your removed zone from the parent zone. +#### Create Kubernetes secret using static credentials -```console -$ gcloud dns record-sets transaction start --zone "gcp-zalan-do" -$ gcloud dns record-sets transaction remove ns-cloud-e{1..4}.googledomains.com. \ - --name "external-dns-test.gcp.zalan.do." --ttl 300 --type NS --zone "gcp-zalan-do" -$ gcloud dns record-sets transaction execute --zone "gcp-zalan-do" -``` +Generate static credentials from the ExternalDNS GSA. -## GKE with Workload Identity +```bash +# download static credentials +gcloud iam service-accounts keys create /local/path/to/credentials.json \ + --iam-account $DNS_SA_EMAIL +``` -The following instructions use [GKE workload -identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) -to provide ExternalDNS with the permissions it needs to manage DNS records. -Workload identity is the Google-recommended way to provide GKE workloads access -to GCP APIs. +Create a Kubernetes secret with the credentials in the same namespace of ExternalDNS. -Create a GKE cluster with workload identity enabled. +```bash +EXTENALDNS_NS="default" -```console -$ gcloud container clusters create external-dns \ - --workload-metadata-from-node=GKE_METADATA_SERVER \ - --identity-namespace=zalando-external-dns-test.svc.id.goog +kubectl create secret generic "external-dns" --namespace $EXTENALDNS_NS \ + --from-file /local/path/to/credentials.json ``` -Create a GCP service account (GSA) for ExternalDNS and save its email address. +After this, follow the steps in [Deploy ExternalDNS](#deploy-externaldns). Make sure to set the `--google-project` flag to match Cloud DNS project name. -```console -$ sa_name="Kubernetes external-dns" -$ gcloud iam service-accounts create sa-edns --display-name="$sa_name" -$ sa_email=$(gcloud iam service-accounts list --format='value(email)' \ - --filter="displayName:$sa_name") -``` +### Workload Identity -Bind the ExternalDNS GSA to the DNS admin role. +[Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) allows workloads in your GKE cluster to impersonate GSA (Google Service Accounts) using KSA (Kubernetes Service Accounts) configured during deployemnt. These are the steps to use this feature with ExternalDNS. -```console -$ gcloud projects add-iam-policy-binding zalando-external-dns-test \ - --member="serviceAccount:$sa_email" --role=roles/dns.admin -``` +#### Create GSA for use with Workload Identity -Link the ExternalDNS GSA to the Kubernetes service account (KSA) that -external-dns will run under, i.e., the external-dns KSA in the external-dns -namespaces. +```bash +DNS_SA_NAME="external-dns-sa" +DNS_SA_EMAIL="$DNS_SA_NAME@${GKE_PROJECT_ID}.iam.gserviceaccount.com" -```console -$ gcloud iam service-accounts add-iam-policy-binding "$sa_email" \ - --member="serviceAccount:zalando-external-dns-test.svc.id.goog[external-dns/external-dns]" \ - --role=roles/iam.workloadIdentityUser +gcloud iam service-accounts create $DNS_SA_NAME --display-name $DNS_SA_NAME +gcloud projects add-iam-policy-binding $DNS_PROJECT_ID \ + --member serviceAccount:$DNS_SA_EMAIL --role "roles/dns.admin" ``` -Create a DNS zone which will contain the managed DNS records. +#### Link KSA to GSA + +Add an IAM policy binding bewtween the workload identity GSA and ExternalDNS GSA. This will link the ExternalDNS KSA to ExternalDNS GSA. -```console -$ gcloud dns managed-zones create external-dns-test-gcp-zalan-do \ - --dns-name=external-dns-test.gcp.zalan.do. \ - --description="Automatically managed zone by ExternalDNS" +```bash +gcloud iam service-accounts add-iam-policy-binding $DNS_SA_EMAIL \ + --role "roles/iam.workloadIdentityUser" \ + --member "serviceAccount:$GKE_PROJECT_ID.svc.id.goog[$EXTENALDNS_NS/external-dns]" ``` -Make a note of the nameservers that were assigned to your new zone. +#### Deploy External DNS -```console -$ gcloud dns record-sets list \ - --zone=external-dns-test-gcp-zalan-do \ - --name=external-dns-test.gcp.zalan.do. \ - --type NS -NAME TYPE TTL DATA -external-dns-test.gcp.zalan.do. NS 21600 ns-cloud-e1.googledomains.com.,ns-cloud-e2.googledomains.com.,ns-cloud-e3.googledomains.com.,ns-cloud-e4.googledomains.com. -``` +Deploy ExternalDNS with the following steps below, documented under [Deploy ExternalDNS](#deploy-externaldns). Set the `--google-project` flag to the Cloud DNS project name. -In this case it's `ns-cloud-{e1-e4}.googledomains.com.` but your's could -slightly differ, e.g. `{a1-a4}`, `{b1-b4}` etc. +#### Link KSA to GSA in Kubernetes -Tell the parent zone where to find the DNS records for this zone by adding the -corresponding NS records there. Assuming the parent zone is "gcp-zalan-do" and -the domain is "gcp.zalan.do" and that it's also hosted at Google we would do the -following. +Add the proper workload identity annotation to the ExternalDNS KSA. -```console -$ gcloud dns record-sets transaction start --zone=gcp-zalan-do -$ gcloud dns record-sets transaction add ns-cloud-e{1..4}.googledomains.com. \ - --name=external-dns-test.gcp.zalan.do. --ttl 300 --type NS --zone=gcp-zalan-do -$ gcloud dns record-sets transaction execute --zone=gcp-zalan-do +```bash +kubectl annotate serviceaccount "external-dns" \ + --namespace $EXTENALDNS_NS \ + "iam.gke.io/gcp-service-account=$DNS_SA_EMAIL" ``` -Connect your `kubectl` client to the cluster you just created and bind your GCP -user to the cluster admin role in Kubernetes. +#### Update ExternalDNS pods -```console -$ gcloud container clusters get-credentials external-dns -$ kubectl create clusterrolebinding cluster-admin-me \ - --clusterrole=cluster-admin --user="$(gcloud config get-value account)" +Update the Pod spec to schedule the workloads on nodes that use Workload Identity and to use the annotated Kubernetes service account. + +```bash +kubectl patch deployment "external-dns" \ + --namespace $EXTENALDNS_NS \ + --patch \ + '{"spec": {"template": {"spec": {"nodeSelector": {"iam.gke.io/gke-metadata-server-enabled": "true"}}}}}' ``` -### Deploy ExternalDNS +After all of these steps you may see several messages with `googleapi: Error 403: Forbidden, forbidden`. After several minutes when the token is refreshed, these error messages will go away, and you should see info messages, such as: `All records are already up to date`. + +## Deploy ExternalDNS -Apply the following manifest file to deploy external-dns. +Then apply the following manifests file to deploy ExternalDNS. ```yaml apiVersion: v1 -kind: Namespace -metadata: - name: external-dns ---- -apiVersion: v1 kind: ServiceAccount metadata: name: external-dns - namespace: external-dns --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole @@ -396,14 +277,11 @@ metadata: name: external-dns rules: - apiGroups: [""] - resources: ["services", "endpoints", "pods"] - verbs: ["get", "watch", "list"] - - apiGroups: ["extensions", "networking.k8s.io"] + resources: ["services","endpoints","pods","nodes"] + verbs: ["get","watch","list"] + - apiGroups: ["extensions","networking.k8s.io"] resources: ["ingresses"] - verbs: ["get", "watch", "list"] - - apiGroups: [""] - resources: ["nodes"] - verbs: ["list"] + verbs: ["get","watch","list"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding @@ -416,13 +294,12 @@ roleRef: subjects: - kind: ServiceAccount name: external-dns - namespace: external-dns + namespace: default # change if namespace is not 'default' --- apiVersion: apps/v1 kind: Deployment metadata: name: external-dns - namespace: external-dns spec: strategy: type: Recreate @@ -434,65 +311,51 @@ spec: labels: app: external-dns spec: + serviceAccountName: external-dns containers: - - args: - - --source=ingress + - name: external-dns + image: k8s.gcr.io/external-dns/external-dns:v0.11.0 + args: - --source=service - - --domain-filter=external-dns-test.gcp.zalan.do + - --source=ingress + - --domain-filter=example.com # will make ExternalDNS see only the hosted zones matching provided domain, omit to process all available hosted zones - --provider=google - - --google-project=zalando-external-dns-test + # - --google-project=my-cloud-dns-project # Use this to specify a project different from the one external-dns is running inside + - --google-zone-visibility=public # Use this to filter to only zones with this visibility. Set to either 'public' or 'private'. Omitting will match public and private zones + - --policy=upsert-only # would prevent ExternalDNS from deleting any records, omit to enable full synchronization - --registry=txt - --txt-owner-id=my-identifier - image: k8s.gcr.io/external-dns/external-dns:v0.8.0 - name: external-dns - securityContext: - fsGroup: 65534 - runAsUser: 65534 - serviceAccountName: external-dns ``` -Then add the proper workload identity annotation to the cert-manager service -account. +Create the deployment for ExternalDNS: ```bash -$ kubectl annotate serviceaccount --namespace=external-dns external-dns \ - "iam.gke.io/gcp-service-account=$sa_email" +kubectl create --namespace "default" --filename externaldns.yaml ``` -### Deploy a sample application +## Verify ExternalDNS works -Create the following sample application to test that ExternalDNS works. +The following will deploy a small nginx server that will be used to demonstrate that ExternalDNS is working. + +### Verify using an external load balancer + +Create the following sample application to test that ExternalDNS works. This example will provision a L4 load balancer. ```yaml -apiVersion: networking.k8s.io/v1 -kind: Ingress -metadata: - name: nginx -spec: - rules: - - host: via-ingress.external-dns-test.gcp.zalan.do - http: - paths: - - backend: - service: - name: nginx - port: - number: 80 - pathType: Prefix ---- apiVersion: v1 kind: Service metadata: - annotations: - external-dns.alpha.kubernetes.io/hostname: nginx.external-dns-test.gcp.zalan.do. name: nginx + annotations: + # change nginx.example.com to match an appropriate value + external-dns.alpha.kubernetes.io/hostname: nginx.example.com spec: + type: LoadBalancer ports: - - port: 80 - targetPort: 80 + - port: 80 + targetPort: 80 selector: app: nginx - type: LoadBalancer --- apiVersion: apps/v1 kind: Deployment @@ -508,78 +371,118 @@ spec: app: nginx spec: containers: - - image: nginx - name: nginx - ports: - - containerPort: 80 + - image: nginx + name: nginx + ports: + - containerPort: 80 ``` -After roughly two minutes check that a corresponding DNS records for your -service and ingress were created. +Create the deployment and service objects: -```console -$ gcloud dns record-sets list \ - --zone "external-dns-test-gcp-zalan-do" \ - --name "via-ingress.external-dns-test.gcp.zalan.do." \ - --type A -NAME TYPE TTL DATA -nginx.external-dns-test.gcp.zalan.do. A 300 104.155.60.49 -nginx.external-dns-test.gcp.zalan.do. TXT 300 "heritage=external-dns,external-dns/owner=my-identifier" -via-ingress.external-dns-test.gcp.zalan.do. TXT 300 "heritage=external-dns,external-dns/owner=my-identifier" -via-ingress.external-dns-test.gcp.zalan.do. A 300 35.187.1.246 +```bash +kubectl create --namespace "default" --filename nginx.yaml ``` -Let's check that we can resolve this DNS name as well. +After roughly two minutes check that a corresponding DNS record for your service was created. -```console -$ dig +short @ns-cloud-e1.googledomains.com. via-ingress.external-dns-test.gcp.zalan.do. -35.187.1.246 +```bash +gcloud dns record-sets list --zone "example-com" --name "nginx.example.com." ``` -Try with `curl` as well. +Example output: -```console -$ curl via-ingress.external-dns-test.gcp.zalan.do - - - -Welcome to nginx! -... - - -... - - +``` +NAME TYPE TTL DATA +nginx.example.com. A 300 104.155.60.49 +nginx.example.com. TXT 300 "heritage=external-dns,external-dns/owner=my-identifier" ``` -### Clean up +Note created `TXT` record alongside `A` record. `TXT` record signifies that the corresponding `A` record is managed by ExternalDNS. This makes ExternalDNS safe for running in environments where there are other records managed via other means. -Make sure to delete all service and ingress objects before terminating the -cluster so all load balancers and DNS entries get cleaned up correctly. +Let's check that we can resolve this DNS name. We'll ask the nameservers assigned to your zone first. -```console -$ kubectl delete ingress nginx -$ kubectl delete service nginx +```bash +dig +short @ns-cloud-e1.googledomains.com. nginx.example.com. +104.155.60.49 ``` -Give ExternalDNS some time to clean up the DNS records for you. Then delete the -managed zone and cluster. +Given you hooked up your DNS zone with its parent zone you can use `curl` to access your site. + +```bash +curl nginx.example.com +``` + +### Verify using an ingress + +Let's check that Ingress works as well. Create the following Ingress. -```console -$ gcloud dns managed-zones delete external-dns-test-gcp-zalan-do -$ gcloud container clusters delete external-dns +```yaml +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: nginx +spec: + rules: + - host: server.example.com + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: nginx-svc + port: + number: 80 ``` -Also delete the NS records for your removed zone from the parent zone. +Create the ingress objects with: -```console -$ gcloud dns record-sets transaction start --zone gcp-zalan-do -$ gcloud dns record-sets transaction remove ns-cloud-e{1..4}.googledomains.com. \ - --name=external-dns-test.gcp.zalan.do. --ttl 300 --type NS --zone=gcp-zalan-do -$ gcloud dns record-sets transaction execute --zone=gcp-zalan-do +```bash +kubectl create --namespace "default" --filename ingress.yaml ``` -## User Demo How-To Blogs and Examples +Note that this will ingress object will use the default ingress controller that comes with GKE to create a L7 load balancer in addition to the L4 load balancer previously with the service object. To use only the L7 load balancer, update the service manafest to change the Service type to `NodePort` and remove the ExternalDNS annotation. + +After roughly two minutes check that a corresponding DNS record for your Ingress was created. + +```bash +gcloud dns record-sets list \ + --zone "example-com" \ + --name "server.example.com." \ +``` +Output: -* A full demo on GKE Kubernetes + CloudDNS + SA-Permissions [How-to Kubernetes with DNS management (ssl-manager pre-req)](https://medium.com/@jpantjsoha/how-to-kubernetes-with-dns-management-for-gitops-31239ea75d8d) -* Run external-dns on GKE with workload identity. See [Kubernetes, ingress-nginx, cert-manager & external-dns](https://blog.atomist.com/kubernetes-ingress-nginx-cert-manager-external-dns/) +``` +NAME TYPE TTL DATA +server.example.com. A 300 130.211.46.224 +server.example.com. TXT 300 "heritage=external-dns,external-dns/owner=my-identifier" +``` + +Let's check that we can resolve this DNS name as well. + +```bash +dig +short @ns-cloud-e1.googledomains.com. server.example.com. +130.211.46.224 +``` + +Try with `curl` as well. + +```bash +curl server.example.com +``` + +### Clean up + +Make sure to delete all Service and Ingress objects before terminating the cluster so all load balancers get cleaned up correctly. + +```bash +kubectl delete service nginx +kubectl delete ingress nginx +``` + +Give ExternalDNS some time to clean up the DNS records for you. Then delete the managed zone and cluster. + +```bash +gcloud dns managed-zones delete "example-com" +gcloud container clusters delete "external-dns" +``` From d179cff5e3ceb3d61db8badb5ab42641ed3a6a7e Mon Sep 17 00:00:00 2001 From: joaquin Date: Sun, 22 May 2022 19:12:01 -0700 Subject: [PATCH 2/3] change spelling of EXTENALDNS_NS --- docs/tutorials/gke.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/docs/tutorials/gke.md b/docs/tutorials/gke.md index 80e26fbef1..19c9591db3 100644 --- a/docs/tutorials/gke.md +++ b/docs/tutorials/gke.md @@ -201,9 +201,7 @@ gcloud iam service-accounts keys create /local/path/to/credentials.json \ Create a Kubernetes secret with the credentials in the same namespace of ExternalDNS. ```bash -EXTENALDNS_NS="default" - -kubectl create secret generic "external-dns" --namespace $EXTENALDNS_NS \ +kubectl create secret generic "external-dns" --namespace ${EXTERNALDNS_NS:-"default"} \ --from-file /local/path/to/credentials.json ``` @@ -231,7 +229,7 @@ Add an IAM policy binding bewtween the workload identity GSA and ExternalDNS GSA ```bash gcloud iam service-accounts add-iam-policy-binding $DNS_SA_EMAIL \ --role "roles/iam.workloadIdentityUser" \ - --member "serviceAccount:$GKE_PROJECT_ID.svc.id.goog[$EXTENALDNS_NS/external-dns]" + --member "serviceAccount:$GKE_PROJECT_ID.svc.id.goog[${EXTERNALDNS_NS:-"default"}/external-dns]" ``` #### Deploy External DNS @@ -244,7 +242,7 @@ Add the proper workload identity annotation to the ExternalDNS KSA. ```bash kubectl annotate serviceaccount "external-dns" \ - --namespace $EXTENALDNS_NS \ + --namespace ${EXTERNALDNS_NS:-"default"} \ "iam.gke.io/gcp-service-account=$DNS_SA_EMAIL" ``` @@ -254,7 +252,7 @@ Update the Pod spec to schedule the workloads on nodes that use Workload Identit ```bash kubectl patch deployment "external-dns" \ - --namespace $EXTENALDNS_NS \ + --namespace ${EXTERNALDNS_NS:-"default"} \ --patch \ '{"spec": {"template": {"spec": {"nodeSelector": {"iam.gke.io/gke-metadata-server-enabled": "true"}}}}}' ``` From 73e657df5aaf2ad2f52260290bbd587cf92421f2 Mon Sep 17 00:00:00 2001 From: joaquin Date: Sun, 22 May 2022 20:02:49 -0700 Subject: [PATCH 3/3] use recommended labels and config for mounting secret credentials.json --- docs/tutorials/gke.md | 26 ++++++++++++++++++++++---- 1 file changed, 22 insertions(+), 4 deletions(-) diff --git a/docs/tutorials/gke.md b/docs/tutorials/gke.md index 19c9591db3..902d4ceeb6 100644 --- a/docs/tutorials/gke.md +++ b/docs/tutorials/gke.md @@ -205,8 +205,7 @@ kubectl create secret generic "external-dns" --namespace ${EXTERNALDNS_NS:-"defa --from-file /local/path/to/credentials.json ``` -After this, follow the steps in [Deploy ExternalDNS](#deploy-externaldns). Make sure to set the `--google-project` flag to match Cloud DNS project name. - +After this, follow the steps in [Deploy ExternalDNS](#deploy-externaldns). Make sure to set the `--google-project` flag to match Cloud DNS project name. Make sure to uncomment out the section that mounts the secret to the ExternalDNS pods. ### Workload Identity [Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) allows workloads in your GKE cluster to impersonate GSA (Google Service Accounts) using KSA (Kubernetes Service Accounts) configured during deployemnt. These are the steps to use this feature with ExternalDNS. @@ -268,11 +267,15 @@ apiVersion: v1 kind: ServiceAccount metadata: name: external-dns + labels: + app.kubernetes.io/name: external-dns --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: external-dns + labels: + app.kubernetes.io/name: external-dns rules: - apiGroups: [""] resources: ["services","endpoints","pods","nodes"] @@ -285,6 +288,8 @@ apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: external-dns-viewer + labels: + app.kubernetes.io/name: external-dns roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole @@ -298,16 +303,18 @@ apiVersion: apps/v1 kind: Deployment metadata: name: external-dns + labels: + app.kubernetes.io/name: external-dns spec: strategy: type: Recreate selector: matchLabels: - app: external-dns + app.kubernetes.io/name: external-dns template: metadata: labels: - app: external-dns + app.kubernetes.io/name: external-dns spec: serviceAccountName: external-dns containers: @@ -323,6 +330,17 @@ spec: - --policy=upsert-only # would prevent ExternalDNS from deleting any records, omit to enable full synchronization - --registry=txt - --txt-owner-id=my-identifier + # # uncomment below if static credentials are used + # env: + # - name: GOOGLE_APPLICATION_CREDENTIALS + # value: /etc/secrets/service-account/credentials.json + # volumeMounts: + # - name: google-service-account + # mountPath: /etc/secrets/service-account/ + # volumes: + # - name: google-service-account + # secret: + # secretName: external-dns ``` Create the deployment for ExternalDNS: