Skip to content
This repository has been archived by the owner on Aug 10, 2023. It is now read-only.

Latest commit

 

History

History
509 lines (350 loc) · 20.5 KB

terraform-asm-mcp.md

File metadata and controls

509 lines (350 loc) · 20.5 KB
Raw
title description author tags date_published
Install Anthos Service Mesh with a Google-managed control plane on GKE with Terraform
Use Terraform to deploy a Kubernetes Engine cluster and install Anthos Service Mesh with a Google-managed control plane.
alizaidis
Kubernetes Engine, ASM, MCP, Managed
2022-01-21

Ali Zaidi | Solutions Architect | Google

Contributed by Google employees.

This tutorial shows you how to install managed Anthos Service Mesh with a Google-managed control plane on two Google Kubernetes Engine (GKE) clusters using the Anthos Service Mesh Terraform submodule. This tutorial has been tested in Google Cloud Shell. Cloud Shell has all of the tools that you need to perform the following steps.

Objectives

  • Prepare Terraform.
  • Build a Virtual Private Cloud (VPC) network.
  • Create two GKE clusters.
  • Install Anthos Service Mesh.
  • Deploy the sample Online Boutique application.
  • Monitor application golden signals.

Costs

This tutorial uses the following Google Cloud products:

Use the pricing calculator to generate a cost estimate based on your projected usage.

Before you begin

This guide assumes that you have owner IAM permissions for your Google Cloud project. In production, you do not require owner permission.

  1. Select or create a Google Cloud project.

  2. Verify that billing is enabled for your project.

Set up your environment

Follow these steps to set up your environment.

  1. In the Google Cloud Console, activate Cloud Shell.

  2. Create a WORKDIR folder:

    mkdir asm-terraform-tutorial
cd asm-terraform-tutorial
export WORKDIR=$(pwd)

  3. Define variables used in this tutorial:

    export PROJECT_ID=PROJECT ID
export REPO_URL="https://gitlab.com/asm7/asm-terraform"
export VPC="vpc"
export GKE1="gke1"
export GKE2="gke2"
export REGION="us-central1"
export GKE1_LOCATION="${REGION}-a"
export GKE2_LOCATION="${REGION}-b"
export GKE1_CTX="gke_${PROJECT_ID}_${GKE1_LOCATION}_${GKE1}"
export GKE2_CTX="gke_${PROJECT_ID}_${GKE2_LOCATION}_${GKE2}"
export GKE1_KUBECONFIG="${WORKDIR}/gke1_kubeconfig"
export GKE2_KUBECONFIG="${WORKDIR}/gke2_kubeconfig"
export GKE_CHANNEL="REGULAR"
export ASM_CHANNEL="regular"
export CNI_ENABLED="true"
export ASM_GATEWAYS_NAMESPACE="asm-gateways"

    Replace the value of PROJECT_ID with your project ID.

    You can use the ASM_CHANNEL variable to set up Anthos Service Mesh with Regular, Rapid, or Stable release channels. For more information, see Anthos Service Mesh control plane revisions.

    To support GKE Autopilot clusters, CNI_ENABLED must be true.

  4. Configure your Google Cloud project:

    gcloud config set project "${PROJECT_ID}"

  5. Enable the Google Cloud APIs required for this tutorial:

    gcloud --project="${PROJECT_ID}" services enable \
    container.googleapis.com \
    compute.googleapis.com \
    gkehub.googleapis.com \
    cloudresourcemanager.googleapis.com

Set up the VPC network and GKE clusters with Terraform

Follow these steps to set up the VPC network and GKE clusters with Terraform.

  1. Set up Terraform authentication:

    gcloud auth application-default login --no-launch-browser

    For more information, see Google Provider Configuration Reference.

  2. Clone the repository and go to the tutorial directory:

    git clone "${REPO_URL}" ${WORKDIR}/asm-terraform
cd ${WORKDIR}/asm-terraform
git checkout aa/tutorial
cd tutorial

  3. Prepare the VPC and GKE terraform modules:

    cd vpc-gke
envsubst < variables.tf.tmpl > variables.tf
envsubst < provider.tf.tmpl > provider.tf

    This step populates the variables and provider files with the variables defined at the beginning of the tutorial.

  4. Deploy the VPC and GKE Terraform module:

    terraform init && \
terraform plan && \
terraform apply --auto-approve

    The deployment can take up to 10 minutes to complete. This module also exports the kubeconfig using the gke_auth module for the two GKE clusters as local_file resources. These kubeconfig files are used in the Anthos Service Mesh module later.

  5. Inspect the deployed resources using gcloud commands starting from the global VPC network called vpc:

    gcloud compute networks describe vpc

    The output is similar to the following:

    autoCreateSubnetworks: false
creationTimestamp: '2021-11-16T20:37:54.520-08:00'
id: '932232537524746701'
kind: compute#network
name: vpc
routingConfig:
  routingMode: GLOBAL
selfLink: https://www.googleapis.com/compute/v1/projects/zl-asm-exp2688feb2/global/networks/vpc
subnetworks:
- https://www.googleapis.com/compute/v1/projects/zl-asm-exp2688feb2/regions/us-central1/subnetworks/subnet-01
x_gcloud_bgp_routing_mode: GLOBAL
x_gcloud_subnet_mode: CUSTOM

  6. List the GKE clusters:

    gcloud container clusters list

    Take note of the MASTER_IP value for each. This is the cluster endpoint, which you will need in the following Anthos Service Mesh module.

    The output is similar to the following:

    NAME: gke1
LOCATION: us-central1-a
MASTER_VERSION: 1.21.5-gke.1802
MASTER_IP: 104.154.57.122
MACHINE_TYPE: e2-standard-4
NODE_VERSION: 1.21.5-gke.1802
NUM_NODES: 2
STATUS: RUNNING

NAME: gke2
LOCATION: us-central1-b
MASTER_VERSION: 1.21.5-gke.1802
MASTER_IP: 23.251.149.3
MACHINE_TYPE: e2-standard-4
NODE_VERSION: 1.21.5-gke.1802
NUM_NODES: 2
STATUS: RUNNING

  7. Verify that the cluster endpoint in each of the generated kubeconfig files matches the value in the previous step:

    • GKE1:

      cat ${WORKDIR}/gke1_kubeconfig

      The output is similar to the following:

          server: https://104.154.57.122
  name: gke1
contexts:
- context:
    cluster: gke1
    user: gke1
  name: gke1

    • GKE2:

      cat ${WORKDIR}/gke2_kubeconfig

      The output is similar to the following:

          server: https://23.251.149.3
  name: gke2
contexts:
- context:
    cluster: gke2
    user: gke2
  name: gke2

Set up Anthos Service Mesh

Follow these steps to set up Anthos Service Mesh.

  1. Register the clusters to a fleet and enable the mesh feature:

    cd ${WORKDIR}/asm-terraform/tutorial/hub-mesh
envsubst < variables.tf.tmpl > variables.tf
envsubst < provider.tf.tmpl > provider.tf

terraform init && \
terraform plan && \
terraform apply --auto-approve

    This step may take a few minutes to complete.

    For more information, see Fleets.

  2. Inspect the deployed resources using gcloud commands:

    gcloud container hub memberships list

    The output is similar to the following:

    NAME: gke1
EXTERNAL_ID: d2f6bf3b-b0df-47be-b841-39b0f4bc25ba

NAME: gke2
EXTERNAL_ID: 84724e99-9acf-45cc-954c-a4db53c7ecf3

    For more details on cluster registration see Registering a cluster.

  3. Verify that Anthos Service Mesh is enabled:

    gcloud beta container hub mesh describe

    The output is similar to the following:

    createTime: '2021-11-17T05:24:36.150402113Z'
membershipStates:
  projects/132310186441/locations/global/memberships/gke1:
    state:
      code: OK
      description: Please see https://cloud.google.com/service-mesh/docs/install for
        instructions to onboard to Anthos Service Mesh.
      updateTime: '2021-11-17T05:26:58.175206417Z'
  projects/132310186441/locations/global/memberships/gke2:
    state:
      code: OK
      description: Please see https://cloud.google.com/service-mesh/docs/install for
        instructions to onboard to Anthos Service Mesh.
      updateTime: '2021-11-17T05:26:59.778268604Z'
name: projects/zl-asm-exp2688feb2/locations/global/features/servicemesh
resourceState:
  state: ACTIVE
spec: {}
updateTime: '2021-11-17T05:27:02.350986468Z'

  4. Verify that you have the ControlPlaneRevision custom resource definition (CRD) in both GKE clusters. The ControlPlaneRevision custom resource is used to deploy managed Anthos Service Mesh.

    • Verify GKE1:

      gcloud --project=${PROJECT_ID} container clusters get-credentials ${GKE1} --zone ${GKE1_LOCATION}
kubectl  wait --for=condition=established crd controlplanerevisions.mesh.cloud.google.com --timeout=5m

    • Verify GKE2:

      gcloud --project=${PROJECT_ID} container clusters get-credentials ${GKE2} --zone ${GKE2_LOCATION}
kubectl  wait --for=condition=established crd controlplanerevisions.mesh.cloud.google.com --timeout=5m

    The output is similar to the following:

    customresourcedefinition.apiextensions.k8s.io/controlplanerevisions.mesh.cloud.google.com condition met

  5. Install Anthos Service Mesh on the GKE clusters:

    cd ${WORKDIR}/asm-terraform/tutorial/asm
envsubst < variables.tf.tmpl > variables.tf
envsubst < provider.tf.tmpl > provider.tf

terraform init && \
terraform plan && \
terraform apply --auto-approve

export ASM_LABEL=$(terraform output asm_label | tr -d '"')

    This module also configures multi-cluster mesh by configuring cross-cluster kubeconfig secrets.

  6. Ensure that Anthos Service Mesh provisioning finishes successfully:

    kubectl --context=${GKE1_CTX} wait --for=condition=ProvisioningFinished controlplanerevision ${ASM_LABEL} -n istio-system --timeout=10m
kubectl --context=${GKE2_CTX} wait --for=condition=ProvisioningFinished controlplanerevision ${ASM_LABEL} -n istio-system --timeout=10m

    This step can take a few minutes to complete.

    The output is similar to the following:

    controlplanerevision.mesh.cloud.google.com/asm-managed condition met

  7. Inspect the deployed resources:

    kubectl get ns --context=${GKE1_CTX}
kubectl get ns --context=${GKE2_CTX}

    The output is similar to the following:

    NAME              STATUS   AGE
default           Active   23m
istio-system      Active   8m30s
kube-node-lease   Active   23m
kube-public       Active   23m
kube-system       Active   23m

    The istio-system namespace should be present on both clusters.

  8. Inspect the status of the ControlPlaneRevision custom resource:

    kubectl describe controlplanerevision ${ASM_LABEL} -n istio-system --context=${GKE1_CTX}
kubectl describe controlplanerevision ${ASM_LABEL} -n istio-system --context=${GKE2_CTX}

    The output is similar to the following:

    Name:         asm-managed
Namespace:    istio-system

...

Status:
  Conditions:
    Last Transition Time:  2021-11-17T04:58:29Z
    Message:               The provisioning process has completed successfully
    Reason:                Provisioned
    Status:                True
    Type:                  Reconciled
    Last Transition Time:  2021-11-17T04:58:29Z
    Message:               Provisioning has finished
    Reason:                ProvisioningFinished
    Status:                True
    Type:                  ProvisioningFinished
    Last Transition Time:  2021-11-17T04:58:29Z
    Message:               Provisioning has not stalled
    Reason:                NotStalled
    Status:                False
    Type:                  Stalled

    This is a useful resource to observe when installing or upgrading Anthos Service Mesh. For more information, see What is a revision.

  9. Inspect the cluster credentials for each cluster stored as opaque secrets in the istio-system namespace present in the other cluster. These are required for cross-cluster service discovery:

    kubectl describe secret gke2-secret-kubeconfig -n istio-system --context=${GKE1_CTX}
kubectl describe secret gke1-secret-kubeconfig -n istio-system --context=${GKE2_CTX}

    The output is similar to the following:

    Name:         gke1-secret-kubeconfig
Namespace:    istio-system
Labels:       istio/multiCluster=true
Annotations:  networking.istio.io/cluster: gke1

Type:  Opaque

Data
====
gke1:  3232 bytes

Configure multi-cluster Anthos Service Mesh

Follow these steps to configure multi-cluster Anthos Service Mesh.

  1. Deploy Anthos Service Mesh ingress gateways in both clusters:

    cd ${WORKDIR}/asm-terraform/tutorial/asm-gateways
envsubst < variables.tf.tmpl > variables.tf
envsubst < provider.tf.tmpl > provider.tf

terraform init && \
terraform plan && \
terraform apply --auto-approve

  2. Confirm that both Anthos Service Mesh ingress gateways are running:

    kubectl --context=${GKE1_CTX} -n ${ASM_GATEWAYS_NAMESPACE} wait --for=condition=available --timeout=5m deployment asm-ingressgateway
kubectl --context=${GKE2_CTX} -n ${ASM_GATEWAYS_NAMESPACE} wait --for=condition=available --timeout=5m deployment asm-ingressgateway

    The output is similar to the following:

    deployment.apps/asm-ingressgateway condition met

  3. Inspect the deployment and service for Anthos Service Mesh ingress gateways on each cluster:

    • Inspect GKE1:

      kubectl --context=${GKE1_CTX} -n ${ASM_GATEWAYS_NAMESPACE} get deploy
kubectl --context=${GKE2_CTX} -n ${ASM_GATEWAYS_NAMESPACE} get deploy

      The output is similar to the following:

      NAME                 READY   UP-TO-DATE   AVAILABLE   AGE
asm-ingressgateway   1/1     1            1           4m19s

    • Inspect GKE2:

      kubectl --context=${GKE1_CTX} -n ${ASM_GATEWAYS_NAMESPACE} get service
kubectl --context=${GKE2_CTX} -n ${ASM_GATEWAYS_NAMESPACE} get service

      The output is similar to the following:

      NAME                 TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)                      AGE
asm-ingressgateway   LoadBalancer   10.100.1.124   35.192.78.53   80:32514/TCP,443:30712/TCP   4m30s

Deploy a sample application

In this section, you deploy a sample application (Online Boutique) on both GKE1 and GKE2 to verify the multi-cluster mesh.

  1. Create the online-boutique namespace on both clusters:

    cat <<EOF > ${WORKDIR}/namespace-online-boutique.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: online-boutique
  labels:
    istio.io/rev: ${ASM_LABEL}
EOF

kubectl --context=${GKE1_CTX} apply -f ${WORKDIR}/namespace-online-boutique.yaml
kubectl --context=${GKE2_CTX} apply -f ${WORKDIR}/namespace-online-boutique.yaml

  2. Clone the sample application repository and deploy Kubernetes manifests to both clusters:

    git clone https://github.com/GoogleCloudPlatform/microservices-demo.git ${WORKDIR}/online-boutique
kubectl --context=${GKE1_CTX} -n online-boutique apply -f ${WORKDIR}/online-boutique/release/kubernetes-manifests.yaml
kubectl --context=${GKE2_CTX} -n online-boutique apply -f ${WORKDIR}/online-boutique/release/kubernetes-manifests.yaml

  3. Delete some deployments from each cluster to set up the application architecture with services communicating transparently across clusters:

    kubectl --context=${GKE1_CTX} -n online-boutique delete deployment adservice
kubectl --context=${GKE1_CTX} -n online-boutique delete deployment cartservice
kubectl --context=${GKE1_CTX} -n online-boutique delete deployment redis-cart
kubectl --context=${GKE1_CTX} -n online-boutique delete deployment currencyservice
kubectl --context=${GKE1_CTX} -n online-boutique delete deployment emailservice

kubectl --context=${GKE2_CTX} -n online-boutique delete deployment paymentservice
kubectl --context=${GKE2_CTX} -n online-boutique delete deployment productcatalogservice
kubectl --context=${GKE2_CTX} -n online-boutique delete deployment shippingservice
kubectl --context=${GKE2_CTX} -n online-boutique delete deployment checkoutservice
kubectl --context=${GKE2_CTX} -n online-boutique delete deployment recommendationservice

  4. Wait for all deployments to be ready:

    kubectl --context=${GKE1_CTX} -n online-boutique wait --for=condition=available --timeout=5m --all deployments
kubectl --context=${GKE2_CTX} -n online-boutique wait --for=condition=available --timeout=5m --all deployments

    The output is similar to the following:

    deployment.apps/adservice condition met
deployment.apps/cartservice condition met
deployment.apps/currencyservice condition met
deployment.apps/emailservice condition met
deployment.apps/frontend condition met
deployment.apps/loadgenerator condition met
deployment.apps/redis-cart condition met

  5. Deploy Anthos Service Mesh manifests to both clusters:

    kubectl --context=${GKE1_CTX} -n online-boutique apply -f ${WORKDIR}/asm-terraform/tutorial/online-boutique/asm-manifests.yaml
kubectl --context=${GKE2_CTX} -n online-boutique apply -f ${WORKDIR}/asm-terraform/tutorial/online-boutique/asm-manifests.yaml

  6. Access Online Boutique through the Anthos Service Mesh ingress:

    export GKE1_ASM_INGRESS_IP=$(kubectl --context=${GKE1_CTX} --namespace ${ASM_GATEWAYS_NAMESPACE} get svc asm-ingressgateway -o jsonpath={.status.loadBalancer.ingress..ip})
export GKE2_ASM_INGRESS_IP=$(kubectl --context=${GKE2_CTX} --namespace ${ASM_GATEWAYS_NAMESPACE} get svc asm-ingressgateway -o jsonpath={.status.loadBalancer.ingress..ip})

echo -e "GKE1 ASM Ingressgateway IP is ${GKE1_ASM_INGRESS_IP} accessible at http://${GKE1_ASM_INGRESS_IP}"
echo -e "GKE2 ASM Ingressgateway IP is ${GKE2_ASM_INGRESS_IP} accessible at http://${GKE2_ASM_INGRESS_IP}"

    Verify multicluster mesh service discovery and routing by accessing the Online Boutique application through the Anthos Service Mesh ingress gateways. You can access the application through either IP address. Browse through the application to observe behavior from each endpoint. The behavior should be the same.

Monitor application golden signals

Inspect service dashboards by accessing the link generated in the following command:

echo -e "https://console.cloud.google.com/anthos/services?project=${PROJECT_ID}"

Explore the topology and metrics for services by switching to Topology view located in the top right section of the page.

Clean up

The easiest way to prevent continued billing for the resources that you created for this tutorial is to delete the project you created for the tutorial.

To delete a project, do the following:

  1. In the Cloud Console, go to the Projects page.
  2. In the project list, select the project you want to delete and click Delete.
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

What's next