Skip to content
This repository has been archived by the owner on Apr 2, 2024. It is now read-only.

Ingress for Anthos

Jump to bottom
Yuriy Lesyuk edited this page Nov 10, 2020 · 1 revision

STATUS: Draft; TASK: normalize to the AHR-* conventions;

Multi-Region: Ingress for Anthos

"Ingress for Anthos is designed to meet the load balancing needs of multi-cluster, multi-regional environments. It's a controller for the external HTTP(S) load balancer to provide ingress for traffic coming from the internet across one or more clusters." [Ingress for Anthos, Concepts]

"Ingress for Anthos builds on the architecture of the External HTTP(S) Load Balancing. HTTP(S) Load Balancing is a globally distributed load balancer with proxies deployed at 100+ Google points of presence (PoPs) around the world."

NOTE: Prerequisites for registering a cluster https://cloud.google.com/anthos/multicluster-management/connect/prerequisites

If not roles/owner, then those roles are required:

    roles/gkehub.admin
    roles/iam.serviceAccountAdmin
    roles/iam.serviceAccountKeyAdmin
    roles/resourcemanager.projectIamAdmin

Required APIs

gcloud services enable --project=$PROJECT \
   container.googleapis.com \
   gkeconnect.googleapis.com \
   gkehub.googleapis.com \
   cloudresourcemanager.googleapis.com

Grant users the cluster-admin RBAC role. Check it

kubectl create clusterrolebinding [BINDING_NAME] --clusterrole cluster-admin --user [USER] 

# check role binding for both region clusters
kubectl --context=dc1-cluster auth can-i '*' '*' --all-namespaces
kubectl --context=dc2-cluster auth can-i '*' '*' --all-namespaces

Enable APIs: gkehub and multiclusteringress

?. Verify gkehub API status

gcloud services list --available | grep gkehub.googleapis.com

gkehub.googleapis.com                      GKE Hub

?. Enable gkehub API

gcloud services enable gkehub.googleapis.com

?. Enable the Ingress for Anthos API for your project:

gcloud services enable multiclusteringress.googleapis.com

Create a Google Cloud service account using gcloud

export GKEHUB_ID=gkehub
export GKEHUB_SA=$SA_DIR/$ORG-gkehub.json
export GKEHUB_SA_ID=$GKEHUB_ID@$PROJECT.iam.gserviceaccount.com

gcloud iam service-accounts create $GKEHUB_ID --project=$PROJECT

gcloud projects add-iam-policy-binding $PROJECT --member="serviceAccount:$GKEHUB_SA_ID" --role="roles/gkehub.connect"

gcloud iam service-accounts keys create $GKEHUB_SA --iam-account=$GKEHUB_SA_ID --project=$PROJECT

Registering your clusters

Each cluster must be registered as a member of a Hub.

?. Find the URIs for your clusters

gcloud container clusters list --uri
export CLUSTER_URI=$(gcloud container clusters list --uri --filter="name=$CLUSTER")
export DC1_CLUSTER_URI=https://container.googleapis.com/v1/projects/emea-cs-hybrid-demo6/zones/us-east1-b/clusters/dc1-cluster

export DC2_CLUSTER_URI=https://container.googleapis.com/v1/projects/emea-cs-hybrid-demo6/zones/asia-east1-b/clusters/dc2-cluster

?. Register the dc1-cluster

# dc1-cluster
(
export CLUSTER=dc1-cluster
export CLUSTER_URI=$(gcloud container clusters list --uri --filter="name=$CLUSTER")

gcloud container hub memberships register $CLUSTER --project=$PROJECT --gke-uri=$CLUSTER_URI --service-account-key-file=$GKEHUB_SA
)

?. Register the dc2-cluster

# dc2-cluster
(
export CLUSTER=dc2-cluster
export CLUSTER_URI=$(gcloud container clusters list --uri --filter="name=$CLUSTER")

gcloud container hub memberships register $CLUSTER --project=$PROJECT --gke-uri=$CLUSTER_URI --service-account-key-file=$GKEHUB_SA
)

?. Verify membership list

gcloud container hub memberships list

NAME         EXTERNAL_ID
dc2-cluster  1d977520-6ae6-11ea-9951-42010af0007c
dc1-cluster  a52d527d-6ae5-11ea-aa7b-42010a8e0fdb

Specify dc1-cluster as a nominated IfA Config cluster

"Ingress for Anthos runs as a service outside of the cluster and is managed by Google Cloud. The ingress controller watches for MultiClusterIngress and MultiClusterService resources in GKE and configures load balancers and NEGs as a result.

Specifying a config cluster enables Ingress for Anthos." [IfA Concepts, Architecture]

?. Enable Ingress for Anthos and select dc1-cluster as the config cluster

gcloud alpha container hub features multiclusteringress enable --config-membership=projects/$PROJECT/locations/global/memberships/dc1-cluster

Waiting for Feature to be created...done.
WARNING: generic::failed_precondition: WARNING: multiclusteringress feature will eventually require enablement of anthos.googleapis.com

?. Verify config cluster status

gcloud alpha container hub features multiclusteringress describe

featureState:
  details:
    code: OK
    description: Multicluster Ingress requires Anthos license enablement. Unlicensed
      usage is unrestricted for the MCI Beta API. Note that licensing will be enforced
      for use of the Generally Available MCI API.
  detailsByMembership:
    projects/755738281610/locations/global/memberships/dc1-cluster:
      code: OK
      description: CRDs are established
  lifecycleState: ENABLED
multiclusteringressFeatureSpec:
  configMembership: projects/emea-cs-hybrid-demo6/locations/global/memberships/dc1-cluster
name: projects/emea-cs-hybrid-demo6/locations/global/features/multiclusteringress
updateTime: '2020-03-24T21:22:55.881503252Z'
NOTE: While lifecycleState is marked as enabled, this feature is not ready to use until your config cluster has an OK status.

Ingress deployment

The Ingress provides a shared virtual IP (VIP) address for the app deployments.

To create the load balancer, you need two resources: a MultiClusterIngress and one or more MultiClusterServices. MultiClusterIngress and MultiClusterService objects are multi-cluster analogs to the existing Kubernetes Ingress and Service resources used in the single cluster context.

Creating a static IP

?. Create a static global IP

gcloud compute addresses create apigee-mci --global

Created [https://www.googleapis.com/compute/v1/projects/emea-cs-hybrid-demo6/global/addresses/apigee-mci].

?. Verify status of the IP address

gcloud compute addresses describe apigee-mci  --global

**address: 35.190.60.112**
addressType: EXTERNAL
creationTimestamp: '2020-03-26T05:16:29.168-07:00'
description: ''
id: '2585767625150768114'
ipVersion: IPV4
kind: compute#address
name: apigee-mci
networkTier: PREMIUM
selfLink: https://www.googleapis.com/compute/v1/projects/emea-cs-hybrid-demo6/global/addresses/apigee-mci
status: RESERVED

?. Copy TLS secret between namespaces

# TODO: Debug: 
# kubectl get secret emea-cs-hybrid-demo6-test-dc1.hybrid-apigee.net-ingressgateway-certs --namespace=istio-system -o yaml | yq w - data.namespace apigee-mci| yq d - metadata | kubectl apply -f -


# ns: apigee
kubectl -n apigee create secret tls emea-cs-hybrid-demo6-test-dc1.hybrid-apigee.net-ingressgateway-certs --key $RUNTIME_SSL_KEY --cert $RUNTIME_SSL_CERT

# ns: apigee-mci
kubectl -n apigee-mci create secret tls emea-cs-hybrid-demo6-test-dc1.hybrid-apigee.net-ingressgateway-certs --key $RUNTIME_SSL_KEY --cert $RUNTIME_SSL_CERT

MultiClusterSerivce

?. Deifine mcs manifest

cat <<EOT > apigee-mcs.yaml
apiVersion: networking.gke.io/v1beta1
kind: MultiClusterService
metadata:
  name: apigee-runtime-emea-cs-hybrid-demo6-test-zone-svc
  namespace: apigee
  annotations:
    networking.gke.io/app-protocols: '{"https-8443":"HTTPS"}'
    beta.cloud.google.com/backend-config: '{"ports": {"8443":"zone-health-check-cfg"}}'
spec:
  template:
    spec:
      selector:
        com.apigee.apigeedeployment: apigee-runtime-emea-cs-hybrid-demo6-test
      ports:
      - name: https-8443
        protocol: TCP
        port: 8443
        targetPort: 8443
  clusters:
  - link: "us-east1-b/dc1-cluster"
  - link: "asia-east1-b/dc2-cluster"
EOT

?. Apply manifest

kubectl apply -f apigee-mcs.yaml

?. Verify mcs The derived Service resource created by MCS are of Headless type. Therefore, None is an expected value of the ClusterIP property.

kubectl get svc

NAME                                                            TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)             AGE
...
mci-apigee-runtime-emea-cs-hybrid-demo6--svc-ybxkj03wo8mpnbtu   ClusterIP   None          <none>        8443/TCP            5d2h

BackendConfig object

?. Define Manifest

cat <<EOT > apigee-bec.yaml
apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
  name: zone-health-check-cfg
  namespace: apigee
spec:
  healthCheck:
    checkIntervalSec: 5
    timeoutSec: 5
    healthyThreshold: 2
    unhealthyThreshold: 3
    type: HTTPS
    port: 8443
    requestPath: /ping
EOT

?. Apply manifest

kubectl apply -f apigee-mcs.yaml

MultiClusterIngress

To be exposed by the same Ingress, these characteristics must be the same across clusters:

  • Be deployed into a namespace with the same name across clusters.
  • Share the same set of labels so they can be selected as a unit across the clusters.

?. Define MultiClusterService object

cat <<EOT > apigee-mci.yaml
apiVersion: networking.gke.io/v1beta1
kind: MultiClusterIngress
metadata:
  name: apigee-hybrid-mci
  namespace: apigee
  annotations:
    networking.gke.io/static-ip: 35.190.60.112
spec:
  template:
    spec:
      backend:
       serviceName: apigee-runtime-emea-cs-hybrid-demo6-test-zone-svc
       servicePort: 8443
      tls:
      - secretName: emea-cs-hybrid-demo6-test-dc1.hybrid-apigee.net-ingressgateway-certs
EOT

?. Verify MCI

kubectl -n apigee-mci describe mci apigee-hybrid-mci

Test requests originated from Different Regions

?. Execute curl request to /ping API

export RUNTIME_HOST_ALIAS=mc-demo6-test.hybrid-apigee.net
export RUNTIME_IP=35.190.60.112

curl --resolve "$RUNTIME_HOST_ALIAS:443:$RUNTIME_IP" https://$RUNTIME_HOST_ALIAS/ping -v

...
> GET /ping HTTP/1.1
> Host: mc-demo6.hybrid-apigee.net
> User-Agent: curl/7.52.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
< user-agent: curl/7.52.1
< accept: */*
...

?. Trace view of the request above

Derivative Objects Tour

It helps to be able to identify manifests and devived resources to investigate and diagnose potential problems.

?. MultiClusterService events

kubectl describe mcs apigee-runtime-emea-cs-hybrid-demo6-test-zone-svc

...
Events:
  Type    Reason  Age                    From                              Message
  ----    ------  ----                   ----                              -------
  Normal  SYNC    22m (x1330 over 5d2h)  multi-cluster-ingress-controller  Derived Service was ensured in cluster us-east1-b/dc1-cluster
  Normal  SYNC    57s (x1334 over 5d2h)  multi-cluster-ingress-controller  Derived Service was ensured in cluster asia-east1-b/dc2-cluster

?. MultiCluster Ingress Events and Status (I.e., references to Firewalls, Forwarding Rules, NEGs, etc.)

kubectl describe mci apigee-hybrid-mci


...
        Secret Name:  emea-cs-hybrid-demo6-test-dc1.hybrid-apigee.net-ingressgateway-certs
Status:
  Cloud Resources:
    Backend Services:
      mci-2dy6qj-8443--i6n413-apigee-apigee-runtime-emea
    Firewalls:
      mci-2dy6qj-default-l7
    Forwarding Rules:
      mci-2dy6qj-fw-apigee-apigee-hybrid-mci
      mci-2dy6qj-fws-apigee-apigee-hybrid-mci
    Health Checks:
      mci-2dy6qj-8443--i6n413-apigee-apigee-runtime-emea
    Network Endpoint Groups:
      zones/us-east1-b/networkEndpointGroups/k8s1-3b2d76ce-apig-mci-apigee-runtime-emea-cs-hybri-84-2bb37757
      zones/us-east1-c/networkEndpointGroups/k8s1-3b2d76ce-apig-mci-apigee-runtime-emea-cs-hybri-84-2bb37757
      zones/us-east1-d/networkEndpointGroups/k8s1-3b2d76ce-apig-mci-apigee-runtime-emea-cs-hybri-84-2bb37757
      zones/asia-east1-a/networkEndpointGroups/k8s1-7c3e1506-apig-mci-apigee-runtime-emea-cs-hybri-84-627b0d18
      zones/asia-east1-b/networkEndpointGroups/k8s1-7c3e1506-apig-mci-apigee-runtime-emea-cs-hybri-84-627b0d18
      zones/asia-east1-c/networkEndpointGroups/k8s1-7c3e1506-apig-mci-apigee-runtime-emea-cs-hybri-84-627b0d18
    Target Proxies:
      mci-2dy6qj-apigee-apigee-hybrid-mci
      mci-2dy6qj-apigee-apigee-hybrid-mci
    URL Map:  mci-2dy6qj-apigee-apigee-hybrid-mci
  VIP:        35.190.60.112
Events:       
...

?. Backend services

gcloud compute backend-services list

?. Network Endpoint Groups

gcloud compute network-endpoint-groups list

NOTES:

  • Have some patience. It takes minutes (5-7) for mcs/mci/bec objects to be created and become eventually consistent across regions. It happens... Eventually
  • Current BEC manifest uses /ping proxy as the healthcheck. This is a temporary hack until IaF will add support for TCP probes (end-of-April). Right now you can use any existing http/https probes.
  • As of now, your mcs/mci/bec object need to be located in the same namespace as the pods you're exposing.

REFERENCES:

Feedback

AHR Logo AHR-*-CTL

AHR: User Guide

AHR: SA Keys Management

Bastion Host

Scenarios

Deep Dives

Archive

Ingress and TLS

DevPortal

Troubleshooting Flows

Clone this wiki locally