Skip to content
Scripts and other tools for installing / operating Fusion on Kubernetes
Shell
Branch: master
Clone or download
dustinguericke Merge pull request #8 from lucidworks/DOCS-1880-eks-docs
Made some minor changes to the EKS install information. (DOCS-1880)
Latest commit 237e05c Oct 11, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
README.adoc
customize_fusion_values.yaml.example Add values.yaml Sep 24, 2019
setup_f5_aks.sh updated for Fusion 5.0.1 Oct 11, 2019
setup_f5_eks.sh eks 5.0.1 Oct 10, 2019
setup_f5_gke.sh bump version 5.0.1 Oct 10, 2019

README.adoc

Fusion Cloud Native on Kubernetes

This repo contains scripts for installing Fusion 5.x on Kubernetes (K8s). The scripts provide an option to create Kubernetes clusters that are suitable for demo / proof-of-concept purposes only. We assume that you’ll want to control how your production clusters are provisioned, secured, and managed, as these are typically concerns we’re not able to script for you.

Prerequisites

This section covers prerequisites and background knowledge needed to help you understand the structure of this document and how the Fusion installation process works with Kubernetes.

Release Name and Namespace

Before installing Fusion, you need to choose a unique release name for Fusion, such as f5; Helm uses the release name to track a specific installation of an application in the cluster.

You also need to choose the Kubernetes namespace to install Fusion into. Think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster in separate namespaces. However, please do not install more than one Fusion release in the same namespace.

Install Helm

Helm is a package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster. Regardless of which Kubernetes platform you’re using, e.g. GKE, EKS, OpenShift, you need to install Helm as it is required to install Fusion for any K8s platform. On MacOS, you can do:

brew install kubernetes-helm

For other OS, please refer to the Helm installation docs: https://helm.sh/docs/using_helm/

The fusion helm chart requires that helm is greater than version 2.12.0. Please ensure that your version is greater than this, you can check by running helm version.

Customize Fusion Chart Settings

Fusion aims to be well-configured out-of-the-box, but you can customize any of the built-in settings using a custom values yaml file.

If you use one of our setup scripts, such as setup_f5_gke.sh, then it will create this file for you the first time you run it. Keep this file handy as you’ll need it to customize Fusion settings and upgrade to a newer version.

If you’re working with Helm directly and not using one of our setup scripts, then simply copy the example file provided in this repo:

cp customize_fusion_values.yaml.example <release>_<namespace>_fusion_values.yaml

where <release> is the name you give to your Fusion release, such as f5 and <namespace> is the Kubernetes namespace where you want to install Fusion, such as default.

Google Kubernetes Engine (GKE)

The setup_f5_gke.sh script provided in this repo is strictly optional. The script is mainly to help those new to Kubernetes and/or Fusion get started quickly. If you’re already familiar with K8s, Helm, and GKE, then you can skip the script and just use Helm directly to install Fusion into an existing cluster or one you create yourself using the process described here.

If you’re new to Google Cloud Platform (GCP), then you need an account on Google Cloud Platform before you can begin deploying Fusion on GKE.

Set up the Google Cloud SDK (one time only)

If you’ve already installed the gcloud command-line tools, you can skip to Create a Fusion cluster in GKE.

These steps set up your local Google Cloud SDK environment so that you’re ready to use the command-line tools to manage your Fusion deployment.

Usually, you only need to perform these setup steps once per local session. After that, you’re ready to create a cluster.

How to set up the Google Cloud SDK
  1. Enable the Kubernetes Engine API.

  2. Log in to Google Cloud: gcloud auth login

  3. Set up the Google Cloud SDK:

    1. gcloud config set compute/zone <zone-name>

      If you are working with regional clusters instead of zone clusters, use gcloud config set compute/region <region-name> instead.

    2. gcloud config set core/account <email address>

    3. New GKE projects only: gcloud projects create <new-project-name>

      If you have already created a project, for example in the Google Cloud Platform console, then skip to the next step.

    4. gcloud config set project <project-name>

Make sure you install the Kubernetes command-line tool kubectl using:

gcloud components install kubectl
gcloud components update

Set up Fusion on GKE

Download and run the setup_f5_gke.sh script to install Fusion 5.x in a GKE cluster. To create a new cluster and install Fusion, simply do:

./setup_f5_gke.sh -c <cluster_name> -p <gcp_project_id> -r <release> -n <namespace>

Use the --help option to see script usage. If you want the script to create a cluster for you (the default behavior), then you need to pass the --create option with either demo or multi_az. If you don’t want the script to create a cluster, then you need to create a cluster before running the script and simply pass the name of the existing cluster using the -c parameter.

If you pass --create demo to the script, then we create a single node GKE cluster. The minimum node type you’ll need for a 1 node cluster is an n1-standard-4 (on GKE) which has 4 CPU and 15 GB of memory. This is cutting it very close in terms of resources as you also need to host all of the Kubernetes system pods on this same node. Obviously, this works for kicking the tires on Fusion 5.0 but is not sufficient for production workloads.

Note: If not provided the script generates a custom values file named <release>_<namespace>_fusion_values.yaml which you can use to customize the Fusion chart.

WARNING The setup_f5_gke.sh script installs Helm’s tiller component into your GKE cluster with the cluster admin role. If you don’t want this, then please see Helm w/o Tiller below.

After running the setup_f5_gke.sh script, proceed to the Verifying the Fusion Installation section below.

The steps below show you how to create several kinds of Fusion clusters.

How to create a single-node Fusion demo cluster

A single-node configuration is useful for exploring Fusion in a demo or development environment.

This type of deployment can take at least 12 minutes, plus 3–5 minutes for cluster startup.

How to create a single-node Fusion demo cluster
  1. Run the setup script:

    ./setup_f5_gke.sh -c <cluster> -p <project> -z <zone-name> --create demo
    • <cluster> value should be the name of a non-existent cluster; the script will create the new cluster.

    • <project> must match the name of an existing project in GKE.

      Run gcloud config get-value project to get this value, or see the GKE setup instructions.

    • <zone-name> must match the name of the zone you set in GKE.

      Run gcloud config get-value compute/zone to get this value, or see the GKE setup instructions to set the value.

    Upon success, the script shows you where to find the Fusion UI. For example:

    Fusion 5 Gateway service exposed at: <some-external-ip>:6764
  2. Access the Fusion UI by pointing your browser to the IP address and port specified in the setup script’s output.

Create a three-node regional cluster to withstand a zone outage

With a three-node regional cluster, nodes are deployed across three separate availability zones.

./setup_f5_gke.sh -c <cluster> -p <project> -z <zone-name> --create multi_az

In this configuration, we want a ZooKeeper and Solr instance on each node, which allows the cluster to retain ZK quorum and remain operational after losing one node, such as during an outage in one availability zone.

When running in a multi-zone cluster, each Solr node has the solr_zone system property set to the zone it is running in, such as -Dsolr_zone=us-west1-a.

GKE Ingress and TLS

The Fusion proxy service provides authentication and serves as an API gateway for accessing all other Fusion services. It’s typical to use an Ingress for TLS termination in front of the proxy service.

The setup_f5_gke.sh supports creating an Ingress with an TLS cert for a domain you own by passing: -t -h <domain_name>

After the script runs, you need to create an A record in GCP’s DNS service to map your domain name to the Ingress IP. Once this occurs, our script setup uses Let’s Encrypt to issue a TLS cert for your Ingress.

Please refer to the Kubernetes documentation on configuring an Ingress for GKE: Setting up HTTP Load Balancing with Ingress

Upgrade Fusion on GKE

During installation, the script generates a file named <release>_<namespace>_fusion_values.yaml; use this file to customize Fusion settings. After making changes to this file, you need to run the following command:

./setup_f5_gke.sh -c <existing_cluster> -p <gcp_project_id> -r <release> -n <namespace> --values <release>_<namespace>_fusion_values.yaml --upgrade

You will also use the --upgrade option to upgrade to a newer version of Fusion, such as 5.0.1.

Amazon Elastic Kubernetes Service (EKS)

The setup_f5_eks.sh script provided in this repo is strictly optional. The script is mainly to help those new to Kubernetes and/or Fusion get started quickly. If you’re already familiar with K8s, Helm, and EKS, then you use Helm directly to install Fusion into an existing cluster or one you create yourself using the process described here.

If you’re new to Amazon Web Services (AWS), then please visit the Amazon Web Services Getting Started Center to set up an account.

If you’re new to Kubernetes and EKS, then we recommend going through Amazon’s EKS Workshop before proceeding with Fusion.

Set up the AWS CLI tools

Before launching an EKS cluster, you need to install and configure kubectl, aws, eksctl, aws-iam-authenticator using the links provided below:

Required AWS Command-line Tools:
  1. kubectl: Install kubectl

  2. aws: Installing the AWS CLI

  3. eksctl: Getting Started with eksctl

  4. aws-iam-authenticator: AWS IAM Authenticator for Kubernetes

Run aws configure to configure a profile for authenticating to AWS. You’ll use the profile name you configure in this step, which defaults to default, as the -p argument to the setup_f5_eks.sh script in the next section.

Note
When working in Ubuntu, avoid using the eksctl snap version. Alternative sources can have different versions that could cause command failures.

Set up Fusion on EKS

Download and run the setup_f5_eks.sh script to install Fusion 5.x in a EKS cluster. To create a new cluster and install Fusion, simply do:

./setup_f5_eks.sh -c <cluster_name> -p <aws_account_profile>

If you want the script to create a cluster for you (the default behavior), then you need to pass the --create option with either demo or multi_az. If you don’t want the script to create a cluster, then you need to create a cluster before running the script and simply pass the name of the existing cluster using the -c parameter.

Use the --help option to see full script usage.

WARNING The setup_f5_eks.sh script installs Helm’s tiller component into your EKS cluster with the cluster admin role. If you don’t want this, then please see Helm w/o Tiller below.

WARNING The setup_f5_eks.sh script creates a service account that provides S3 read-only permissions to the created pods.

After running the setup_f5_eks.sh script, proceed to the Verifying the Fusion Installation section below.

EKS cluster overview

The EKS cluster is created using eksctl (https://eksctl.io/). By default it will setup the following resources in your AWS account:

  • A dedicated VPC for the EKS cluster in the specified region with CIDR: 192.168.0.0/16

  • 3 Public and 3 Private subnets within the created VPC, each with a /19 CIDR range, along with the corresponding route tables.

  • A NAT gateway in each Public subnet

  • An Auto Scaling Group of the instance type specified by the script, which defaults to m5.2xlarge, with 3 instances spanning the public subnets.

See https://eksctl.io/usage/vpc-networking/ for more information on the networking setup.

EKS Ingress

The setup_f5_eks.sh script exposes the Fusion proxy service on an external IP over HTTP. This is done for demo or getting started purposes. However, you’re strongly encouraged to configure a K8s Ingress with TLS termination in front of the proxy service. See: https://aws.amazon.com/premiumsupport/knowledge-center/terminate-https-traffic-eks-acm/

Upgrade Fusion on EKS

During installation, the script generates a file named <release>_<namespace>_fusion_values.yaml. Use this file to customize Fusion settings. After making changes to this file, run the following command:

./setup_f5_eks.sh -c <existing_cluster> -p <aws_account_profile> -r <release> -n <namespace> --values <release>_<namespace>_fusion_values.yaml --upgrade

You will also use the --upgrade option to upgrade to a newer version of Fusion, such as 5.0.1.

Provide access to the EKS cluster to other users

Initially, only the user that created the Amazon EKS cluster has system:masters permissions to configure the cluster. In order to extend the permissions, a ConfigMap should be created to allow access to IAM users or roles.

For providing these permissions, use the following yaml file as a template, replacing the required values:

aws-auth.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: <node_instance_role_arn>
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes
  mapUsers: |
    - userarn: arn:aws:iam::<account_id>:user/<username>
      username: <username>
      groups:
        - system:masters

Use the following command for applying the yaml file: kubectl apply -f aws-auth.yaml

Other Kubernetes Platforms

If you’re not running on managed K8s platform like GKE or EKS, you can use Helm to install the Fusion chart to an existing Kubernetes cluster.

Use Helm to Install Fusion

Start by copying the example custom values yaml file provided in this repo:

cp customize_fusion_values.yaml.example <release>_<namespace>_fusion_values.yaml

where <release> is the name you give to your Fusion release, such as f5 and <namespace> is the Kubernetes namespace where you want to install Fusion, such as default.

Review the settings in the custom values yaml file to ensure the defaults are appropriate for your environment, such as the number of Solr and Zookeeper replicas. After making changes to the custom values yaml file, install Fusion using the following commands:

RELEASE=f5
NAMESPACE=default

helm repo add lucidworks https://charts.lucidworks.com
helm repo update
helm install --timeout 240 --namespace "${NAMESPACE}" -n "${RELEASE}" lucidworks/fusion --values "${RELEASE}_${NAMESPACE}_fusion_values.yaml"
kubectl rollout status deployment/${RELEASE}-api-gateway --timeout=600s --namespace "${NAMESPACE}"

For more information, please see the Fusion documentation: https://doc.lucidworks.com/fusion-server/5.0/deployment/kubernetes/index.html

Don’t want to use Tiller? (Helm v2)

Tiller, the server-side component of Helm, has known security concerns. If your K8s administrators do not allow the use of Tiller for installing Helm charts, then you can use Helm 3 (currently in beta) or simply do:

RELEASE=f5
NAMESPACE=default

helm repo add lucidworks https://charts.lucidworks.com
helm repo update
helm fetch --untar lucidworks/fusion
helm template -n "${RELEASE}" --namespace "${NAMESPACE}" --values "${RELEASE}_${NAMESPACE}_fusion_values.yaml" fusion > ${RELEASE}_${NAMESPACE}_fusion_install.yaml
kubectl apply -f "${RELEASE}_${NAMESPACE}_fusion_install.yaml" --namespace "${NAMESPACE}"

Alternatively, you can run tiller locally using the steps described here: https://docs.aws.amazon.com/eks/latest/userguide/helm.html

Helm v3 beta3 instructions without Tiller

HELM3_HOME=/usr/local/helm3
RELEASE=f5
NAMESPACE=default

${HELM3_HOME}/helm repo add lucidworks https://charts.lucidworks.com
${HELM3_HOME}/helm repo update
${HELM3_HOME}/helm install "${RELEASE}" --namespace "${NAMESPACE}" --values "${RELEASE}_${NAMESPACE}_fusion_values.yaml" lucidworks/fusion
kubectl rollout status deployment/${RELEASE}-api-gateway --timeout=600s --namespace "${NAMESPACE}"

Upgrade Existing Installation

To update an existing installation, do:

helm repo update
helm upgrade ${RELEASE} "lucidworks/fusion" --timeout 180 --namespace "${NAMESPACE}" --values "${RELEASE}_${NAMESPACE}_fusion_values.yaml"

Except for Zookeeper, all K8s deployments and statefulsets use a RollingUpdate update policy, e.g.:

  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate

Zookeepers use OnDelete to avoid changing critical stateful pods in the Fusion deployment. Thus, after performing the upgrade, to get changes to Zookeeper to apply, you need to manually delete the pods, e.g.

kubectl delete po f5-zookeeper-0

Do this one-by-one for each pod and verify the new pod is healthy and serving traffic before deleting the next healthy pod.

Alternatively, you can set the updateStrategy under the zookeeper section in your ${RELEASE}_${NAMESPACE}_fusion_values.yaml file:

solr:
  ...
  zookeeper:
    updateStrategy:
      type: "RollingUpdate"

Verifying the Fusion Installation

In this section, we provide some tips on how to verify the Fusion installation. First, let’s review some useful kubectl commands.

Useful kubectl commands

When working with Kubernetes on the command-line, it’s useful to create a shell alias for kubectl, e.g.:

alias k=kubectl

Set the namespace for kubectl if not using the default:

kubectl config set-context --current --namespace=<NAMESPACE>

This saves you from having to pass -n with every command.

Get a list of running pods: k get pods

Get logs for a pod using a label: k logs –l app.kubernetes.io/component=query-pipeline

Get pod deployment spec and details: k get pods <pod_id> -o yaml

Get details about a pod events: k describe po <pod_id>

Port forward to a specific pod: k port-forward <pod_id> 8983:8983

SSH into a pod: k exec -it <pod_id> — /bin/bash

CPU/Memory usage report for pods: k top pods

Forcefully kill a pod: k delete po <pod_id> --force --grace-period 0

Scale up (or down) a deployment: k scale deployment.v1.apps/<id> --replicas=N

Check Fusion Pods and Services

Once the install script completes, you can check that all pods and services are available using:

kubectl get pods

If all goes well, you should see a list of pods similar to:

NAME                                     READY   STATUS    RESTARTS   AGE
f5-admin-ui-564b7d7d4b-6ksdq             1/1     Running   0          11m
f5-api-gateway-7b497bbbdc-vtghx          1/1     Running   0          11m
f5-auth-ui-7cbc457db7-hrqmz              1/1     Running   0          11m
f5-classic-rest-service-0                1/1     Running   1          11m
f5-cx-api-0                              1/1     Running   0          11m
f5-cx-scheduler-64cc788dfd-brszp         1/1     Running   0          11m
f5-cx-script-executor-587bb5959c-qxl47   1/1     Running   0          11m
f5-cx-ui-7dbd9fb449-q4lkp                1/1     Running   0          11m
f5-devops-ui-7cd6d66f98-n8lp4            1/1     Running   0          11m
f5-fusion-admin-64b8578944-94t2c         1/1     Running   1          11m
f5-fusion-indexing-cb5c7b449-bgtbv       1/1     Running   1          11m
f5-insights-7dd4cfc5b8-m8c5d             1/1     Running   0          11m
f5-job-launcher-7ffc8cc999-lrt7c         1/1     Running   1          11m
f5-job-rest-server-85c46c4bf7-dgkc2      1/1     Running   1          11m
f5-kafka-0                               1/1     Running   1          11m
f5-ml-model-service-78cb897f8d-stq58     2/2     Running   2          11m
f5-query-pipeline-889f554d9-9pkkm        1/1     Running   0          11m
f5-rest-service-6cbb798d7f-6h7gd         1/1     Running   0          11m
f5-rpc-service-767999dc87-pjqrw          1/1     Running   0          11m
f5-rules-ui-5d565b9c85-fzm6f             1/1     Running   0          11m
f5-solr-0                                1/1     Running   0          11m
f5-sql-service-cm-5949c89f4c-k9m7v       1/1     Running   0          11m
f5-sql-service-cr-6c77759498-ztrnt       1/1     Running   0          11m
f5-webapps-5c4cbb4576-fv7x2              1/1     Running   2          11m
f5-zookeeper-0                           1/1     Running   0          11m

The number of pods per deployment / statefulset will vary based on your cluster size and replicaCount settings in <release>_<namespace>_fusion_values.yaml. Also, don’t worry if you see some pods having been restarted as that just means they were too slow to come up and Kubernetes killed and restarted them. You do want to see at least one pod running for every service. If a pod is not running after waiting a sufficient amount of time, use kubectl logs <pod_id> to see the logs for that pod; to see the logs for previous versions of a pod, use: kubectl logs <pod_id> -p. You can also look at the actions Kubernetes performed on the pod using kubectl describe po <pod_id>.

To see a list of Fusion services, do:

kubectl get svc

For an overview of the various Fusion 5 microservices, see: https://doc.lucidworks.com/fusion-server/5.0/deployment/kubernetes/microservices.html

You can’t perform that action at this time.