Skip to content
Switch branches/tags

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time


Kubernetes cluster configuration that uses GitOps to keep state.

Includes Flux, Helm, cert-manager, Nginx Ingress Controller and Sealed Secrets.



  • 2020-02-13 Flux 1.1, annotations, and Helm 3
  • 2019-11-07 Flux 0.16, annotations and Helm 2

Pre requisite

Kubernetes tools and cluster

  • brew install kubectl
  • Create Kubernetes cluster (see Kubernetes as a Service providers below)

  • Set up Kubernetes context (see provider CLIs and kubectx below)

  • Test cluster connection:

    kubectl cluster-info

Lemmings install

Fork/Clone repository

  • brew install hub;
    hub clone flurdy/lemmings my-lemmings;
    cd my-lemmings;
    hub create -p my-lemmings;
    hub remote add upstream flurdy/lemmings;
    git push -u origin master
  • Replace my-lemmings with whatever you want to call your cluster

  • Flux can also talk to Bitbucket, Gitlab and self-hosted repos

Install Helm

  • Helm released v3 in November 2019.
  • If you have other clusters using Helm 2 tread carefully to make sure you can support both v2 and v3.

Helm 3

  brew install helm@3
  • Vanilla Helm 3 includes no chart repositories. Lets add the core one

    helm repo add stable \

Helm 2

  • If you need to use Helm 2, refer to an earlier version of Lemmings more aimed at Helm 2 + Tiller
  • Note you may need to specify specific annotations and HelmRelease versions to keep it working as things start to assume Helm 3

Configure cert-manager

  • Change email address in:
    • issuer/issuer-staging.yml
    • issuer/issuer-prod.yml
  • git add -p issuer;
    git commit -m "Updated email in certificate issuers";
    git push

Install Flux

helm repo add fluxcd;
helm repo update;
kubectl apply -f flux/crd-flux-helm.yml;
kubectl create namespace flux;
helm upgrade -i flux \
   --namespace=flux \
   --set helm.versions=v3 \
   --set \
  • Replace YOURUSERNAME/me-lemmings with your username and repository.

Install Flux Helm Operator

helm upgrade -i helm-operator \
   --namespace=flux \
   --set git.ssh.secretName=flux-git-deploy \
   --set helm.versions=v3 \

Fluxctl & SSH

  • Install fluxctl and generate SSH key

    brew install fluxctl;
    fluxctl identity --k8s-fwd-ns flux
  • Add Flux's SSH key to your github repo with write access:

  • Tail log:

    kubectl logs -n flux deploy/flux -f

Your GitOps based Kubernetes cluster is live!

If you waited a few minutes then your GitOps configured Kubernetes cluster is now live.

Your first application

  • (We baked one earlier for you: workflows/hello)

  • View/Edit workflows/hello/deployment.yml

    apiVersion: apps/v1
    kind: Deployment
      name: hello-deployment
      annotations: "true"
          app: hello
      replicas: 2
            app: hello
          - name: hello-container
            image: nginxdemos/hello:0.2
            - containerPort: 80
    • Note the annotation which tells Flux to upgrade it when possible.
  • Edit workflows/hello/service.yml

    apiVersion: v1
    kind: Service
      name: hello-service
         app: hello
      - protocol: TCP
         port: 80
         targetPort: 80
  • Edit workflows/hello/ingress.yml

    apiVersion: extensions/v1beta1
    kind: Ingress
      name: hello-ingress
      annotations: nginx
    # letsencrypt-staging
    #  tls:
    #  - hosts:
    #    -
    #    secretName: letsencrypt-certificate-staging
      - host:
          - backend:
              serviceName: hello-service
              servicePort: 80

    The SSL certificate part is commented out for now as it wont work without a real domain that Letsencrypt can do a reverse verification call to.

Launch application

  • If you made any changes:

    git add workflows/hello
    git commit -m "App: Hello"
    git push
  • Wait (max 5 minutes) for Flux to detect your changes and apply them

  • Or if impatient:

    fluxctl sync
  • Check if the Hello cluster resources are running

    kubectl get deployment hello-deployment
    kubectl get service hello-service
    kubectl get ingress hello-ingress
    kubectl get pods -l app=hello

Test application

  • Find the ingress controller's External IP

    kubectl get services nginx-ingress-controller
  • Use curl to resolve the URL. Replace with the external IP.

  • And lynx to view it

    curl -H "Host:" \
      --resolve \
      --resolve \ | lynx -stdin
  • This should show a basic hello world page, with an Nginx logo and some server address, name and date details.

Update application

  • Now if ever bumps its version to higher than 0.2, Flux will detect it and bump your version in Kubernetes, and also commit the change back to your Git repository.
  • Unfortunately that image has not been updated for 2 years and unlikely to be updated, but when you start to use your own images they will be automagically updated if the deployment's annotation is set to true.

Delete application

git rm -r workflows/hello
git commit -m "Removed app Hello"
git push
fluxctl sync
kube delete ingress hello-ingress
kube delete service hello-service
kube delete deployment hello-deployment

Sealed Secrets

  • If you need secrets, (for private docker registry authentication, database passwords, API tokens), DO NOT commit them in plain text to the Git repository.

  • Instead we will use Sealed Secrets to encrypt them.

  • Check if Sealed Secrets was installed by Flux:

    kube get services sealed-secrets -n kube-system
  • Install CLI

    brew install kubeseal
  • Fetch the cluster's Sealed Secrets' public key

    kubeseal --fetch-cert \
       --controller-namespace=flux \
       --controller-name=sealed-secrets \
       > sealed-secrets/sealed-secrets-cert.pem
    git add sealed-secrets/sealed-secrets-cert.pem
    git commit -m "Sealed secret public key"
  • Create secrets but do not apply them to the cluster.

    • E.g with the --dry-run argument.

      mkdir secrets;
      kubectl create secret generic basic-auth \
         --from-literal=user=admin \
         --from-literal=password=admin \
         --dry-run \
         -o json > secrets/basic-auth.json
  • Encrypt secret and transform to a sealed secret:

    kubeseal --format=yaml \
       --cert=sealed-secrets/sealed-secrets-cert.pem \
       < secrets/basic-auth.json \
       > secrets/secret-basic-auth.yml
    git add secrets/secret-basic-auth.yml
    rm secrets/basic-auth.json
    git commit -m "Sealed basic auth secret"
    git push
  • After a little time you should see the secret in your cluster

    kubectl describe secret basic-auth
  • Afterwards if you no longer need it, delete the file and secret:

    git rm secrets/secret-basic-auth.yml
    git commit -m "Removed basic auth"
    git push
    fluxctl sync
    kubectl delete secret basic-auth
    kubectl delete SealedSecret basic-auth
  • Note: Step by step Docker Registry Cookbook

Go wild

  • Add/update your deployments, services, charts, docker registries, secrets, etc

Don't touch

  • Once Flux is running, by convention avoid using kubectl create|apply etc.
  • Nearly all changes should be via Git and Flux.
  • Any kubectl interaction should be read only.
  • Flux will not remove some resources, so you might need kubectl delete occasionally.

More information, alternatives, suggestions


  • Certain operation takes a few minutes, e.g. pod creation.
  • cert-manager 0.11+ requires kubernetes 1.15+
  • Client tools are also available on Linux, Windows and more.


Kubernetes cluster configuration that uses GitOps to keep state.








No packages published