Skip to content
master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 

Lemmings

Kubernetes cluster configuration that uses GitOps to keep state.

Includes Flux, Helm, cert-manager, Nginx Ingress Controller and Sealed Secrets.

Contributors

Versions

  • 2020-02-13 Flux 1.1, fluxcd.io annotations, and Helm 3
  • 2019-11-07 Flux 0.16, flux.weave.works annotations and Helm 2

Pre requisite

Kubernetes tools and cluster

  • brew install kubectl
    
  • Create Kubernetes cluster (see Kubernetes as a Service providers below)

  • Set up Kubernetes context (see provider CLIs and kubectx below)

  • Test cluster connection:

    kubectl cluster-info
    

Lemmings install

Fork/Clone repository

  • brew install hub;
    hub clone flurdy/lemmings my-lemmings;
    cd my-lemmings;
    hub create -p my-lemmings;
    hub remote add upstream flurdy/lemmings;
    git push -u origin master
    
  • Replace my-lemmings with whatever you want to call your cluster

  • Flux can also talk to Bitbucket, Gitlab and self-hosted repos

Install Helm

  • Helm released v3 in November 2019.
  • If you have other clusters using Helm 2 tread carefully to make sure you can support both v2 and v3.

Helm 3

  brew install helm@3
  • Vanilla Helm 3 includes no chart repositories. Lets add the core one

    helm repo add stable \
        https:// kubernetes-charts.storage.googleapis.com/
    

Helm 2

  • If you need to use Helm 2, refer to an earlier version of Lemmings more aimed at Helm 2 + Tiller
  • Note you may need to specify specific flux.weave.works annotations and HelmRelease versions to keep it working as things start to assume Helm 3

Configure cert-manager

  • Change email address in:
    • issuer/issuer-staging.yml
    • issuer/issuer-prod.yml
  • git add -p issuer;
    git commit -m "Updated email in certificate issuers";
    git push
    

Install Flux

helm repo add fluxcd https://charts.fluxcd.io;
helm repo update;
kubectl apply -f flux/crd-flux-helm.yml;
kubectl create namespace flux;
helm upgrade -i flux \
   --namespace=flux \
   --set helm.versions=v3 \
   --set git.url=git@github.com:YOURUSERNAME/my-lemmings \
   fluxcd/flux
  • Replace YOURUSERNAME/me-lemmings with your username and repository.

Install Flux Helm Operator

helm upgrade -i helm-operator \
   --namespace=flux \
   --set git.ssh.secretName=flux-git-deploy \
   --set helm.versions=v3 \
   fluxcd/helm-operator

Fluxctl & SSH

  • Install fluxctl and generate SSH key

    brew install fluxctl;
    export FLUX_FORWARD_NAMESPACE=flux;
    fluxctl identity --k8s-fwd-ns flux
    
  • Add Flux's SSH key to your github repo with write access:

  • Tail log:

    kubectl logs -n flux deploy/flux -f
    

Your GitOps based Kubernetes cluster is live!

If you waited a few minutes then your GitOps configured Kubernetes cluster is now live.

Your first application

  • (We baked one earlier for you: workflows/hello)

  • View/Edit workflows/hello/deployment.yml

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: hello-deployment
      annotations:
        fluxcd.io/automated: "true"
    spec:
      selector:
        matchLabels:
          app: hello
      replicas: 2
      template:
        metadata:
          labels:
            app: hello
        spec:
          containers:
          - name: hello-container
            image: nginxdemos/hello:0.2
            ports:
            - containerPort: 80
    
    • Note the fluxcd.io/automated annotation which tells Flux to upgrade it when possible.
  • Edit workflows/hello/service.yml

    apiVersion: v1
    kind: Service
    metadata:
      name: hello-service
    spec:
      selector:
         app: hello
      ports:
      - protocol: TCP
         port: 80
         targetPort: 80
    
  • Edit workflows/hello/ingress.yml

    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: hello-ingress
      annotations:
        kubernetes.io/ingress.class: nginx
    #    cert-manager.io/cluster-issuer: letsencrypt-staging
    spec:
    #  tls:
    #  - hosts:
    #    - hello.example.com
    #    secretName: letsencrypt-certificate-staging
      rules:
      - host: hello.example.com
        http:
          paths:
          - backend:
              serviceName: hello-service
              servicePort: 80
    

    The SSL certificate part is commented out for now as it wont work without a real domain that Letsencrypt can do a reverse verification call to.

Launch application

  • If you made any changes:

    git add workflows/hello
    git commit -m "App: Hello"
    git push
    
  • Wait (max 5 minutes) for Flux to detect your changes and apply them

  • Or if impatient:

    fluxctl sync
    
  • Check if the Hello cluster resources are running

    kubectl get deployment hello-deployment
    kubectl get service hello-service
    kubectl get ingress hello-ingress
    kubectl get pods -l app=hello
    

Test application

  • Find the ingress controller's External IP

    kubectl get services nginx-ingress-controller
    
  • Use curl to resolve the URL. Replace 11.22.33.44 with the external IP.

  • And lynx to view it

    curl -H "Host: hello.example.com" \
      --resolve hello.example.com:80:11.22.33.44 \
      --resolve hello.example.com:443:11.22.33.44 \
      http://hello.example.com | lynx -stdin
    
  • This should show a basic hello world page, with an Nginx logo and some server address, name and date details.

Update application

  • Now if ever hub.docker.com/r/nginxdemos/hello bumps its version to higher than 0.2, Flux will detect it and bump your version in Kubernetes, and also commit the change back to your Git repository.
  • Unfortunately that image has not been updated for 2 years and unlikely to be updated, but when you start to use your own images they will be automagically updated if the deployment's annotation fluxcd.io/automated is set to true.

Delete application

git rm -r workflows/hello
git commit -m "Removed app Hello"
git push
fluxctl sync
kube delete ingress hello-ingress
kube delete service hello-service
kube delete deployment hello-deployment

Sealed Secrets

  • If you need secrets, (for private docker registry authentication, database passwords, API tokens), DO NOT commit them in plain text to the Git repository.

  • Instead we will use Sealed Secrets to encrypt them.

  • Check if Sealed Secrets was installed by Flux:

    kube get services sealed-secrets -n kube-system
    
  • Install CLI

    brew install kubeseal
    
  • Fetch the cluster's Sealed Secrets' public key

    kubeseal --fetch-cert \
       --controller-namespace=flux \
       --controller-name=sealed-secrets \
       > sealed-secrets/sealed-secrets-cert.pem
    git add sealed-secrets/sealed-secrets-cert.pem
    git commit -m "Sealed secret public key"
    
  • Create secrets but do not apply them to the cluster.

    • E.g with the --dry-run argument.

      mkdir secrets;
      kubectl create secret generic basic-auth \
         --from-literal=user=admin \
         --from-literal=password=admin \
         --dry-run \
         -o json > secrets/basic-auth.json
      
  • Encrypt secret and transform to a sealed secret:

    kubeseal --format=yaml \
       --cert=sealed-secrets/sealed-secrets-cert.pem \
       < secrets/basic-auth.json \
       > secrets/secret-basic-auth.yml
    git add secrets/secret-basic-auth.yml
    rm secrets/basic-auth.json
    git commit -m "Sealed basic auth secret"
    git push
    
  • After a little time you should see the secret in your cluster

    kubectl describe secret basic-auth
    
  • Afterwards if you no longer need it, delete the file and secret:

    git rm secrets/secret-basic-auth.yml
    git commit -m "Removed basic auth"
    git push
    fluxctl sync
    kubectl delete secret basic-auth
    kubectl delete SealedSecret basic-auth
    
  • Note: Step by step Docker Registry Cookbook

Go wild

  • Add/update your deployments, services, charts, docker registries, secrets, etc

Don't touch

  • Once Flux is running, by convention avoid using kubectl create|apply etc.
  • Nearly all changes should be via Git and Flux.
  • Any kubectl interaction should be read only.
  • Flux will not remove some resources, so you might need kubectl delete occasionally.

More information, alternatives, suggestions

Notes:

  • Certain operation takes a few minutes, e.g. pod creation.
  • cert-manager 0.11+ requires kubernetes 1.15+
  • Client tools are also available on Linux, Windows and more.

About

Kubernetes cluster configuration that uses GitOps to keep state.

Topics

Resources

License

Stars

Watchers

Forks

Packages

No packages published