Skip to content
$ kubectl hacking - my journey with kate
HCL Shell
Branch: master
Clone or download
Latest commit 8027117 Jun 12, 2019
Type Name Latest commit message Commit time
Failed to load latest commit information.
fzf update Jun 5, 2019
kubeone-cluster add kubeone config Jun 11, 2019
pics add first draft of kubectl hacking slides Jun 5, 2019
powerline-go add first draft of kubectl hacking slides Jun 5, 2019
.gitignore add first draft of kubectl hacking slides Jun 5, 2019 update slides Jun 12, 2019 update Jun 5, 2019

title author patat
$ kubectl hacking - my journey with kate
Tobias Schneck, Loodse GmbH
wrap pandocExtensions margins theme
left right
syntaxHighlighting code codeBlock

My journey to Kubernetes

Java programmer -> Testautomation -> Docker -> OpenShift -> Kubernetes

                   ...  ..++..++..++..  ...   
                    ..+++..  .++.  ..+++..    
                     .+.++.. .++. ..++.+.     
                    .+.  .++++++++++.  .+.    
                    ++.  ..++....++..  .++    
                .....++..  .++++++.  ..++.....
                     .++.  ++.  .++  .++.     
                      ..+..+.    .+..+..      
                        .+.        .+.        
                        +.          .+                          




kubectl - what's this about?

basic bash config

  • Enable bash auto completion (update compatible)

    # add to ~/.bashrc
    source <(kubectl completion bash)
    alias k=kubectl
    complete -F __start_kubectl k
  • beautify your bash prompt -> powerline go

    1. Install binary go get -v -u
    2. ensure powerline-go is available in the path: export PATH=$PATH:$GOPATH/bin
    3. config ~/.bashrc -> source ./powerline-go/.bashrc

kubectl help?

  • kubectl [command] --help is very helpful!
  • kubectl explain [object][.field][...] information about spec fields!
  • kubectl api-resources shows available objects - with CRDs!


kubectl get --help
#options for all commands
kubectl options

kubectl explain pod.spec.containers.ports
kubectl explain svc.spec.type

kubectl api-resources --api-group=apps
kubectl api-resources -o wide


  • Default configuration file: ~/.kube/config

    • endpoints
    • SSL keys
    • contexts
  • $KUBECONFIG environment variable or --kubeconfig flag

    • useful to manage multiple cluster
    • merge multiple config files
    KUBECONFIG=conf1:conf2 kubectl config view --flatten > merged.conf
  • Configures your current working environment.

    • Attention: set-context modifies context values! use-context change the current context!
      # list and change context
      kubectl config get-context
      kubectl config use-context CONTEXT_NAME
      # set the used default namespace
      kubectl config set-context --current --namespace=default

kubeconfig - tooling

Special thx to Ahmet Alp Balkan


Fastest way to install kubectx, kubens and fzf

cd $HOME/bin
chmod 755 kubectx kubens
tar xf fzf-0.18.0-linux_amd64.tgz
# → kubectx, kubens, fzf

kubeconfig - tooling

Usage kubectx, kubens

  • fast context switching

    # fuzzy search list
    # direct select e.g. `default` context 
    kubectx default
    # select last context
    kubectx -
  • fast namespace switching

    # fuzzy search list
    # direct select e.g. `kube-system` namespace 
    kubens kube-system
    # select last namespace
    kubens -

kubeconfig - tooling

Use fuzzy search fzf

  • Search a file the fuzzy way

    # with preview
    fzf --preview 'cat {}' 
  • Key binding → add ./fzf/.fzf.bash to your ~/.bashrc

    [ -f ~/.fzf.bash ] && source ~/.fzf.bash
  • Pipe kubectl output, e.g. logs, config

    k logs POD_NAME | fzf
    k get pod POD_NAME -o yaml | fzf

kubectl output parameter

  • --v=9 Debug verbosity 0-10

  • -o wide, -o yaml shows more important information about an object

  • --show-labels and --label-columns=k8s-app structure your output

  • -l k8s-app=my-app, --field-selector=status.phase=Running select objects

  • -o json | jq 'expresion' combine JSON and jq to get more details (useful for scripting)

  • jsonpath=JSONPATH_EXP powerful one line helper to get multiple valuesx of a json output

  • kubectl describe OBJECT shows information and events

kubectl output parameter


# all runnings pods
k get pod --field-selector=status.phase=Running
# node kernel version
k get nodes -o json | jq '.items[].status.nodeInfo.kernelVersion' -r

# all used images
kubectl get pods --all-namespaces \
  -o jsonpath='{range .items[*]}{}{" "}{@.spec.containers[*].image}{"\n"}{end}'

# Check which nodes are ready
JSONPATH='{range .items[*]}{"\n---\n"}{}: 
{"\n"}{range @.status.conditions[*]}{@.type}={@.status}; {"\n"}{end}{end}' \
 && kubectl get nodes -o jsonpath="$JSONPATH"

# troubleshoot node state
kubectl describe node NODE_NAME

Quick wins - let kubectl help you!

use run for resource creation (deprecated)

  • --image=image Docker image

  • --env="key=value" environment variable(s)

  • --port=port exposing port of container

  • --replicas=replicas count of replicas

  • --label="myapp=app1" add some label(s)

  • --restart trigger different kind of object creation:

    kubectl run # without flag creates a deployment
    kubectl run --restart=Never  # creates a Pod
    kubectl run --restart=OnFailure # creates a job
    kubectl run --restart=OnFailure -schedule="* * * * *" # creates a cronjob
  • run ... -- argument pass the arguments directly to the container

    # start a simple web image and test it with bussy box
    kubectl run --image=loodse/demo-www --port 80 web-deployment
    kubectl run --image=busybox --restart=Never --rm -it -- bash
    # ... inside the conainer: wget $WEB_DEPLOYMENT_SERVICE_HOST -O -

Quick wins - let kubectl help you!

new create for resource creation

kubectl create 
clusterrole          deployment           priorityclass        secret
clusterrolebinding   job                  quota                service
configmap            namespace            role                 serviceaccount
cronjob              poddisruptionbudget  rolebinding
  • --image=image Docker image
  • ... less options see kubectl create OBJECT --help


kubectl create deployment web-deployment --image=loodse/demo-www

Quick wins - let kubectl help you!

use expose for service creation

Can reference pod (po), service (svc), replicationcontroller (rc), deployment (deploy), replicaset (rs).

  • --port listing port to match at referenced resource
  • --type type of Service: ClusterIP (default), NodePort, LoadBalancer, ExternalName
  • --traget-port port at the service
  • --selector specify label selector
k expose deployment web-deployment --type=NodePort --port=80
k expose deployment web-deployment --type=LoadBalancer --port=80

k get nodes -o wide
k get nodes!=master \
  -o jsonpath={.items[0].status.addresses[?\(@.type==\"ExternalIP\"\)].address}
  • Combine with port-forward for quick testing or debugging
    • can target pod, deployment, service
    • use localport:remoteport for port mapping
k port-forward svc/web-deployment 8080:80 &
curl localhost:8080

Quick wins - let kubectl help you!

create templates

  • --dry-run combined with -o yaml and run --restart or create creates a template for common resource

    # create a deployment yaml file
    kubectl run --image=loodse/demo-www --port 80 --dry-run -o yaml web-template > dep.yaml
    kubectl create deployment web-template --image=loodse/demo-www --dry-run -o yaml > dep.yaml
    # job with 10 sleep 
    kubectl run --image=busybox --restart=OnFailure --dry-run -o yaml job -- /bin/sleep 10 > job.yaml
  • --export get a pod's YAML without cluster specific information

k get deployment web-deployment -o yaml --export > dep.export.yaml
vim dep.export.yaml 
k apply -f dep.export.yaml   

k get service web-deployment --export -o yaml > svc.export.yaml

Quick wins - let kubectl help you!

Modify resources

  • Use inplace editor functionality

    • KUBE_EDITOR sets the local editor
    • kubectl edit TYP OBJECT open in cluster resource
  • Use apply for mutable objects, replace for immutable objects. Note: You can use -f FOLDER for using multi manifests!

    kubectl apply -f dep.yaml
    # delete resource and recreates it
    kubectl replace --force -f pod.yaml
  • Use scaling functions

    • k autoscale deployment foo --min=2 --max=10 add HPA
    • k scale deployment --replicas=10 web-deployment scales up

Quick wins - let kubectl help you!

Modify resources

  • Manipulate current objects, e.g. the image value
    # use set for common modification
    k set image deployment/web-deployment web-deployment=loodse/demo-www
    k set env deployment/web-deployment TEST=val
    # use patch for all other, e.g. service type
    kubectl patch svc/web-deployment -p '{"spec":{"type":"LoadBalancer"}}' 
    # Update a container's image; spec.containers[*].name is required because it's a merge key
    kubectl patch pod/podname -p \

Basic cluster information

  • What cluster do I use?

    kubectl cluster-info
  • Whats about the components?

    kubectl get componentstatuses
    kubectl get cs
  • Troubleshoot the whole cluster

    # download the state
    kubectl cluster-info dump --output-directory=./output/cluster-state
    # diagnose it
    tree ./output/cluster-state
    grep -r Error output/cluster-state
    grep -C 5 -r Error output/cluster-state

Extend kubectl with plugins

  • Enable kubectl plugin manager krew

    # add to ~/.bashrc
    # export KREW_ROOT=/path/to/krew-folder
    export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"
  • Plugin management

kubectl-krew search
kubectl-krew insatll view-secret
  • Example: decode base64 secrets
kubectl get secret
kubectl view-secret default-token-976rc namespace

Use kubectl the fuzzy way with fubectl

  • bash/zsh wrapper based on kubectl, jq and fzf

  • Interactive search and interaction with Kubernetes objects

    • support's also CRD's
    • No in-cluster installation needed
  • Installation

    curl -LO
    # add to `~/.bashrc`
    [ -f <path-to>/fubectl.source ] && source <path-to>/fubectl.source  

  • Usage of fubectl:
    # [ka] get all pods in namespace
    # [kall] get all pods in cluster
    # [kwa] watch all pods in the current namespace
    # [kwall] watch all pods in cluster
    # [kp] open kubernetes dashboard with proxy
    # [kwatch] watch resource
    # [kdebug] start debugging in cluster
    # [kube_ctx_name] get the current context
    # [kube_ctx_namespace] get current namespace
    # [kget] get a resource by its YAML
    # [ked] edit a resource by its YAML
    # [kdes] describe resource
    # [kdel] delete resource
    # [klog] fetch log from container
    # [kex] execute command in container
    # [kfor] port-forward a container port to your local machine
    # [ksearch] search for string in resources
    # [kcl] context list
    # [kcs] context set
    # [kcns] context set default namespace
    # [kwns] watch pods in a namespace


  • Take a look for objects in state Pending, Error, CrashLoopBackOff

  • Use port-forward to test different connections, e.g. service or pod

  • Use prepared debug container for e.g. network debugging

    kubectl run --image=amouat/network-utils --restart=Never --rm -it -- bash
  • top for resource usage, requires metrics-server

    kubectl top node
    kubectl top pod   
  • Reproduce the event and stream all matching logs, e.g. with label name=myLabel

    • kubectl logs -f -l name=myLabel --all-containers
  • exec into running container

    • kubectl exec my-pod -- ls -la /
    • kubectl exec my-pod -it -- sh

Cluster Inspection Tools

++ need no running components in the cluster


  • provides a curses based terminal UI
  • interactive view similar to htop


  • Kubernetes Cluster Sanitizer
  • Find errors and warnings

Cluster Management by Cluster API

  • Manage Cluster's by CRDs in depedent of the provider (cloud/on-prem)
  • Currently mostly used machine creation, see as e.g. machine-controller implementations
    • Used by e.g. HA cluster management tool kubeOne
  • Immutable machine objects handle cluster nodes similar to pods
    • Deployment -> ReplicaSet -> Pod -> Container
    • MachineDeployment -> MachineSet -> Machine -> Node
    # see the machine definition
    k describe machine -n kube-system MACHINE_NAME
    k get machinedeployment,machineset,machine,node -n kube-system
    # update e.g. kubernetes version, machine size, ...
    k edit machinedeployment
    # machine to node reference:
    k get machine -n kube-system \
      -o jsonpath='{range .items[*]}{}{" >> "}{}{"\n"}{end}}'

Manage VMs with kubevirt


I'm happy to answer!


Take a look at

Something to add?

Open a pull request 😉

Thx for your attention!


You can’t perform that action at this time.