Skip to content
$ kubectl hacking - my journey with kate
HCL Shell
Branch: master
Clone or download
Latest commit 8027117 Jun 12, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
fzf update Jun 5, 2019
kubeone-cluster add kubeone config Jun 11, 2019
pics add first draft of kubectl hacking slides Jun 5, 2019
powerline-go add first draft of kubectl hacking slides Jun 5, 2019
.gitignore add first draft of kubectl hacking slides Jun 5, 2019
README.md update slides Jun 12, 2019
presentation-notes.md update Jun 5, 2019

README.md

title author patat
$ kubectl hacking - my journey with kate
Tobias Schneck, Loodse GmbH
wrap pandocExtensions margins theme
true
patat_extensions
autolink_bare_uris
left right
5
5
syntaxHighlighting code codeBlock
decVal
bold
onDullRed
dullBlack
onDullWhite
dullBlack
onDullWhite

My journey to Kubernetes

Java programmer -> Testautomation -> Docker -> OpenShift -> Kubernetes

                              ..              
                              ..              
                           ........           
                   ...  ..++..++..++..  ...   
                    ..+++..  .++.  ..+++..    
                     .+.++.. .++. ..++.+.     
                    .+.  .++++++++++.  .+.    
                    ++.  ..++....++..  .++    
                  ..++.+++++++..+++++++.++..  
                .....++..  .++++++.  ..++.....
                     .++.  ++.  .++  .++.     
                      ..+..+.    .+..+..      
                         .+.++++++.+.         
                        .+.        .+.        
                        +.          .+                          

📧 mailto:tobi@loodse.com

🐦 https://twitter.com/toschneck

🐙 https://github.com/toschneck


kubectl - what's this about?


basic bash config

  • Enable bash auto completion (update compatible)

    # add to ~/.bashrc
    source <(kubectl completion bash)
    alias k=kubectl
    complete -F __start_kubectl k
  • beautify your bash prompt -> powerline go

    1. Install binary go get -v -u github.com/justjanne/powerline-go
    2. ensure powerline-go is available in the path: export PATH=$PATH:$GOPATH/bin
    3. config ~/.bashrc -> source ./powerline-go/.bashrc

kubectl help?

  • kubectl [command] --help is very helpful!
  • kubectl explain [object][.field][...] information about spec fields!
  • kubectl api-resources shows available objects - with CRDs!

Examples

kubectl get --help
#options for all commands
kubectl options

kubectl explain pod.spec.containers.ports
kubectl explain svc.spec.type

kubectl api-resources --api-group=apps
kubectl api-resources -o wide

kubeconfig

  • Default configuration file: ~/.kube/config

    • endpoints
    • SSL keys
    • contexts
  • $KUBECONFIG environment variable or --kubeconfig flag

    • useful to manage multiple cluster
    • merge multiple config files
    KUBECONFIG=conf1:conf2 kubectl config view --flatten > merged.conf
  • Configures your current working environment.

    • Attention: set-context modifies context values! use-context change the current context!
      # list and change context
      kubectl config get-context
      kubectl config use-context CONTEXT_NAME
      
      # set the used default namespace
      kubectl config set-context --current --namespace=default

kubeconfig - tooling

Special thx to Ahmet Alp Balkan

Installation

Fastest way to install kubectx, kubens and fzf

cd $HOME/bin
wget https://raw.githubusercontent.com/ahmetb/kubectx/master/kubectx
wget https://raw.githubusercontent.com/ahmetb/kubectx/master/kubens
chmod 755 kubectx kubens
wget https://github.com/junegunn/fzf-bin/releases/download/0.18.0/fzf-0.18.0-linux_amd64.tgz
tar xf fzf-0.18.0-linux_amd64.tgz
# → kubectx, kubens, fzf

kubeconfig - tooling

Usage kubectx, kubens

  • fast context switching

    # fuzzy search list
    kubectx
    # direct select e.g. `default` context 
    kubectx default
    # select last context
    kubectx -
  • fast namespace switching

    # fuzzy search list
    kubens
    # direct select e.g. `kube-system` namespace 
    kubens kube-system
    # select last namespace
    kubens -

kubeconfig - tooling

Use fuzzy search fzf

  • Search a file the fuzzy way

    fzf
    # with preview
    fzf --preview 'cat {}' 
  • Key binding → add ./fzf/.fzf.bash to your ~/.bashrc

    [ -f ~/.fzf.bash ] && source ~/.fzf.bash
  • Pipe kubectl output, e.g. logs, config

    k logs POD_NAME | fzf
    k get pod POD_NAME -o yaml | fzf

kubectl output parameter

  • --v=9 Debug verbosity 0-10

  • -o wide, -o yaml shows more important information about an object

  • --show-labels and --label-columns=k8s-app structure your output

  • -l k8s-app=my-app, --field-selector=status.phase=Running select objects

  • -o json | jq 'expresion' combine JSON and jq to get more details (useful for scripting)

  • jsonpath=JSONPATH_EXP powerful one line helper to get multiple valuesx of a json output

  • kubectl describe OBJECT shows information and events


kubectl output parameter

Examples

# all runnings pods
k get pod --field-selector=status.phase=Running
 
# node kernel version
k get nodes -o json | jq '.items[].status.nodeInfo.kernelVersion' -r

# all used images
kubectl get pods --all-namespaces \
  -o jsonpath='{range .items[*]}{@.metadata.name}{" "}{@.spec.containers[*].image}{"\n"}{end}'

# Check which nodes are ready
JSONPATH='{range .items[*]}{"\n---\n"}{@.metadata.name}: 
{"\n"}{range @.status.conditions[*]}{@.type}={@.status}; {"\n"}{end}{end}' \
 && kubectl get nodes -o jsonpath="$JSONPATH"

# troubleshoot node state
kubectl describe node NODE_NAME

Quick wins - let kubectl help you!

use run for resource creation (deprecated)

  • --image=image Docker image

  • --env="key=value" environment variable(s)

  • --port=port exposing port of container

  • --replicas=replicas count of replicas

  • --label="myapp=app1" add some label(s)

  • --restart trigger different kind of object creation:

    kubectl run # without flag creates a deployment
    kubectl run --restart=Never  # creates a Pod
    kubectl run --restart=OnFailure # creates a job
    kubectl run --restart=OnFailure -schedule="* * * * *" # creates a cronjob
  • run ... -- argument pass the arguments directly to the container

    # start a simple web image and test it with bussy box
    kubectl run --image=loodse/demo-www --port 80 web-deployment
    kubectl run --image=busybox --restart=Never --rm -it -- bash
    # ... inside the conainer: wget $WEB_DEPLOYMENT_SERVICE_HOST -O -

Quick wins - let kubectl help you!

new create for resource creation

kubectl create 
clusterrole          deployment           priorityclass        secret
clusterrolebinding   job                  quota                service
configmap            namespace            role                 serviceaccount
cronjob              poddisruptionbudget  rolebinding
  • --image=image Docker image
  • ... less options see kubectl create OBJECT --help

Example

kubectl create deployment web-deployment --image=loodse/demo-www

Quick wins - let kubectl help you!

use expose for service creation

Can reference pod (po), service (svc), replicationcontroller (rc), deployment (deploy), replicaset (rs).

  • --port listing port to match at referenced resource
  • --type type of Service: ClusterIP (default), NodePort, LoadBalancer, ExternalName
  • --traget-port port at the service
  • --selector specify label selector
k expose deployment web-deployment --type=NodePort --port=80
k expose deployment web-deployment --type=LoadBalancer --port=80

k get nodes -o wide
k get nodes --selector=kubernetes.io/role!=master \
  -o jsonpath={.items[0].status.addresses[?\(@.type==\"ExternalIP\"\)].address}
  • Combine with port-forward for quick testing or debugging
    • can target pod, deployment, service
    • use localport:remoteport for port mapping
k port-forward svc/web-deployment 8080:80 &
curl localhost:8080

Quick wins - let kubectl help you!

create templates

  • --dry-run combined with -o yaml and run --restart or create creates a template for common resource

    # create a deployment yaml file
    kubectl run --image=loodse/demo-www --port 80 --dry-run -o yaml web-template > dep.yaml
    kubectl create deployment web-template --image=loodse/demo-www --dry-run -o yaml > dep.yaml
    
    # job with 10 sleep 
    kubectl run --image=busybox --restart=OnFailure --dry-run -o yaml job -- /bin/sleep 10 > job.yaml
  • --export get a pod's YAML without cluster specific information

#deployment
k get deployment web-deployment -o yaml --export > dep.export.yaml
vim dep.export.yaml 
k apply -f dep.export.yaml   

#service
k get service web-deployment --export -o yaml > svc.export.yaml

Quick wins - let kubectl help you!

Modify resources

  • Use inplace editor functionality

    • KUBE_EDITOR sets the local editor
    • kubectl edit TYP OBJECT open in cluster resource
  • Use apply for mutable objects, replace for immutable objects. Note: You can use -f FOLDER for using multi manifests!

    kubectl apply -f dep.yaml
    # delete resource and recreates it
    kubectl replace --force -f pod.yaml
  • Use scaling functions

    • k autoscale deployment foo --min=2 --max=10 add HPA
    • k scale deployment --replicas=10 web-deployment scales up

Quick wins - let kubectl help you!

Modify resources

  • Manipulate current objects, e.g. the image value
    # use set for common modification
    k set image deployment/web-deployment web-deployment=loodse/demo-www
    k set env deployment/web-deployment TEST=val
    
    # use patch for all other, e.g. service type
    kubectl patch svc/web-deployment -p '{"spec":{"type":"LoadBalancer"}}' 
    
    # Update a container's image; spec.containers[*].name is required because it's a merge key
    kubectl patch pod/podname -p \
     '{"spec":{"containers":[{"name":"web-deployment","image":"loodse/demo-www"}]}}'

Basic cluster information

  • What cluster do I use?

    kubectl cluster-info
    
  • Whats about the components?

    kubectl get componentstatuses
    kubectl get cs
  • Troubleshoot the whole cluster

    # download the state
    kubectl cluster-info dump --output-directory=./output/cluster-state
    
    # diagnose it
    tree ./output/cluster-state
    
    grep -r Error output/cluster-state
    grep -C 5 -r Error output/cluster-state

Extend kubectl with plugins

  • Enable kubectl plugin manager krew

    # add to ~/.bashrc
    #
    # export KREW_ROOT=/path/to/krew-folder
    export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"
  • Plugin management

kubectl-krew search
kubectl-krew insatll view-secret
  • Example: decode base64 secrets
kubectl get secret
kubectl view-secret default-token-976rc namespace

Use kubectl the fuzzy way with fubectl

  • bash/zsh wrapper based on kubectl, jq and fzf

  • Interactive search and interaction with Kubernetes objects

    • support's also CRD's
    • No in-cluster installation needed
  • Installation

    curl -LO https://rawgit.com/kubermatic/fubectl/master/fubectl.source
    # add to `~/.bashrc`
    [ -f <path-to>/fubectl.source ] && source <path-to>/fubectl.source  

  • Usage of fubectl:
    khelp
    # [ka] get all pods in namespace
    # [kall] get all pods in cluster
    # [kwa] watch all pods in the current namespace
    # [kwall] watch all pods in cluster
    # [kp] open kubernetes dashboard with proxy
    # [kwatch] watch resource
    # [kdebug] start debugging in cluster
    # [kube_ctx_name] get the current context
    # [kube_ctx_namespace] get current namespace
    # [kget] get a resource by its YAML
    # [ked] edit a resource by its YAML
    # [kdes] describe resource
    # [kdel] delete resource
    # [klog] fetch log from container
    # [kex] execute command in container
    # [kfor] port-forward a container port to your local machine
    # [ksearch] search for string in resources
    # [kcl] context list
    # [kcs] context set
    # [kcns] context set default namespace
    # [kwns] watch pods in a namespace

Troubleshooting

  • Take a look for objects in state Pending, Error, CrashLoopBackOff

  • Use port-forward to test different connections, e.g. service or pod

  • Use prepared debug container for e.g. network debugging

    kubectl run --image=amouat/network-utils --restart=Never --rm -it -- bash
  • top for resource usage, requires metrics-server

    kubectl top node
    kubectl top pod   
  • Reproduce the event and stream all matching logs, e.g. with label name=myLabel

    • kubectl logs -f -l name=myLabel --all-containers
  • exec into running container

    • kubectl exec my-pod -- ls -la /
    • kubectl exec my-pod -it -- sh

Cluster Inspection Tools

++ need no running components in the cluster

k9s

  • provides a curses based terminal UI
  • interactive view similar to htop

popeye

  • Kubernetes Cluster Sanitizer
  • Find errors and warnings

Cluster Management by Cluster API

  • Manage Cluster's by CRDs in depedent of the provider (cloud/on-prem)
  • Currently mostly used machine creation, see as e.g. machine-controller implementations
    • Used by e.g. HA cluster management tool kubeOne
  • Immutable machine objects handle cluster nodes similar to pods
    • Deployment -> ReplicaSet -> Pod -> Container
    • MachineDeployment -> MachineSet -> Machine -> Node
    # see the machine definition
    k describe machine -n kube-system MACHINE_NAME
    k get machinedeployment,machineset,machine,node -n kube-system
    
    # update e.g. kubernetes version, machine size, ...
    k edit machinedeployment
    
    # machine to node reference:
    k get machine -n kube-system \
      -o jsonpath='{range .items[*]}{@.metadata.name}{" >> "}{@.status.nodeRef.name}{"\n"}{end}}'

Manage VMs with kubevirt


Questions?

I'm happy to answer!

Slides?

Take a look at https://github.com/loodse/kubectl-hacking

Something to add?

Open a pull request 😉

Thx for your attention!


References:


You can’t perform that action at this time.