workshop.slide

Basics with rkt, the container engine by CoreOS

27th June 2016

Sergiusz Urbaniak
rkt Engineer, CoreOS
sur@coreos.com
@_surbaniak

* whoami

Software Engineer at CoreOS

rkt developer

Yes ... I do use the Linux Desktop :-)

* Overview

.image rkt8s.png _ 600

- Learn about rkt
- Use rkt
- Learn about Kubernetes
- Use Kubernetes with rkt

Requirements:

- Vagrant
- Virtualbox

* Setup

  git clone https://github.com/coreos/rktnetes-workshop
  cd vagrant
  vagrant up
  vagrans ssh

* rkt - What is it?

rkt (pronounced _rock-it_) is a CLI for running app containers on Linux. rkt is designed to be secure, composable, and standards-based.

- It runs on Linux only (currently).
- It runs on Intel `amd64`.
- Port to `arm64` is underway.

* rkt - a brief history

*December*2014* - _v0.1.0_

- Prototype
- Drive conversation (security, standards) and competition (healthy OSS) in container ecosystem

*February*2016* - _v1.0.0_

- Used in Production
- API stability guarantees

*June*2016* - _v1.9.0_

- Packaged in Debian, Fedora, Arch, NixOS

* rkt - Philosophy, Idioms

*UX:*"secure-by-default"*

- Verify image signatures by default
- Verify image integrity by default

*Architecture:*Unix*philosophy*

- Well-defined operations
- No central priviledged long-running components
- Separate privileges for different operations, i.e. "fetch"
- Verbose at times

* rkt - Security

- User namespaces (Container `euid` `!=` `host_euid`)
- SELinux contexts (Isolate individual pods)
- VM containment (LKVM)
- TPM measurements (Tamper-proof audit log of what's running)

* rkt - Security (soon)

- Finer-grained Linux capabilities
- Seccomp defaults
- Better SELinux support
- Cgroup2 + cgroup namespaces support
- Qemu (hypervisor) support
- Unpriviledged containers

* rkt - Composability

*External*

- Integrate with existing init system(s)
- Work well with higher level cluster orchestration tools (i.e. Kubernetes)
- Simple process model: rkt command _is_ the container
- Any context applied to rkt (cgroups, etcd.) applies transitively to the pods inside rkt
- Optional gRPC API server

*Internal*

- stage based architecture
- Swappable execution engines running containers

* rkt vs. Docker

.image rkt-vs-docker-process-model.png 500 _

* rkt vs. Docker

.image rkt-vs-docker-fetch.png 500 _

* rkt - Speaking of images

.image image-standards.png

* rkt - Image support

- ACI - yes
- Docker - yes

Caveat: Docker images do not suport signing, start with `--insecure-options=image`

- OCI - WIP

*Workshop*time*

1. Start nginx

  rkt run --insecure-options=image docker://nginx

2. Start an interactive container

  rkt run --insecure-options=image --interactive docker://progrium/busybox

quit by hitting `Ctrl-]` three times

* rkt vs. ?

.link https://github.com/coreos/rkt/blob/master/Documentation/rkt-vs-other-projects.md

* rkt stages

Execution with rkt is divided into several distinct stages.

*stage0*

The `rkt` binary itself.

- Fetch images
- Generate pod UUIDs, manifest
- Create filesystem for the pod
- Set up further stages
- Unpack stage1 into pod file system
- Unpack app images into stage2 directories

* rkt stages - stage1

*stage1*:

An image providing:

- Initializes container isolation
- Starts and initializes container supervision
- Reads pods and image manifests
- Starts the actual applications

This is swappable, comes in different "flavors":

- _fly_: does nothing really ... just a chroot environment
- _coreos_: uses Linux cgroups, namespaces, leverages systemd
- _kvm_: Uses LKVM supervisor to boot a real virtualized kernel

* rkt stages - coreos stage1

Provides container isolation

stage1-coreos.aci

  host OS
  └─ rkt
    └─ systemd-nspawn
      └─ systemd
        └─ chroot
          └─ user-app1

- *systemd-nspawn*: Implements the initialization of cgroups, namespaces
- *systemd*: Implements supervision of containers

* rkt stages - fly stage1

stage1-fly.aci

  host OS
  └─ rkt
    └─ chroot
      └─ user-app1

- Just a chroot'ed application

but

- Secure image retrieval including signature validation
- Lightweight, isolated package manager

* rkt stages - ??? stage1

Crazy ideas:

- _xhyve_: OSX virtualization allowing rkt to run natively on a Mac
- _runit_: Don't like systemd? Bring your own isolator/supervisor

PRs/Contributions welcome!

* rkt stages - fly stage2

.image you.png _ 400

* rkt - build your own image

*Workshop*time*

.link https://github.com/s-urbaniak/inspector
.link https://coreos.com/rkt/docs/latest/signing-and-verification-guide.html

- Build a small Go application
- Build an ACI image
- Sign the image using `gpg`
- Make the image discoverable via github pages

* Let's step back

.image os-procs.png _ 200

In a classical "OS" setup we have:

- A supervisor, aka "init daemon", aka PID1
- Not only one process, but many processes
- Processes work together, either via localhost net, IPC
- Communicate with outside world

* rkt - Pods

.image pod-apps.png _ 300

- Grouping of applications executing in a shared context (network, namespaces, volumes)
- Shared fate
- The _only_ execution primitive: single applications are modelled as singleton pods

* rkt - Sample Pod: micro-service talking to Redis

*Workshop*time*

.image redis-service.png _ 230

  sudo rkt run --insecure-options=image docker://redis \
    s-urbaniak.github.io/rktnetes-workshop/redis-service:0.0.1
  rkt list && rkt cat-manifest deadbeef
  curl <pod-ip>:8080

.link https://github.com/s-urbaniak/redis-service

* Pods - Patterns, patterns everywhere

Container/App Design Patterns

- Kubernetes enables new design patterns
- Similar to OO patterns
- Key difference: technologically agnostic

.link http://blog.kubernetes.io/2016/06/container-design-patterns.html
.link https://www.usenix.org/system/files/conference/hotcloud16/hotcloud16_burns.pdf

* Pods - Sidecar pattern

.image pattern-sidecar.png _ 400

- Auxiliary app
- Extend, enhance main app

Pros:

- Separate packaging units
- Each app contained in a separate failure boundary
- Potentially different technologies/languages

* Pods - Ambassador pattern

.image pattern-ambassador.png _ 400

- Proxy communication
- Separation of concerns
- Main app has simplified view

* Pods - Adapter pattern

.image pattern-adapter.png _ 400

- Use an interface of an existing app as another interface
- Very useful for legacy apps, translating protocols

* Pods - Leader election pattern

.image pattern-leader.png _ 400

- Separate the leader logic from the election logic
- Swappable algorithms/technologies/environments

Ready-to-use generic leader elector:

.link http://blog.kubernetes.io/2016/01/simple-leader-election-with-Kubernetes.html

* Pods - Work queue pattern

.image pattern-work-queue.png _ 400

- Separate app logic from queue enqueing/dequeing

* Pods - Scatter gather pattern

.image pattern-scatter-gather.png _ 400

- Main app sends a simple request
- Auxiliary app implements complex scatter/gather logic
- Fan-Out/Fan-In requests separate from main app

* rkt - Networking

The CNI (Container Network Interface)

.image pod-net.png _ 300

- Abstraction layer for network configuration
- Single API to multiple, extensible networks
- Narrow, simple API
- Plugins for third-party implementations

* rkt - Networking - Host Mode

.image host-mode.png _ 300

  rkt run --net=host ...

- Inherit the network namespace of the process that is invoking rkt.
- Pod apps are able to access everything associated with the host’s network interfaces.

*Workshop*time*

1. Start nginx using `--net=host`

* rkt - Networking - Default Mode (CNI ptp)

.image ptp.png _ 300

  rkt run --net ...
  rkt run --net=default ...

.link https://github.com/containernetworking/cni/blob/master/Documentation/ptp.md

- Creates a virtual ethernet pair
- One placed in the pod
- Other one placed on the host

* rkt - Networking - CNI brigde

.image bridge.png _ 300

.link https://github.com/containernetworking/cni/blob/master/Documentation/bridge.md

- Creates a virtual ethernet pair
- One placed in the pod
- Other one placed on the host
- Host veth pluggind into a linux bridge

* rkt - Networking - CNI macvlan

.image macvlan.png _ 300

.link https://github.com/containernetworking/cni/blob/master/Documentation/macvlan.md

- Functions like a switch
- Pods get different MAC addresses
- Pods share the same physical device

* rkt - Networking - CNI ipvlan

.image ipvlan.png _ 300

.link https://github.com/containernetworking/cni/blob/master/Documentation/ipvlan.md

- Functions like a switch
- Pods share the same MAC address
- Pods get different IPs
- Pods share the same physical device

* rkt - Networking

*Workshop*time*

Configure a ptp network, start two `busybox` pods pinging each other.

* rkt - Networking - SDN (software defined networking)

.image pod-net-canal.png 300 _

- Communicate with pods across different _hosts_
- Each pod across all hosts gets its own IP
- Virtual overlay network

* Kubernetes - Overview

.image flower.png

.link https://github.com/kubernetes/kubernetes

- Open Source project initiated by Google
- Cluster-level container orchestration

Handles:

- Scheduling/Upgrades
- Failure recovery
- Scaling

* k8s - Components - API Server

- Validates, configures, persists for all Kubernetes API objects
- Provides REST based operations via JSON
- Uses etcd is its database

* k8s - Components - Controller Manager

.image control-loop.png _ 300

- Embeds the core control loops
- Is _state-less_
- Is decoupled from etcd via the API-server

Just a daemon talking to etcd

* k8s - Components - Scheduler

Schedules pods on nodes

- Policy-rich
- Topology-aware
- Workload-specific
- Considers resource requirements, QoS, HW/SW/policy constraints, ...

Again ... just a daemon talking to etcd

* k8s - Components - Kubelet

- Primary node agent
- Starts/Stops/Supervises pods on its node

Sigh ... just a daemon talking to etcd and to *rkt* !!!

Instructs rkt to:

- Fetch pods
- Start, and stop pods
- Exec into pods
- Tail pod logs

PS: We are working very hard to make *rkt* a first class runtime in Kubernetes.

* k8s - Components - Kube Proxy

Primary network proxy on each node.

- configures `iptables` to reflect services
- Forwards TCP/UDP streams across a set of backends

* k8s - The big picture

.image k8s-overview.png 350 _

1. Watches created pods, assigns them to nodes
2. Runs controllers (node ctl, replication ctl, endpoint ctl, service account ctl, ...)
3. Watches for pods assigned to its node, runs *rkt*!
4. Manipulates network rules (iptables) for services on a node, does connection forwarding.

* k8s - Let's do it!

*Workshop*time*

1. Start Kubernetes
2. Create a service
3. Launch nginx
4. Launch busybox
5. Exec into busybox
6. Launch the Kubernetes dashboard
7. Open it from the host machine

See the next slides for templates.

* rktnetes - nginx Service

*Workshop*time*

  kind: Service
  apiVersion: v1
  metadata:
    labels:
      app: nginx
    name: nginx
  spec:
    type: NodePort
    ports:
    - port: 80
      targetPort: 80
    selector:
      app: nginx

 

  kubectl create -f nginx-svc.yaml

* rktnetes - nginx Replication Controller

    kind: ReplicationController
    apiVersion: v1
    metadata:
      labels:
        app: nginx
      name: nginx
    spec:
      replicas: 1
      selector:
        app: nginx
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - name: nginx
            image: nginx
            ports:
            - containerPort: 80
              protocol: TCP

 

  kubectl create -f nginx-rc.yaml

* rktnetes - busybox pod

*Workshop*time*

  apiVersion: v1
  kind: Pod
  metadata:
    name: busybox
  spec:
    containers:
    - name: busybox
      image: progrium/busybox
      args:
      - sleep
      - "1000000"

 

  kubectl create -f busybox.yaml

Exec into the pod:

  kubectl exec -ti busybox /bin/sh

* rktnetes - Dashboard

Deploy the Kubernetes Dashboard

*Workshop*time*

  kubectl create -f https://rawgit.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml
  ip address show dev eth1
  kubect get po --namespace=kube-system kubernetes-dashboard -o yaml

1. Get the assigned NodePort
2. Open http://<vm-ip>:NodePort on the host

* rktnetes - Ingress

Deploy an Ingress Controller based on traefik

*Workshop*time*

  kubectl create -f /vagrant/traefik.yaml

Add the following entry to `/etc/hosts`, replace `vm-ip` with the VM IP

  <vm-ip>	traefik.rktnetes

.link http://traefik.rktnetes http://traefik.rktnetes

* rktnetes - Ingress

.image traefik-ingress.png _ 1000

* rktnetes - Expose Dashboard via Ingress

*Workshop*time*

  apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
   name: kubernetes-dashboard-ingress
   namespace: kube-system
  spec:
   rules:
     - host: dashboard.rktnetes
       http:
         paths:
           - path: /
             backend:
               serviceName: kubernetes-dashboard
               servicePort: 80

Add the following entry to `/etc/hosts`, replace `vm-ip` with the VM IP

  <vm-ip>	dashboard.rktnetes

.link http://dashboard.rktnetes http://dashboard.rktnetes

* rktnetes - Expose Dashboard via Ingress

.image dashboard-ingress.png _ 1000