kubectl-dev

kubectl-dev is a kubectl plugin to support image building, workload debugging, and CliApp especially in a single-node minikube cluster which is used to replace Docker Desktop.

Currently, the plugin can only work on containerd.

Features

Build image or binary

Docker build conformance

The kubectl-dev build command is fully compatible to the docker build command. no-cache, build-arg and target are also supported.

# Build image foo:bar using Dockerfile in the current directory.
kubectl dev build -t foo:bar

# Build image foo:bar using foobar.dockerfile as the Dockerfile in diretory ~/image.
kubectl dev build -t foo:bar -f foobar.dockerfile ~/image

The builder also support pushing private images and insecure registries.

# Log into a image registry as using "docker login"
kubectl dev login

# Build image and push it to an insecure registry.
kubectl dev build -t foo:bar --push --insecure

Build artifacts

The build command also can copy artifacts from a complicated context to a local directory.

# Build the stage mac-cli and copy generated to the local directory _output.
kubectl dev build  -f hack/dev/Dockerfile --local _output/ --target mac-cli

Auto-generate image name for testing

Build command will automatically generate image name if no -t(--tag) or --local provided. The default name is in the format of build.local/x/%s:v%d. The %s is going to be replaced by the build context directory. The %d will be replaced by an integer incremented by 1 in each build.

Users can change the default pattern by setting --tag-pattern.

Apply k8s manifests after build

You can specify a k8s manifest file through --manifest to apply after build. Its image configuration will be updated with the built image name.

Remember arguments for replaying

Once you've built an image in some directory, all command line arguments are saved. You can build the same image in the same directory with just kubectl dev build command.

Debug workloads

If an app failed, it would crash, wait for deps and has no responding, fails on some libraries, say some .so files, or get wrong mounted ConfigMaps or Secrets. K8s provides nothing to figure them out. The only thing may help is logs your app printed.

The debug command provides a new way to start the workload. It creates an identical Pod in the same namespace, except the image of the target container. debug opens a bash session after the Pod started. The target image is mount to /app-root. You can check the original image context or debug the binary in the opened session.

Deployment, StatefulSet, DaemonSet, ReplicaSet, Job, CronJob, and Pod are all supported.

# Debug a running or failed workload. Run the same command again could open a new session to the same debugger.
kubectl dev debug -n cliapp-system deploy buildkitd

# The debugger Pod would not fork environment variables from the original workload.
kubectl dev debug -n cliapp-system deploy buildkitd --with-original-envs=false

# Specify container name if more than one containers in the Pod. Or, an error would arise.
kubectl dev debug -n cliapp-system deploy buildkitd -c buildkitd

# Pass the local HTTP_PROXY to the debugger Pod.
kubectl dev debug -n cliapp-system deploy buildkitd --use-proxy

# Debug a Pod with a new versioned image. 
kubectl dev debug pod foo --image bar:new-version

#Debug an image.
kubectl dev debug --image foo:latest

The default distro of debugger is alpine. ubuntu would be another option. You can also choose one of bash or zsh as your favorite in debuggers via option --shell.

kubectl dev debug -n cliapp-system deploy buildkitd --with-original-envs=false --shell zsh --distro ubuntu

Use CliApp

CliApp provides the capability of running cli commands, which are installed in the cluster, from a local terminal.

Besides installing a CliApp object in the cluster, a shortcut w/ the same is created in the directory ~/.cliapps/ and is linked to /usr/local/bin/.

# Install cliapp crictl via image docker.io/warmmetal/app-crictl:v0.1.0.
# The last argument "crictl" shows that command crictl will be executed in the Pod once the app is executed. 
# If omitted, the command same with the app name is started instead.
sudo -E kubectl dev app install --name crictl \
	--image docker.io/warmmetal/app-crictl:v0.1.0 \
	--env CONTAINER_RUNTIME_ENDPOINT=unix:///var/run/containerd/containerd.sock \
	--hostpath /var/run/containerd/containerd.sock --use-proxy \
	crictl
# ❯ command -v crictl
# /usr/local/bin/crictl
# ❯ ls -l /usr/local/bin/crictl
# lrwxr-xr-x  1 root  wheel  25 Mar 14 18:57 /usr/local/bin/crictl -> /Users/kh/.cliapps/crictl

You can install a CliApp via a Dockerfile and the builtin buildkit will help build the necessary image.

sudo -E kubectl dev app install --name ctr \
	--dockerfile https://raw.githubusercontent.com/warm-metal/cliapps/master/ctr/Dockerfile \
	--env CONTAINERD_NAMESPACE=k8s.io \
	--hostpath /var/run/containerd/containerd.sock --use-proxy

Installation

From Homebrew

The Homebrew formulae is available for MacOS.

brew install warm-metal/rc/kubectl-dev

From the pre-built binary

You can also download the pre-built binary.

# For MacOS, the administrator privilege is required to save kubectl-dev to /usr/local/bin. Run
sudo sh -c 'curl -skL https://github.com/warm-metal/kubectl-dev/releases/download/v0.4.0/kubectl-dev.darwin-amd64.tpxz | tar -C /usr/local/bin/ -xpf -'

# For Linux, run
sudo sh -c 'curl -skL https://github.com/warm-metal/kubectl-dev/releases/download/v0.4.0/kubectl-dev.linux-amd64.tpxz | tar -C /usr/local/bin/ -xpf -'

Initialization

After installed, run one of the commands below to install deps.

# For minikube clusters
kubectl dev prepare --minikube

# Inherit current HTTP_PROXY in the buildkit workspace.
# If you are in mainland China, this flag could accelerate the speed of image and dependency pulling while building.
kubectl dev prepare --minikube --use-proxy

# Install cliapp and set the environment variable to buildkit.
kubectl dev prepare --builder-env GOPROXY='https://goproxy.cn|https://goproxy.io|direct'

# For containerd
kubectl dev prepare

Build from Source

# For MacOS, run
kubectl dev build  -f hack/dev/Dockerfile --local _output/ --target mac-cli

# For Linux, run
kubectl dev build  -f hack/dev/Dockerfile --local _output/ --target linux-cli

Or, use Makefile instead.

Prepare a minikube cluster for program development

kubectl-dev offers you some capabilities to build images and debug them in k8s clusters directly. You don't need to install many runtime and many more versions of them in your laptop. No runtime changing and management. Also, no out-of-date dep garbage. All these are replaced by a k8s cluster.

To own a local minikube cluster on your laptop is not easy as running minikube start. It could be a little tricky.

Create (start the first time) a cluster w/ containerd

mini_create() {
  PROFILE=minikube
  if [[ $# -gt 0 ]]; then
    PROFILE=$1
  fi

  minikube start -p $PROFILE \
      --service-cluster-ip-range="10.24.0.0/16" \
      --container-runtime=containerd \
      --memory=8g \
      --cpus=4 \
      --disk-size=100g
}

Start an existed cluster

After the cluster created, we must disable both --preload and --cache-images. Or,
with --preload enabled, the containerd content store will be override by a pre-downloaded tarball. If --cache-images enabled, minikube always try to save images to local tarballs.

mini_start() {
  PROFILE=minikube
  if [[ $# -gt 0 ]]; then
    PROFILE=$1
  fi

  minikube start -p $PROFILE --preload=false --cache-images=false
}

Time Sync

With hyberkit, the guest can't sync its datetime to local host. The easiest way to keep time sync is using NTP. But, if you are in a poor network or behind a powerful firewall, the default NTP settings is useless.

To fix it, you can try another wheel we built, kube-systemd.