Docker Desktop 代替を検証する #67

suzuki-shunsuke · 2021-09-11T05:13:07Z

Mac で普段開発していて、 Docker を使うときは Docker for Mac を使ってきた。
普段は停止しておいて Docker が必要なときに起動するようにしている。

最近 Docker Desktop の有償化の話もあったので、 Docker Desktop の代替を検証してみた。

https://www.docker.com/blog/updating-product-subscriptions/

suzuki-shunsuke · 2021-09-11T05:46:25Z

Docker Desktop
lima: https://github.com/lima-vm/lima
- Mac で Linux VM を動かす
- containerd for Mac
Podman: https://podman.io/
- daemonless container engine
nerdctl: https://github.com/containerd/nerdctl
- Docker-compatible CLI for containerd, with support for Compose
docker
docker-engine
containerd: https://containerd.io/
- CNCF graduated project
minikube: https://minikube.sigs.k8s.io/docs/

suzuki-shunsuke · 2021-09-11T05:49:08Z

suzuki-shunsuke · 2021-09-11T05:53:25Z

Podman

https://podman.io/getting-started/installation

Podman は Linux 上で動く
Mac の場合、 podman machine コマンドで VM を管理できる

suzuki-shunsuke · 2021-09-11T06:05:10Z

やりたいこと

コンテナ操作 (docker run, etc)
image build (docker build)
Docker Compose (docker-compose up, etc)
- Podman はサポートしてないはず？
- nerdctl はしてるらしい
k8s (minikube, kind)

suzuki-shunsuke · 2021-09-11T06:09:44Z

podman は簡単にインストールできたし、普通に podman run でコンテナ起動できた

$ brew install podman
$ podman machine init
$ podman machine start
$ podman run -e GITHUB_TOKEN=$GITHUB_TOKEN -ti --rm alpine:3.14.2 sh

build もできた

FROM alpine:3.14.2
RUN apk add curl

$ podman build -t foo .

suzuki-shunsuke · 2021-09-11T06:19:18Z

podman で kind を動かす

kind 的にはサポートしてそうに見える。

https://kind.sigs.k8s.io/

Dockershim deprecation does NOT impact kind. 🐳
If you already use kind you've actually been testing your workloads on containerd!

While kind uses docker or podman on your host, it uses CRI / containerd "inside" the nodes and does not use dockershim.

Carry on and be KiND!

雑に実行してみたら動かない

$ kind create cluster --name test

ERROR: failed to create cluster: failed to list clusters: command "docker ps -a --filter label=io.x-k8s.kind.cluster=test --format '{{.Names}}'" failed with error: exit status 1
Command Output: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

https://kind.sigs.k8s.io/docs/user/rootless/#creating-a-kind-cluster-with-rootless-podman

$ KIND_EXPERIMENTAL_PROVIDER=podman kind create cluster --name test

using podman due to KIND_EXPERIMENTAL_PROVIDER
enabling experimental podman provider
Cgroup controller detection is not implemented for Podman. If you see cgroup-related errors, you might need to set systemd property "Delegate=yes", see https://kind.sigs.k8s.io/docs/user/rootless/
Creating cluster "test" ...
 ✗ Ensuring node image (kindest/node:v1.21.1) 🖼 
ERROR: failed to create cluster: failed to pull image "kindest/node@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6": command "podman pull kindest/node@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6" failed with error: exit status 125
Command Output: Error: short-name resolution enforced but cannot prompt without a TTY

$ podman pull kindest/node@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6
podman pull kindest/node@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6
Error: short-name resolution enforced but cannot prompt without a TTY

suzuki-shunsuke · 2021-09-11T06:46:54Z

Podman では Docker image の short name の扱いが Docker と違う

podman info で見ると docker.io が補完されそうに見える

$ podman info

registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io

Error: short-name resolution enforced but cannot prompt without a TTY

short-name resolution enforced この設定が Mac だとどこでされているのか？

https://podman.io/getting-started/installation#configuration-files

VM の中にあった。

$ podman machine ssh
[core@localhost ~]$ grep short-name-mode /etc/containers/registries.conf
short-name-mode="enforcing"
[core@localhost ~]$ sudo vi /etc/containers/registries.conf

permissive にしてみる。再起動とかは不要で即反映された。

$ podman pull kindest/node@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6

ただし、 short name はセキュリティ的なリスクもある。

suzuki-shunsuke · 2021-09-11T07:04:07Z

permissive にしたところ、 podman pull はできたものの、 kind create cluster は失敗した。

$ KIND_EXPERIMENTAL_PROVIDER=podman kind create cluster --name test
KIND_EXPERIMENTAL_PROVIDER=podman kind create cluster --name test
using podman due to KIND_EXPERIMENTAL_PROVIDER
enabling experimental podman provider
Cgroup controller detection is not implemented for Podman. If you see cgroup-related errors, you might need to set systemd property "Delegate=yes", see https://kind.sigs.k8s.io/docs/user/rootless/
Creating cluster "test" ...
 ✓ Ensuring node image (kindest/node:v1.21.1) 🖼 
 ✗ Preparing nodes 📦  
ERROR: failed to create cluster: podman run error: command "podman run --hostname test-control-plane --name test-control-plane --label io.x-k8s.kind.role=control-plane --privileged --tmpfs /tmp --tmpfs /run --volume 2f4404a838c9b263953667b6eeb2feb29a272ee282f36bdeac2925e6c7bc29f9:/var:suid,exec,dev --volume /lib/modules:/lib/modules:ro --detach --tty --net kind --label io.x-k8s.kind.cluster=test -e container=podman --publish=127.0.0.1:51695:6443/tcp -e KUBECONFIG=/etc/kubernetes/admin.conf kindest/node@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6" failed with error: exit status 126
Command Output: Error: error configuring network namespace for container 444fbb949aee1b431870e978c9ee022422de4aaf26e9e134a9bfae1e6fc0ba4c: error adding pod test-control-plane_test-control-plane to CNI network "kind": failed to list chains: running [/usr/sbin/ip6tables -t nat -S --wait]: exit status 3: modprobe: ERROR: could not insert 'ip6_tables': Operation not permitted
ip6tables v1.8.7 (legacy): can't initialize ip6tables table `nat': Table does not exist (do you need to insmod?)
Perhaps ip6tables or your kernel needs to be upgraded.

podman run --hostname test-control-plane --name test-control-plane --label io.x-k8s.kind.role=control-plane --privileged --tmpfs /tmp --tmpfs /run --volume 2f4404a838c9b263953667b6eeb2feb29a272ee282f36bdeac2925e6c7bc29f9:/var:suid,exec,dev --volume /lib/modules:/lib/modules:ro --detach --tty --net kind --label io.x-k8s.kind.cluster=test -e container=podman --publish=127.0.0.1:51695:6443/tcp -e KUBECONFIG=/etc/kubernetes/admin.conf kindest/node@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6

error configuring network namespace for container 444fbb949aee1b431870e978c9ee022422de4aaf26e9e134a9bfae1e6fc0ba4c:
error adding pod test-control-plane_test-control-plane to CNI network "kind":
failed to list chains: running [/usr/sbin/ip6tables -t nat -S --wait]: exit status 3:
modprobe: ERROR: could not insert 'ip6_tables':
Operation not permitted
ip6tables v1.8.7 (legacy):
can't initialize ip6tables table `nat':
Table does not exist (do you need to insmod?)
Perhaps ip6tables or your kernel needs to be upgraded.

containers/podman#8784

suzuki-shunsuke · 2021-09-11T07:52:41Z

minikube

ローカルで k8s 動かすなら minikube もあり。

minikube も複数クラスタに対応している。 https://minikube.sigs.k8s.io/docs/faq/#how-can-i-create-more-than-one-cluster-with-minikube

$ minikube start

😄  Darwin 11.5.2 上の minikube v1.23.0
✨  hyperkitドライバーが自動的に選択されました。他の選択肢:  virtualbox, ssh, podman (experimental)
💾  docker-machine-driver-hyperkit ドライバをダウンロードしています:
    > docker-machine-driver-hyper...: 65 B / 65 B [----------] 100.00% ? p/s 0s
    > docker-machine-driver-hyper...: 10.52 MiB / 10.52 MiB  100.00% 10.78 MiB 
🔑  The 'hyperkit' driver requires elevated permissions. The following commands will be executed:

    $ sudo chown root:wheel /Users/shunsuke-suzuki/.minikube/bin/docker-machine-driver-hyperkit 
    $ sudo chmod u+s /Users/shunsuke-suzuki/.minikube/bin/docker-machine-driver-hyperkit 


Password:
💿  VM ブートイメージをダウンロードしています...
    > minikube-v1.23.0.iso.sha256: 65 B / 65 B [-------------] 100.00% ? p/s 0s
    > minikube-v1.23.0.iso: 229.15 MiB / 229.15 MiB [ 100.00% 10.24 MiB p/s 23s
👍  コントロールプレーンのノード minikube を minikube 上で起動しています
💾  Kubernetes v1.22.1 のダウンロードの準備をしています
    > preloaded-images-k8s-v12-v1...: 515.04 MiB / 515.04 MiB  100.00% 10.38 Mi
🔥  hyperkit VM (CPUs=2, Memory=4000MB, Disk=20000MB) を作成しています...
🐳  Docker 20.10.8 で Kubernetes v1.22.1 を準備しています...
    ▪ 証明書と鍵を作成しています...
    ▪ Control Plane を起動しています...
    ▪ RBAC のルールを設定中です...
🔎  Kubernetes コンポーネントを検証しています...
    ▪ イメージ gcr.io/k8s-minikube/storage-provisioner:v5 を使用しています
🌟  有効なアドオン: storage-provisioner, default-storageclass
🏄  完了しました！ kubectl が「"minikube"」クラスタと「"default"」ネームスペースを使用するよう構成されました

Pod も普通に deploy できた

(⎈ |minikube:default)❯ pbpaste | k apply -f -

deployment.apps/nginx-deployment created

(⎈ |minikube:default)❯ k get po

NAME                                READY   STATUS              RESTARTS   AGE
nginx-deployment-66b6c48dd5-9gmkj   0/1     ContainerCreating   0          14s
nginx-deployment-66b6c48dd5-rc8lp   0/1     ContainerCreating   0          14s
nginx-deployment-66b6c48dd5-zbrmk   0/1     ContainerCreating   0          14s

suzuki-shunsuke · 2021-09-11T07:58:06Z

lima を試す

https://github.com/lima-vm/lima

$ brew install lima
$ limactl start

? Creating an instance "default" Proceed with the default configuration

lima で VM に SSH できる

$ lima

コンテナも起動できた。

$ lima nerdctl run -ti --rm alpine:3.14.2 sh

image build もできた

$ lima nerdctl build -t foo .

まぁコマンドは lima nerdctl より podman のほうが簡潔ではある。けどまぁそれは alias 貼ればいい話でもありそう。
lima は Compose をサポートしているので、 docker-compose.yaml を動かしたいときは便利そう。

$ lima nerdctl compose --help                            

NAME:
   nerdctl compose - Compose

USAGE:
   nerdctl compose command [command options] [arguments...]

COMMANDS:
   up       Create and start containers
   logs     View output from containers.
   build    Build or rebuild services
   down     Remove containers and associated resources
   help, h  Shows a list of commands or help for one command

OPTIONS:
   --file value, -f value          Specify an alternate compose file
   --project-directory value       Specify an alternate working directory
   --project-name value, -p value  Specify an alternate project name
   --env-file value                Specify an alternate environment file
   --help, -h                      show help (default: false)

docker-compose と比べて

exec
stop
ps

あたりがない気がする。

Docker Compose CLI: https://zenn.dev/skanehira/articles/2021-06-03-new-docker-compose

Docker Compose CLI にはあるので nerdctl compose の互換性の問題のような気がする

書いてあった https://github.com/containerd/nerdctl#unimplemented-docker-commands

lima nerdctl compose up -d でコンテナを起動しようとするも、すぐ終了してしまう。

$ lima nerdctl compose up -d
WARN[0000] The GITHUB_TOKEN variable is not set. Defaulting to a blank string. 

$ lima nerdctl ps -a        

CONTAINER ID    IMAGE                                     COMMAND      CREATED          STATUS                      PORTS    NAMES
946916f023c0    docker.io/library/tutorial_main:latest    "/bin/sh"    7 seconds ago    Exited (0) 7 seconds ago             tutorial_main_1

docker-compose.yaml で以下の設定をしているが、これだとだめなのかもしれない。

    tty: true
    stdin_open: true

代わりに tail -f /dev/null を command で実行するようにしたらいけた。

    command: ["tail", "-f", "/dev/null"]

しかし、環境変数がうまく渡っていない。
VM から host の環境変数が読めないからかな？

    environment:
    - "PATH=/root/.aqua/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    - GITHUB_TOKEN # 空になる

suzuki-shunsuke · 2022-07-24T22:50:50Z

「マルチCPUアーキテクチャ」に対応したイメージをビルド

Docker

Docker の場合、 buildx を使うことで実現可能

buildah

buildah でも可能っぽい？

Supporting building multi-platform images (podman buildx) containers/buildah#1590

nerdctl

nerdctl もサポートしている？

https://github.com/containerd/nerdctl#features-present-in-nerdctl-but-not-present-in-docker

Better multi-platform support, e.g., nerdctl pull --all-platforms IMAGE

この例は pull だけど、 build もサポートしているんだろうか？

$ lima nerdctl build --help
  ...
  --platform strings         Set target platform for build (e.g., "amd64", "arm64")

サポートしてそう

podman

$ podman build --help
  --all-platforms                                attempt to build for all base image platforms
  --arch string                                  set the ARCH of the image to the provided value instead of the architecture of the host (default "arm64")
  --os string                                    set the OS to the provided value instead of the current operating system of the host (default "darwin")
  --platform linux/arm                           set the OS/ARCH/VARIANT of the image to the provided value instead of the current operating system and architecture of the host (for example linux/arm) (default [darwin/arm64/v8])

suzuki-shunsuke added study container labels Sep 11, 2021

This comment has been minimized.

Sign in to view

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker Desktop 代替を検証する #67

Docker Desktop 代替を検証する #67

suzuki-shunsuke commented Sep 11, 2021 •

edited

suzuki-shunsuke commented Sep 11, 2021 •

edited

suzuki-shunsuke commented Sep 11, 2021

suzuki-shunsuke commented Sep 11, 2021 •

edited

suzuki-shunsuke commented Sep 11, 2021 •

edited

suzuki-shunsuke commented Sep 11, 2021 •

edited

suzuki-shunsuke commented Sep 11, 2021 •

edited

suzuki-shunsuke commented Sep 11, 2021 •

edited

suzuki-shunsuke commented Sep 11, 2021 •

edited

suzuki-shunsuke commented Sep 11, 2021 •

edited

suzuki-shunsuke commented Sep 11, 2021 •

edited

This comment has been minimized.

suzuki-shunsuke commented Jul 24, 2022 •

edited

Docker Desktop 代替を検証する #67

Docker Desktop 代替を検証する #67

Comments

suzuki-shunsuke commented Sep 11, 2021 • edited

suzuki-shunsuke commented Sep 11, 2021 • edited

suzuki-shunsuke commented Sep 11, 2021

suzuki-shunsuke commented Sep 11, 2021 • edited

Podman

suzuki-shunsuke commented Sep 11, 2021 • edited

やりたいこと

suzuki-shunsuke commented Sep 11, 2021 • edited

suzuki-shunsuke commented Sep 11, 2021 • edited

podman で kind を動かす

suzuki-shunsuke commented Sep 11, 2021 • edited

Podman では Docker image の short name の扱いが Docker と違う

suzuki-shunsuke commented Sep 11, 2021 • edited

suzuki-shunsuke commented Sep 11, 2021 • edited

minikube

suzuki-shunsuke commented Sep 11, 2021 • edited

lima を試す

This comment has been minimized.

suzuki-shunsuke commented Jul 24, 2022 • edited

「マルチCPUアーキテクチャ」に対応したイメージをビルド

Docker

buildah

nerdctl

podman

suzuki-shunsuke commented Sep 11, 2021 •

edited

suzuki-shunsuke commented Sep 11, 2021 •

edited

suzuki-shunsuke commented Sep 11, 2021 •

edited

suzuki-shunsuke commented Sep 11, 2021 •

edited

suzuki-shunsuke commented Sep 11, 2021 •

edited

suzuki-shunsuke commented Sep 11, 2021 •

edited

suzuki-shunsuke commented Sep 11, 2021 •

edited

suzuki-shunsuke commented Sep 11, 2021 •

edited

suzuki-shunsuke commented Sep 11, 2021 •

edited

suzuki-shunsuke commented Sep 11, 2021 •

edited

suzuki-shunsuke commented Jul 24, 2022 •

edited