Skip to content

Building images on AKS worker nodes with k8s version = 1.19 (containerd CRI) using Docker buildx

Jump to bottom
AdamSharif-MSFT edited this page Jul 29, 2021 · 2 revisions

If you try to simply run docker buildx on the worker nodes or in a container running on a node, you'll receive an error that you cannot connect to Docker daemon:

/testing # docker buildx build .

WARN[0000] No output specified for docker-container driver. Build
result will only remain in the build cache. To push result image into
registry use --push or to load image into docker use --load

[+] Building 0.0s (0/0)

error: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Understanding the problem

Docker buildx expects to be able to connect to the Docker daemon socket, therefore you need to specify a Kubernetes driver to create a buildkit pod to build the container image inside the cluster.

Steps to run a Docker pod to perform local image building

  1. Create a Service Account for the Docker pod:

    kubectl create serviceaccount buildx-sa

  2. Create a clusterrole/clusterrolebinding for the service account:

    kubectl create clusterrole buildx-role --verb create,get --resource deployment --verb list --resource pod --verb create --resource pods/exec
kubectl create clusterrolebinding buildx-rolebinding --serviceaccount default:buildx-sa --clusterrole buildx-role

  3. Run a Docker pod using the created service account, download/install buildx plugin, rename and place into plugin directory:

    cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: buildx-pod
  name: buildx-pod
spec:
  containers:
  - image: docker:latest
    name: buildx-pod
    command:
      - "sh"
      - "-c"
      - |
        mkdir ~/.docker
        mkdir ~/.docker/cli-plugins && cd ~/.docker/cli-plugins
        wget https://github.com/docker/buildx/releases/download/v0.5.1/buildx-v0.5.1.linux-amd64
        mv ./buildx-v0.5.1.linux-amd64 docker-buildx
        chmod +x ./docker-buildx
        docker buildx create --use --name k8s node-amd64 --driver kubernetes --driver-opt  image=moby/buildkit:master
        while true; do sleep 3600; done;
  restartPolicy: Never
  serviceAccount: buildx-sa
EOF

    Note: make sure you choose the latest build of buildx from here that suits your architecture: https://github.com/docker/buildx/releases

  4. Exec into buildx pod:

    kubectl exec buildx-pod -it -- sh

  5. Test

    mkdir testing && cd testing
/testing # echo "FROM nginx" > Dockerfile
/testing # docker buildx build .

  6. A deployment "k8s" gets created and a pod is spun-up to do the build:

    mkdir testing && cd testing
/testing # echo "FROM nginx" > Dockerfile
/testing # docker buildx build .
	WARN[0000] No output specified for kubernetes driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load

	[+] Building 2.3s (4/4) FINISHED
	 => [internal] load build definition from Dockerfile                                                                                                                                    0.2s
	 => => transferring dockerfile: 48B                                                                                                                                                     0.0s
	 => [internal] load .dockerignore                                                                                                                                                       0.2s
	 => => transferring context: 2B                                                                                                                                                         0.0s
	 => [internal] load metadata for docker.io/library/nginx:latest                                                                                                                         1.6s
	 => [1/1] FROM docker.io/library/nginx@sha256:75a55d33ecc73c2a242450a9f1cc858499d468f077ea942867e662c247b5e412                                                                          0.5s
         => => resolve docker.io/library/nginx@sha256:75a55d33ecc73c2a242450a9f1cc858499d468f077ea942867e662c247b5e412                                                                          0.0s

  7. This just creates the image in the build cache, but to push the result image into a registry you need to use --push or --load to load into Docker

Resources:

Example push command:

/testing # docker login adamsharif.azurecr.io -u adamsharif -p [redacted]

WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded

/testing # docker buildx build . --push -t adamsharif.azurecr.io/adsharif-nginx:latest

[+] Building 1.1s (5/5) FINISHED
 => [internal] load build definition from Dockerfile                                                                    0.1s
 => => transferring dockerfile: 31B                                                                                     0.0s
 => [internal] load .dockerignore                                                                                       0.1s
 => => transferring context: 2B                                                                                         0.0s
 => [internal] load metadata for docker.io/library/nginx:latest                                                         0.4s
 => CACHED [1/1] FROM docker.io/library/nginx@sha256:75a55d33ecc73c2a242450a9f1cc858499d468f077ea942867e662c247b5e412   0.0s
 => => resolve docker.io/library/nginx@sha256:75a55d33ecc73c2a242450a9f1cc858499d468f077ea942867e662c247b5e412          0.1s
 => exporting to image                                                                                                  0.5s
 => => exporting layers                                                                                                 0.0s
 => => exporting manifest sha256:2db0664b0c146dde37559c8f1e47a6e18f0739f2174233db30fe5f9afbd2ad73                       0.0s
 => => exporting config sha256:2a72caafba65f57dafa86161487c79ba45853a6c8ca3e210e60c3da6bab8b7bf                         0.0s
 => => pushing layers                                                                                                   0.4s
 => => pushing manifest for adamsharif.azurecr.io/adsharif-nginx:latest@sha256:2db0664b0c146dde37559c8f1e47a6e18f0739f  0.1s

Simplified method for creating a Docker image to execute the build process from:

FROM docker
COPY --from=docker/buildx-bin:latest /buildx /usr/libexec/docker/cli-plugins/docker-buildx
RUN sleep infinity

Push to your registry of choice and use this in a deployment:

cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: buildxdemo
  name: buildxdemo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: buildxdemo
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: buildxdemo
    spec:
      containers:
      - image: adamsharif.azurecr.io/dockerbuildxdemo:v3
        name: dockerbuildxdemo
        resources: {}
      serviceAccount: buildx-sa
EOF
Clone this wiki locally