Support zstd-compressed docker images

[g4-otayyan]

Would you consider supporting docker images compressed using zstd, docker v25 already supports it but for some reason (I am guessing containerd) they aren't supported on bottlerocket and cannot be pulled.

[yeazelm]

Thanks @g4-otayyan for cutting this issue. containerd should already have support and this might just be an issue with docker configuration. I'm working on confirming we see the same failure to pull, but in the mean time can you provide some logs that showed it failed to pull? Can you confirm which image you are using (aws-ecs-2, aws-k8s-1.29, etc) and where you are hosting the containers just to make sure we are looking at the same issue?

[yeazelm]

I was able to launch a container compressed with zstd both on an ECS and K8s Bottlerocket node.

$ docker buildx imagetools \
    inspect --raw xxxxxxx.dkr.ecr.us-west-2.amazonaws.com/openai-zstd:latest
{
   "mediaType": "application/vnd.oci.image.manifest.v1+json",
   "schemaVersion": 2,
   "config": {
      "mediaType": "application/vnd.oci.image.config.v1+json",
      "digest": "sha256:c2024fb4633e7afbd539120e14858ee0eba9432dd638edd283777bd5e9f9e82f",
      "size": 8007
   },
   "layers": [
      {
         "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
         "digest": "sha256:70da26ed5fce52ad4694154b867025c165f3aed37f4cf545335a1235e8a6a28e",
         "size": 23944047
      },
      {
         "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
         "digest": "sha256:6ca97bcc9a7b671aa544cfbf31450af64373002beb0b8182cbd4910a5a90b1e6",
         "size": 3306189
      },
      {
         "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
         "digest": "sha256:95d577cb9f9748f1e365a5b519dcdcd79622f31f646b67b20ecaa7e4cc19165a",
         "size": 11961355
      },
      {
         "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
         "digest": "sha256:189fc3f6aad3b836ebded8e2bf3fb6a411e019c3be6c34434679136f44a6cf8d",
         "size": 220
      },
      {
         "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
         "digest": "sha256:e827925d59450c588ed8d80f7d6bad3dd127b73907ab2423db2f2b62c93fc12d",
         "size": 2641900
      },
      {
         "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
         "digest": "sha256:8ee8de0e5f0dbcf6c96bdfab23b1204db866f8435c66c033e25169be4457d2b8",
         "size": 40192381
      },
      {
         "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
         "digest": "sha256:f97f6e46221bdab6e8d0f0e74cad080567528a203625d61eb67ece43e7a4d9f5",
         "size": 226266438
      }
   ]

ECS from the host

bash-5.1# docker ps
CONTAINER ID   IMAGE                                                             COMMAND                  CREATED         STATUS              PORTS     NAMES
55694a74ad8b   xxxxx.dkr.ecr.us-west-2.amazonaws.com/openai-zstd:latest   "sh -c 'sleep infini…"   2 minutes ago   Up About a minute             ecs-openai-zstd-1-openai-zstd-f4a6f79eea9397803b00

@g4-otayyan it might be useful to see the logs from the instance since I think it is working right now.

[g4-otayyan]

@yeazelm I am using the latest version of the aws-ecs-2 AMI taken directly from /aws/service/bottlerocket/aws-ecs-2/arm64/latest/image_id

The use case I have is that I need to create a pre-baked AMI with few docker images cached in it, the I was able to achieve this with the following code

for i in "${IMAGES_TO_PULL[@]}"; do
    echo "pulling: $i"
    apiclient exec admin bash -c "sheltie ctr -n moby image pull --user 'AWS:${DOCKER_ECR_TEMP_PASSWORD}' $i"
    apiclient exec admin bash -c "sheltie ctr -n moby image export /local/img.tar $i"
    apiclient exec admin bash -c "sheltie ctr -n moby image rm $i"
    apiclient exec admin bash -c "sheltie docker image load -i /local/img.tar"
done

The reason I cannot use docker pull directly is because bottlerocket doesn't allow docker login to run in this context - which I need for my private repo - as it raises a Error saving credentials: mkdir /root/.docker: read-only file system error.

I checked the version of containerd used in the project and it's 1.6.x which doesn't support zstd compressed images, but the latest versions of containerd do.

EDIT: fixed the reason docker pull cannot be used

[yeazelm]

I checked the version of containerd used in the project and it's 1.6.x which doesn't support zstd compressed images, but the latest versions of containerd do

1.6.x does have zstd support, it was backported a while ago to older releases: https://github.com/containerd/containerd/releases/tag/v1.5.0. And in fact I was using 1.19.2 and aws-ecs-2 as well for my test, so I can confirm that the normal process of pulling zstd containers does work in this setup.

As for the series of commands, I'm not sure if this approach is expected to work. We typically recommend users launch ECS tasks on an host to get the pulls to complete normally, then snapshot from that. It might still be worth digging into why this seems to not work via your pattern. Do you have logs to show where this fails?

[g4-otayyan]

My apologies, in the latest bottlerocket version these are the logs that result out of calling apiclient exec admin bash -c "sheltie ctr -n moby image pull --user 'AWS:${DOCKER_ECR_TEMP_PASSWORD}' $i" on a zstd compressed image:

time="2024-03-06T20:31:22Z" level=warning msg="reference for unknown type: application/vnd.docker.image.rootfs.diff.tar.zstd"
time="2024-03-06T20:31:22Z" level=warning msg="reference for unknown type: application/vnd.docker.image.rootfs.diff.tar.zstd"
time="2024-03-06T20:31:22Z" level=warning msg="reference for unknown type: application/vnd.docker.image.rootfs.diff.tar.zstd"
time="2024-03-06T20:31:22Z" level=warning msg="reference for unknown type: application/vnd.docker.image.rootfs.diff.tar.zstd"
time="2024-03-06T20:31:22Z" level=warning msg="reference for unknown type: application/vnd.docker.image.rootfs.diff.tar.zstd"
time="2024-03-06T20:31:22Z" level=warning msg="reference for unknown type: application/vnd.docker.image.rootfs.diff.tar.zstd"
time="2024-03-06T20:31:22Z" level=warning msg="reference for unknown type: application/vnd.docker.image.rootfs.diff.tar.zstd"
time="2024-03-06T20:31:22Z" level=warning msg="reference for unknown type: application/vnd.docker.image.rootfs.diff.tar.zstd"
time="2024-03-06T20:31:22Z" level=warning msg="reference for unknown type: application/vnd.docker.image.rootfs.diff.tar.zstd"
time="2024-03-06T20:31:22Z" level=warning msg="reference for unknown type: application/vnd.docker.image.rootfs.diff.tar.zstd"
time="2024-03-06T20:31:22Z" level=warning msg="reference for unknown type: application/vnd.docker.image.rootfs.diff.tar.zstd"
time="2024-03-06T20:31:22Z" level=warning msg="reference for unknown type: application/vnd.docker.image.rootfs.diff.tar.zstd"
time="2024-03-06T20:31:22Z" level=warning msg="reference for unknown type: application/vnd.docker.image.rootfs.diff.tar.zstd"
x.dkr.ecr.x.amazonaws.com/x:x: resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:bd2cc1d7877ebc1414aab51d21a951ea3a4a14acc3e00f1ab5fd5421c2d0c38a:           done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:1d682a82b32cad30f4a8b73d0f66a7c18db6cfcc1174115f9293302dbc2cf49c:        done           |++++++++++++++++++++++++++++++++++++++|
unknown-sha256:11c6b967d91d8f0368ec7ad64026df6a3998b64a8160bba0220ba7b64fe429a3:         done           |++++++++++++++++++++++++++++++++++++++|
unknown-sha256:964c971e724254153b0c0416376d984c910b19de50047c7a7b88cefad2505f27:         done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:e0eb58eea5522872205ef2fe24d94e2f9537d868fde6f0cb1c0be94b6bd0f76e:          done           |++++++++++++++++++++++++++++++++++++++|
unknown-sha256:4c3e6c4b7b998a58a0b55f47bf704d8da6e6e2aa1dcd66a755edd385bbd0ce29:         done           |++++++++++++++++++++++++++++++++++++++|
unknown-sha256:7121ad798546b898cd1ce9ea799d02852c4434cced81dd4f5ca565d262a416f2:         done           |++++++++++++++++++++++++++++++++++++++|
unknown-sha256:9ae9d6828c71fe034c1a0747a2c6580739ab00b84c590e37899ad07a24e73ddb:         done           |++++++++++++++++++++++++++++++++++++++|
unknown-sha256:93e5719156f0a876f129f3ee626d80a9420ef9056503404abdd57e772e115e85:         done           |++++++++++++++++++++++++++++++++++++++|
unknown-sha256:c3cf754191dcaf0ea966cecf4cb794c0c4582164d9bf191a76810bc46625e1a4:         done           |++++++++++++++++++++++++++++++++++++++|
unknown-sha256:a4179a9859a0df9c349ed7f0729a5d8042b75e9e56d2a293004cab52e803fb5b:         done           |++++++++++++++++++++++++++++++++++++++|
unknown-sha256:8f35caee9685f6316b1a1674265ef603ea318fa597d1a9a6c47191a7c553b3f9:         done           |++++++++++++++++++++++++++++++++++++++|
unknown-sha256:ab1435a78319c6a1296266c286dc842908adf3c748875d13f696a977fc14ad0d:         done           |++++++++++++++++++++++++++++++++++++++|
unknown-sha256:176adfaabeeeab89f4577dcf4c81775beba37dc13f98c010844480381c8794ab:         done           |++++++++++++++++++++++++++++++++++++++|
unknown-sha256:9d48b3f86785929da90f57918147e12c16a38e442e7453aeab78243c513f8613:         done           |++++++++++++++++++++++++++++++++++++++|
unknown-sha256:ba14d2ff578bcdce5845a40a14921c98e54eed53ff42072d388f296ab0b26735:         done           |++++++++++++++++++++++++++++++++++++++|
unpacking linux/amd64 sha256:bd2cc1d7877ebc1414aab51d21a951ea3a4a14acc3e00f1ab5fd5421c2d0c38a...
ctr: mismatched image rootfs and manifest layers

[g4-otayyan]

Perhaps the issue is with the manifest of the multi-arch image and not zstd compression?

[g4-otayyan]

I actually managed to make it work, here's a code snippet if anyone needs this later:

#!/bin/bash
set -euo pipefail

ROOT_URL=x.dkr.ecr.x.amazonaws.com

declare -a IMAGES_TO_PULL=(
    "$ROOT_URL/x:x"
    "$ROOT_URL/xx:xx"
)

apiclient exec admin bash -c "sheltie mkdir -p /tmp/docker/"
apiclient exec admin bash -c "sheltie docker --config /tmp/docker/ login --username AWS --password ${DOCKER_ECR_TEMP_PASSWORD} ${ROOT_URL}"

for i in "${IMAGES_TO_PULL[@]}"; do
    echo "pulling: $i"
    apiclient exec admin bash -c "sheltie docker --config /tmp/docker/ pull $i"
done

apiclient exec admin bash -c "sheltie rm -rf /tmp/docker/"

[yeazelm]

Thanks @g4-otayyan for the update! Glad you figured this out. I'll resolve this issue then.