Spegel Mirroring not working on AWS EKS 1.33

[ITAdmin2026]

Description

Spegel mirrors are not being used for image pulls. Containerd falls back to the upstream registry (ECR) even when the image exists in the local Spegel cache on another node.

Environment

• Spegel version: v0.6.0 (also tested v0.5.1)
• Kubernetes distribution: EKS
• Kubernetes version: 1.33 (AL2023 AMI)
• containerd version: v2.1.4
• CNI: Amazon VPC CNI

What we were trying

Deploy Spegel to enable P2P image distribution in an EKS cluster. When a pod is scheduled on a new node, it should pull images from another node's local Spegel cache instead of ECR. The size of the image is near to 6G.

Expected behavior

1. Node A pulls image from ECR, Spegel caches it
2. Node B schedules a pod needing the same image
3. Containerd on Node B queries local Spegel mirror
4. Spegel finds the image via DHT lookup on Node A
5. Image is served from Node A's cache in seconds

What actually happened

1. Node A pulls image from ECR, Spegel caches it ✓
2. Node B schedules a pod needing the same image ✓
3. Containerd on Node B queries local Spegel mirror
4. Image is not found.
5. Containerd falls back to ECR, taking ~3 minutes instead of seconds

Evidence

Spegel logs confirm image is tracked and advertised:
{"time":"2026-02-20T10:16:14.930673268Z","level":"INFO","msg":"OCI event","ref":"316971283637.dkr.ecr.us-east-1.amazonaws.com/fury:mlflow_staging-260216-1210-f72d444ad","type":"CREATE"}

[phillebaba]

It is hard to tell what is not working without knowing much more about your setup. Did you follow the guide to configure your EKS nodes properly?

[ITAdmin2026]

closed the issue accidentally, hence reopening it.

1. we are running on EKS 1.33 and use karpenter to provision node. All workloads run on spot, expect for databases where on demand nodes are used.
2. we tested Spegel on selected subset of nodes.
3. We added below property to karpenter nodeclass under userdata.

    MIME-Version: 1.0
    Content-Type: multipart/mixed; boundary="BOUNDARY"

    --BOUNDARY
    Content-Type: application/node.eks.aws

    ---
    apiVersion: node.eks.aws/v1alpha1
    kind: NodeConfig
    spec:
      containerd:
        config: |
          [plugins."io.containerd.cri.v1.images".registry]
            config_path = "/etc/containerd/certs.d"
          [plugins.'io.containerd.cri.v1.images']
            discard_unpacked_layers = false

    --BOUNDARY--

4. Checked all the nodes provisioned and all of them this this enabled, verified this by running containerd config dump.
5. Deployed a service to run only on those restricting using affinity and tolerations. The size of docker image here is 6G compressed.
6. Started by setting the replicas count to 1. Takes 4 minutes 30 second to pull the image.
7. Increase the replicas count to 2. See a race condition where service pods start pulling image before Spegel pod on the node is running.
8. fix issue 7 by adding an init container to check for readiness of Spegel pods before starting the main container.
9. After 8. The service main container starts after Spegel pod on the node transitions to running state.
10. The image is still pulled from ECR, since the pull time is near to 3m57s.
11. Think 10 could be because the active peers are very few. Scale replicas to 3 at the same time. Every service pull the image when they run for the first time on a node.
12. We have 3 active peer. Scale up the deployment to 4 replicas, expect Spegel would work. Again it take 3m 50s to bring up the pod on the node.
13. However when you port-forward and access /web/debug and try to test mirroring. The time to pull image comes to 35 seconds. But image pull via containerd does not seem to be functional here.

Please let me know if you need any more details. Happy to share.

[2del4u]

I have the same issue, but I found that it only happens with the ECR registry. The Docker registry works fine, though

[2del4u]

Hello @phillebaba , any thoughts/updates about this issue?
Thank you in advance.

[phillebaba]

Thank you for the ping on this.
Currently, I am focusing on Spegel bugs and features on the homelab/individual consumer use case. Also, because this project is maintained and free to use, I don't have funds (and time, sadly) to spend on spinning up an EKS cluster to debug this issue at this time. Hopefully this changes in the near future.
However, if community members wants to work together to solve this issues and submit PRs, I am happy to take a look.

[royra]

Confirmed working on EKS 1.35 with this NodeConfig:

kind: NodeConfig
   spec:
       containerd:
         config: |
           [plugins."io.containerd.grpc.v1.cri".containerd]
           discard_unpacked_layers = false
           [plugins."io.containerd.cri.v1.images".registry]
           config_path = '/etc/containerd/certs.d'

The issue reporter's fix likely failed because they put config_path under [plugins."io.containerd.grpc.v1.cri".containerd] (same section as discard_unpacked_layers) instead of the separate [plugins."io.containerd.cri.v1.images".registry] section — different TOML path.

[dampcake]

@royra the docs currently show using:

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  containerd:
    config: |
      [plugins."io.containerd.cri.v1.images".registry]
        config_path = "/etc/containerd/certs.d"
      [plugins.'io.containerd.cri.v1.images']
        discard_unpacked_layers = false

This seems to match the containerd configs for v2 (https://github.com/containerd/containerd/blob/main/docs/cri/config.md#full-configuration). I believe your example is showing a mix of containerd 1.x and 2.x configurations.

[royra]

@dampcake you're right. updated to v2 paths only (as the docs show) and it's still working on EKS 1.35 / containerd 2.1.5.

[shahmirwahid-bolt]

I can also confirm that it's working on EKS 1.34 with the following configuration changes only:

---
    apiVersion: node.eks.aws/v1alpha1
    kind: NodeConfig
    spec:
      containerd:
        config: |
          [plugins.'io.containerd.cri.v1.images']
            discard_unpacked_layers = false

          [plugins."io.containerd.cri.v1.images".registry]
            config_path = "/etc/containerd/certs.d"`

[2del4u]

Hi everyone and thank you for you response.
QQ: @shahmirwahid-bolt @royra did you check image distribution from private ECR repos and everything works , correct?
Thank you

[ITAdmin2026]

thanks everyone for all the updates. This configuration was not working for me earlier, however we just moved EKS to 1.34. I'll try and share the outcome.

[phillebaba]

Could it be that this is the same issue described in this discussion?
#1294