"should" -> "must" regarding redis eviction policy

[dpedu]

I run an instance of Distribution that is used to store images as well as a build cache for docker image builds e.g. docker buildx build --cache-from type=registry,ref=my-registry.domain.com/repo:tag ...

Upon upgrading from registry 2.8.1 to 3.0.0, the above command could no longer push layers to Distribution to be cached (note that this has been anonymized):

[2025-11-18T23:02:10.105Z] #26 writing layer sha256:cac89e58741c64e6ecfe0a6bf61f5c13071b13e73ef55fd98f28ef0bd751857b
[2025-11-18T23:02:10.105Z] #26 writing layer sha256:cac89e58741c64e6ecfe0a6bf61f5c13071b13e73ef55fd98f28ef0bd751857b 0.2s done
[2025-11-18T23:02:10.105Z] #26 writing config sha256:7e8345978b1195f33f1041c349d3c566487234a2cc7e356aef520963bbaa8bdb
[2025-11-18T23:02:10.401Z] #26 writing config sha256:7e8345978b1195f33f1041c349d3c566487234a2cc7e356aef520963bbaa8bdb 0.2s done
[2025-11-18T23:02:10.401Z] #26 writing cache manifest sha256:222228409d93939918fc83fd653ee2d3ded4f393c2db97817c566b034f8ae9d4
[2025-11-18T23:02:10.401Z] #26 preparing build cache for export 2.3s done
[2025-11-18T23:02:10.401Z] #26 writing cache manifest sha256:222228409d93939918fc83fd653ee2d3ded4f393c2db97817c566b034f8ae9d4 0.1s done
[2025-11-18T23:02:10.401Z] #26 ERROR: error writing manifest blob: failed commit on ref "sha256:222228409d93939918fc83fd653ee2d3ded4f393c2db97817c566b034f8ae9d4": unexpected status from PUT request to https://my-registry.domain.com/v2/repo/manifests/tag: 400 Bad Request

And Distribution logs the following (also anonymized):

time="2025-11-18T23:12:11.06652883Z"
level=error 
msg="error from cache setting desc"
blob="sha256:da4864f7969f5e3fb82412d177706d0d77fe2025c4789a5611431b2b9f6a19ea"
error="OOM command not allowed when used memory > 'maxmemory'."
go.version=go1.23.7 
http.request.host=my-registry.domain.com 
http.request.id=aeed02d7-d1dc-4925-87f6-ba6863f5a9e2 
http.request.method=GET 
http.request.remoteaddr=89.123.45.67 
http.request.uri="/v2/repo/blobs/sha256:da4864f7969f5e3fb82412d177706d0d77fe2025c4789a5611431b2b9f6a19ea"
http.request.useragent="docker/27.1.2 go/go1.21.13 git-commit/f9522e5 kernel/6.8.0-1014-aws os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)"
instance.id=5face511-0db3-4eca-a4eb-2857ee5e8833 
service=registry 
vars.digest="sha256:da4864f7969f5e3fb82412d177706d0d77fe2025c4789a5611431b2b9f6a19ea"
vars.name=repo
version=3.0.0

The OOM command not allowed when used memory > 'maxmemory'. message is logged when attempting to write to a full redis instance:

$ nc -vv localhost 6379
Connection to localhost (127.0.0.1) 6379 port [tcp/redis] succeeded!
AUTH abc123
+OK
SET foo bar
-OOM command not allowed when used memory > 'maxmemory'.

And upon rolling back to distribution 2.8.1, uploading cache items works again. Without clearing redis. Likewise, I was able to push cache objects to a registry 3.0.0 instance with a healthy redis backing it.

It appears to me that 3.x returns an error to the client upon encountering issues with redis, but 2.x did not. Or 2.x did not involve the cache on this path. This is fine, but the documentation should be updated to note that this is critical, not optional.

I am using redis via Amazon's Elasticache service where the default maxmemory-policy is volatile-lru. This policy only evicts keys that have the expire field set to true. Distribution does not set an expiration value on keys, so this configuration will eventually lead to redis filling up and distribution returning errors.

I am not sure if regular image pushing is affected as well, I have only tested caching.

[milosgajdos]

Please sign your commit

[dpedu]

@milosgajdos "This commit was created on GitHub.com and signed with GitHub’s verified signature."

Sounds like it is signed. Let me know what needs to be done.

[milosgajdos]

Doesnt seem to be, no https://github.com/distribution/distribution/pull/4742/checks?check_run_id=55849257265

See: https://github.com/distribution/distribution/blob/main/CONTRIBUTING.md#sign-your-work

Maybe GH is just bad at this

[dpedu]

@milosgajdos ah, I misunderstood what kind of signature. I have updated the commit. Thank you!

[milosgajdos]

LGTM

[wy65701436]

lgtm