tag resolution TTL

[clementnuss]

I've been wrapping my head around this for the entire week: I'd like a middle ground between "tags are only resolved once" and "containerd resolves all tags" (aka the resolveFlags=flase case).

the use case is ominous: rolling tags for images such as alpine:3.24.

I know there are ways to exclude/filter some tags with regexp, but that doesn't cut it for cases where upstream, as a convenience, the tag is rolling and updated from times to times.

(this discussion is not a debate about rolling tag vs precise digests; rolling tags are a fact we need to cope with)

the feature request is something like a tag resolution TTL; with a 24h example, if any node in the cluster pulls at time t, for the next 24h the upstream's reply (say v1) will be the valid one in the cluster. after the ttl explires, a 404 should be returned by Spegel so that containerd fetches again from the upstream and potentially resolves a new v2 or the old v1, and the TTL is renewed.

while it sounds simple, I didn't manage so far to implement it: distributing and sharing an expiration timestamp for each tag resolution and having each node respect it isn't trivial.

I'm now at the point where I would consider (mis-)using the DHT to store a tag->timestamp attribute, so the TTL is known and shared by all nodes.

with that approach, when resolving a tag, we would check whether that tag is available and the TTL is valid in the DHT. if invalid, spegel would return a 404 so containerd would proceed to resolve the tag, and if valid we would use it.

because it's quite a large change, I wanted to discuss it first before submitting a PR, so here we are. it would be extremely beneficial for many environments where rolling tags are (even sometimes indirectly) being used.

[phillebaba]

Tag resolution has always been that one tricky part, even without Spegel. Your solution seems novel, and on the surface simple to implement. When I thought about things some more there are a few complications in implementing this, mostly because Spegel is a distributed system. I am not really sure how it would work, as records are advertised before they are expired and there is no way of determining when the last time a tag was re-resolved.

I personally think that in Kubernetes resolving tags to digests as a webhook is the most optimal solution. Have you checked out the k8s-digester documentation for resolving tags?
https://spegel.dev/docs/usage/resolving-tags/#k8s-digester

The only downside with the k8s-digester is that it creates a hard dependency on the upstream registry. This is also why I added my own implementation in the kvick-operator that is able to implement different strategies like falling back to the latest resolved digest instead.
https://spegel.dev/docs/usage/resolving-tags/#kvick-operator