Timeout errors trying to use polling

[ohshitgorillas]

I am new to Kubernetes, running a simple two-node bare metal cluster on my LAN and I want to use Keel to enable auto-updates for some pods.

I am testing Keel auto-updates by applying the following annotations to a few containers that I am positive have updates:

metadata:
  annotations:
    keel.sh/policy: all
    keel.sh/trigger: poll
    keel.sh/pollInterval: "@every 1m"

Using @every 1m is just for testing purposes, eventually I'll switch it to @hourly once it works.

This generates the following errors in the keel logs:

time="2024-04-06T17:58:05Z" level=error msg="trigger.poll.RepositoryWatcher.Watch: failed to add image watch job" error="Head \"https://index.docker.io/v2/linuxserver/jellyfin/manifests/latest\": dial tcp: lookup index.docker.io: i/o timeout" image="namespace:default,image:index.docker.io/linuxserver/jellyfin:latest,provider:kubernetes,trigger:poll,sched:@every 1m,secrets:[]"
time="2024-04-06T17:58:15Z" level=error msg="trigger.poll.manager: got error(-s) while watching images" error="encountered errors while adding images: Head \"https://index.docker.io/v2/linuxserver/jellyfin/manifests/latest\": dial tcp: lookup index.docker.io: i/o timeout, Head \"https://index.docker.io/v2/linuxserver/netbootxyz/manifests/latest\": dial tcp: lookup index.docker.io: i/o timeout"
time="2024-04-06T17:58:25Z" level=error msg="trigger.poll.RepositoryWatcher.addJob: failed to get image digest" error="Head \"https://index.docker.io/v2/linuxserver/jellyfin/manifests/latest\": dial tcp: lookup index.docker.io: i/o timeout" image="linuxserver/jellyfin:latest" password= username=

I can confirm that the pods can reach index.docker.io:

-> % kubectl exec -n kube-system keel-6f564fd6c-r5h9k -it -- nslookup index.docker.io 
Server:		10.96.0.10
Address:	10.96.0.10:53

Non-authoritative answer:
Name:	index.docker.io
Address: 2600:1f18:2148:bc00:8d61:9b62:40aa:8bb8
Name:	index.docker.io
Address: 2600:1f18:2148:bc01:571f:e759:a87a:2961
Name:	index.docker.io
Address: 2600:1f18:2148:bc02:445d:9ace:d20b:c303

Non-authoritative answer:
Name:	index.docker.io
Address: 54.236.113.205
Name:	index.docker.io
Address: 54.198.86.24
Name:	index.docker.io
Address: 54.227.20.253

Keel was deployed using helm with the following values.yaml file (this is a local cluster and not accessible outside of my LAN):

basicAuth:
  username: admin
  password: admin

Not sure where to go from here, any help or advice would be appreciated!

[ohshitgorillas]

This turned out to be related to internal DNS errors. Never figured out the cause, but I deleted and rebuilt the entire cluster and now it seems to work fine so far.