Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spegel seems not to be working in EKS #503

Closed
Versa78 opened this issue May 29, 2024 · 8 comments
Closed

Spegel seems not to be working in EKS #503

Versa78 opened this issue May 29, 2024 · 8 comments
Labels
bug Something isn't working

Comments

@Versa78
Copy link

Versa78 commented May 29, 2024

Spegel version

0.0.22

Kubernetes distribution

EKS

Kubernetes version

1.28

CNI

Calico

Describe the bug

We are in the process of upgrading our clusters to 1.28 and in doing so we are also upgrading spegel from 0.0.14 to 0.0.22 and the nodes from 1.27.9-20240117 to 1.28.8-20240514. We previously encountered #350 so we are aware of the EKS specific settings now needed which is documented at https://github.com/spegel-org/spegel/blob/main/docs/COMPATIBILITY.md. We modified our current userdata and added the suggested EKS settings.

After trying this on a test cluster it seems spegel is not working, although I am not sure. Most obvious sign is that after draining some nodes and starting new ones I start to get issues with pods not starting because we have exceeded our pull limits to docker.io.

Here is some logs pulled from one of the spegel pods.

{"time":"2024-05-28T09:05:45.014335574Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/bitnami/nginx/blobs/sha256:100cc416bdcdeda1e010dc0408abfbdb58b04b8a6185871d5ea923574cf1dbec","status":200,"method":"GET","latency":"2.912132ms","ip":"10.100.217.197"}
{"time":"2024-05-28T09:05:45.079354738Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/bitnami/nginx/blobs/sha256:c93535954a16a3e0cabc7709c608dfe36f1ea5628e0a6353aced2d77f6935698","status":200,"method":"GET","latency":"24.238426ms","ip":"10.100.217.197"}
{"time":"2024-05-28T09:05:45.082000702Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/bitnami/nginx/blobs/sha256:3fc15e54809f67d93efb415da02e3e726ddcede6ae96c26c519922a5f84586d1","status":200,"method":"GET","latency":"16.376232ms","ip":"10.100.217.197"}
{"time":"2024-05-28T09:05:45.14075545Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"ghcr.io/xenitab/azad-kube-proxy:v0.0.47@sha256:38e0b439e4745715e70e2999af2e9a44cc9425e4e22e3cba2c0a8105927d2abb","type":"CREATE"}
{"time":"2024-05-28T09:05:45.143295052Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/bitnami/nginx/blobs/sha256:4c9e4fd5a74e74241e4b03789c7606911185b251006bf516377617a472a7ba5a","status":200,"method":"GET","latency":"52.951016ms","ip":"10.100.217.197"}
{"time":"2024-05-28T09:05:45.150567161Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"ghcr.io/xenitab/azad-kube-proxy:v0.0.47@sha256:38e0b439e4745715e70e2999af2e9a44cc9425e4e22e3cba2c0a8105927d2abb","type":"UPDATE"}
{"time":"2024-05-28T09:05:45.172559176Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"ghcr.io/xenitab/azad-kube-proxy@sha256:38e0b439e4745715e70e2999af2e9a44cc9425e4e22e3cba2c0a8105927d2abb","type":"CREATE"}
{"time":"2024-05-28T09:05:46.555245152Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"docker.io/bitnami/nginx:1.22.0-debian-11-r0@sha256:38bf6c296be111f9ad2123babf3047a156676c3b47293c258e1babf2e8259400","type":"CREATE"}
{"time":"2024-05-28T09:05:46.563197154Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"docker.io/bitnami/nginx:1.22.0-debian-11-r0@sha256:38bf6c296be111f9ad2123babf3047a156676c3b47293c258e1babf2e8259400","type":"UPDATE"}
{"time":"2024-05-28T09:05:46.565256867Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"docker.io/bitnami/nginx@sha256:38bf6c296be111f9ad2123babf3047a156676c3b47293c258e1babf2e8259400","type":"CREATE"}
{"time":"2024-05-28T09:07:23.811379227Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/falcosecurity/falco-no-driver/manifests/sha256:f6e5859b190eba71ed3625426516ee323d5929cf3f34759e49463107478b8fb0","status":200,"method":"GET","latency":"938.657µs","ip":"10.100.225.242"}
{"time":"2024-05-28T09:07:23.942711199Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/falcosecurity/falco-no-driver/blobs/sha256:31b663e266a79abb58384f20607f4a6044ef822507adc4e0ba1695330416a830","status":200,"method":"GET","latency":"669.036µs","ip":"10.100.225.242"}
{"time":"2024-05-28T09:08:36.429779382Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/falcosecurity/falco-no-driver/blobs/sha256:bd159e379b3b1bc0134341e4ffdeab5f966ec422ae04818bb69ecef08a823b05","status":200,"method":"GET","latency":"61.526571ms","ip":"10.100.227.78"}
{"time":"2024-05-28T09:10:06.577284885Z","level":"INFO","source":{"function":"k8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire.func1","file":"/go/pkg/mod/k8s.io/client-go@v0.28.8/tools/leaderelection/leaderelection.go","line":260},"msg":"successfully acquired lease spegel/spegel-leader-election\n"}
{"time":"2024-05-28T09:13:59.930071613Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":33},"msg":"running scheduled image state update"}
{"time":"2024-05-28T09:22:59.929895557Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":33},"msg":"running scheduled image state update"}

After this is mainly repeats the last log lines.

I have compared the output with the
FAQ but logs dont seem to match well. This is the only pod that successfully acquires a lease which might indicate it's selected as a leader?

I am quite new as a cluster administrator so it is very possible that I have done something wrong in our setup. But would appreciate some help with understanding the logs and how to proceed.
@Versa78 Versa78 added the bug Something isn't working label May 29, 2024
@Versa78
Copy link
Author

Versa78 commented May 29, 2024

I realize these logs only contain 200-responses so here is a snippet from another cluster

{"time":"2024-05-28T12:33:05.877595258Z","level":"ERROR","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handleMirror.func1","file":"/build/pkg/registry/registry.go","line":272},"msg":"proxy failed attempting next","key":"sha256:ec7526ad34c20113824da392a3ff4646f38cd5c884ad2871817b1271cf272afe","path":"/v2/calico/kube-controllers/blobs/sha256:ec7526ad34c20113824da392a3ff4646f38cd5c884ad2871817b1271cf272afe","ip":"10.100.30.247","err":"expected mirror to respond with 200 OK but received: 500 Internal Server Error"}
{"time":"2024-05-28T12:33:05.877613501Z","level":"ERROR","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":135},"msg":"","err":"mirror resolve retries exhausted for key: sha256:ec7526ad34c20113824da392a3ff4646f38cd5c884ad2871817b1271cf272afe","path":"/v2/calico/kube-controllers/blobs/sha256:ec7526ad34c20113824da392a3ff4646f38cd5c884ad2871817b1271cf272afe","status":404,"method":"GET","latency":"3.841029ms","ip":"10.100.30.247"}
{"time":"2024-05-28T12:33:06.012606164Z","level":"ERROR","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handleMirror.func2","file":"/build/pkg/registry/registry.go","line":277},"msg":"mirror failed attempting next","key":"sha256:a26da0f61ecbf48440a5921ea6bc8bafbebc76f139cb43387e6e6a3987505fda","path":"/v2/calico/kube-controllers/blobs/sha256:a26da0f61ecbf48440a5921ea6bc8bafbebc76f139cb43387e6e6a3987505fda","ip":"10.100.30.247","err":"expected mirror to respond with 200 OK but received: 500 Internal Server Error"}
{"time":"2024-05-28T12:33:06.012631679Z","level":"ERROR","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handleMirror.func1","file":"/build/pkg/registry/registry.go","line":272},"msg":"proxy failed attempting next","key":"sha256:a26da0f61ecbf48440a5921ea6bc8bafbebc76f139cb43387e6e6a3987505fda","path":"/v2/calico/kube-controllers/blobs/sha256:a26da0f61ecbf48440a5921ea6bc8bafbebc76f139cb43387e6e6a3987505fda","ip":"10.100.30.247","err":"expected mirror to respond with 200 OK but received: 500 Internal Server Error"}
{"time":"2024-05-28T12:33:07.00066912Z","level":"ERROR","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":135},"msg":"","err":"request closed for key: sha256:a26da0f61ecbf48440a5921ea6bc8bafbebc76f139cb43387e6e6a3987505fda","path":"/v2/calico/kube-controllers/blobs/sha256:a26da0f61ecbf48440a5921ea6bc8bafbebc76f139cb43387e6e6a3987505fda","status":404,"method":"GET","latency":"1.002218451s","ip":"10.100.30.247"}

@bittrance
Copy link
Contributor

It seems there is partial success in replicating image layers. Might it be that you get problems during node startup? It is also possible that this is the suspected "EKS bug", see #469 (comment). There are two things you can do to help us pin down this issue:

  • Do you have observability so you can plot the errors over time? That might be helpful to see whether the error is ongoing or spiked after node startup.
  • The error logs above are from the mirroring node. Can you provide the matching entries from the "mirroree" so we can see what the 500 error was?

Also, just to be sure: you are sure that the pods not starting are because of docker.io rate limiting? The reason I'm asking is that one of the hypotheses on the "EKS bug" is that EKS containerd was doing something special when talking to ECR.

@Versa78
Copy link
Author

Versa78 commented Jun 4, 2024

I am not sure of this node is the mirroring node or the mirroree but this one has much less errors then the others and not the same

{"time":"2024-05-28T14:50:57.017610243Z","level":"ERROR","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":135},"msg":"","err":"mirror resolve retries exhausted for key: sha256:bac155346eb5f43f6b555838bb60e065ced11f6f27675a2b7f166304a86443bc","path":"/v2/falcosecurity/falco-driver-loader/blobs/sha256:bac155346eb5f43f6b555838bb60e065ced11f6f27675a2b7f166304a86443bc","status":404,"method":"GET","latency":"20.222778ms","ip":"10.100.25.254"}

I tried spinning up a new node today and I think all the logs looked good.

{"time":"2024-06-04T06:06:53.856250154Z","level":"INFO","source":{"function":"main.registryCommand","file":"/build/main.go","line":216},"msg":"running Spegel","registry":":5000","router":":5001"}
{"time":"2024-06-04T06:06:53.856901792Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/routing.(*P2PRouter).Run","file":"/build/pkg/routing/p2p.go","line":123},"msg":"starting p2p router","logger":"p2p","id":"/ip4/192.168.35.73/tcp/5001/p2p/12D3KooWKAUDmcptsxCmuGRKRGBnivAWu6S9R6KFvkTzcEFFkXbE"}
{"time":"2024-06-04T06:06:53.856984794Z","level":"INFO","source":{"function":"k8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire","file":"/go/pkg/mod/k8s.io/client-go@v0.28.8/tools/leaderelection/leaderelection.go","line":250},"msg":"attempting to acquire leader lease spegel/spegel-leader-election...\n"}
{"time":"2024-06-04T06:06:53.857425815Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":33},"msg":"running scheduled image state update"}
{"time":"2024-06-04T06:07:04.179787364Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/falcosecurity/falco-no-driver/manifests/0.33.0","status":200,"method":"HEAD","latency":"3.727163ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:07:04.190076571Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/falcosecurity/falco-no-driver/manifests/sha256:f6e5859b190eba71ed3625426516ee323d5929cf3f34759e49463107478b8fb0","status":200,"method":"GET","latency":"3.758016ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:07:04.206919936Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/falcosecurity/falco-no-driver/manifests/sha256:307deb97d3b022f26cce289647491cc3033a0ff3562123edee928f52654f08d7","status":200,"method":"GET","latency":"2.061671ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:07:04.266727005Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/falcosecurity/falco-no-driver/blobs/sha256:31b663e266a79abb58384f20607f4a6044ef822507adc4e0ba1695330416a830","status":200,"method":"GET","latency":"45.569297ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:07:04.348331412Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/falcosecurity/falco-no-driver/blobs/sha256:123a7620f3e0394a65b6188d62491d321060918bb25c43d3d122a65d0aa19add","status":200,"method":"GET","latency":"32.82235ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:07:04.377254121Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/falcosecurity/falco-no-driver/blobs/sha256:bd159e379b3b1bc0134341e4ffdeab5f966ec422ae04818bb69ecef08a823b05","status":200,"method":"GET","latency":"55.769512ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:07:05.275205956Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"docker.io/falcosecurity/falco-no-driver:0.33.0@sha256:f6e5859b190eba71ed3625426516ee323d5929cf3f34759e49463107478b8fb0","type":"CREATE"}
{"time":"2024-06-04T06:07:05.281247957Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"docker.io/falcosecurity/falco-no-driver:0.33.0@sha256:f6e5859b190eba71ed3625426516ee323d5929cf3f34759e49463107478b8fb0","type":"UPDATE"}
{"time":"2024-06-04T06:07:05.285000489Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"docker.io/falcosecurity/falco-no-driver@sha256:f6e5859b190eba71ed3625426516ee323d5929cf3f34759e49463107478b8fb0","type":"CREATE"}
{"time":"2024-06-04T06:15:53.857764657Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":33},"msg":"running scheduled image state update"}
{"time":"2024-06-04T06:21:04.313881512Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/xenitab/azad-kube-proxy/manifests/v0.0.47","status":200,"method":"HEAD","latency":"5.655564ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:04.342285871Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/xenitab/azad-kube-proxy/manifests/sha256:38e0b439e4745715e70e2999af2e9a44cc9425e4e22e3cba2c0a8105927d2abb","status":200,"method":"GET","latency":"13.050644ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:04.43471706Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/xenitab/azad-kube-proxy/manifests/sha256:9e8d1e60b4794c5e42f8998385cfd5623175726a81ee728fe2e90d990ace8e3e","status":200,"method":"GET","latency":"54.764761ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:04.478152276Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/xenitab/azad-kube-proxy/blobs/sha256:44142d127ecb76b0603a48cf431dd8b8d6ed54a22a1c85819ec1d0954f71f149","status":200,"method":"GET","latency":"5.257783ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:04.543386049Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/xenitab/azad-kube-proxy/blobs/sha256:b02a7525f878e61fc1ef8a7405a2cc17f866e8de222c1c98fd6681aff6e509db","status":200,"method":"GET","latency":"29.252393ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:04.543873739Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/xenitab/azad-kube-proxy/blobs/sha256:a7ca0d9ba68fdce7e15bc0952d3e898e970548ca24d57698725836c039086639","status":200,"method":"GET","latency":"21.153331ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:04.545064117Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/xenitab/azad-kube-proxy/blobs/sha256:fe5ca62666f04366c8e7f605aa82997d71320183e99962fa76b3209fdfbb8b58","status":200,"method":"GET","latency":"8.58283ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:04.613170269Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/xenitab/azad-kube-proxy/blobs/sha256:fcb6f6d2c9986d9cd6a2ea3cc2936e5fc613e09f1af9042329011e43057f3265","status":200,"method":"GET","latency":"5.878567ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:04.634180899Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/xenitab/azad-kube-proxy/blobs/sha256:e8c73c638ae9ec5ad70c49df7e484040d889cca6b4a9af056579c3d058ea93f0","status":200,"method":"GET","latency":"17.727024ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:04.652051487Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/xenitab/azad-kube-proxy/blobs/sha256:1e3d9b7d145208fa8fa3ee1c9612d0adaac7255f1bbc9ddea7e461e0b317805c","status":200,"method":"GET","latency":"28.916146ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:04.676551141Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/xenitab/azad-kube-proxy/blobs/sha256:7c881f9ab25e0d86562a123b5fb56aebf8aa0ddd7d48ef602faf8d1e7cf43d8c","status":200,"method":"GET","latency":"10.217992ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:04.685577796Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/xenitab/azad-kube-proxy/blobs/sha256:5627a970d25e752d971a501ec7e35d0d6fdcd4a3ce9e958715a686853024794a","status":200,"method":"GET","latency":"6.398632ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:04.862226772Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/xenitab/azad-kube-proxy/blobs/sha256:8e194026a9da5e5406e74b62f917da661940f25822b19fba1b33d82dd200ce80","status":200,"method":"GET","latency":"29.273094ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:04.864495242Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/xenitab/azad-kube-proxy/blobs/sha256:3569c07ce5e82afc212d7ca8de507fdd6b7b4df301c2ce82b371421c79a504df","status":200,"method":"GET","latency":"30.48386ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:04.86485843Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/openpolicyagent/gatekeeper/manifests/v3.9.0","status":200,"method":"HEAD","latency":"30.762211ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:04.895294727Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/xenitab/azad-kube-proxy/blobs/sha256:3ac7cb81c22056d173c05e4446fc19b683ade9368b52db0e955541866a9a708b","status":200,"method":"GET","latency":"113.202427ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:05.046611672Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/openpolicyagent/gatekeeper/manifests/sha256:fe1ef3bb611456d6bf301f9f2a219306564ee9ad17269f78da59eb0afb6c956a","status":200,"method":"GET","latency":"119.332218ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:05.094413949Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/openpolicyagent/gatekeeper/manifests/v3.9.0","status":200,"method":"HEAD","latency":"100.324117ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:05.138150004Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/openpolicyagent/gatekeeper/manifests/sha256:3cab711e3bc3da16b7c605a57026480aaac728108ffce423219935abb474fbcf","status":200,"method":"GET","latency":"56.366677ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:05.24588819Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/openpolicyagent/gatekeeper/blobs/sha256:7694b49c90c35a1589df6b8314ad9a03d8a70adb2297398f8ab7d8f7d578a646","status":200,"method":"GET","latency":"18.727778ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:05.47130337Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/openpolicyagent/gatekeeper/blobs/sha256:b9f88661235d25835ef747dab426861d51c4e9923b92623d422d7ac58eb123e9","status":200,"method":"GET","latency":"147.893917ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:05.875682703Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/bitnami/external-dns/manifests/0.13.1-debian-11-r19","status":200,"method":"HEAD","latency":"297.800581ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:05.898391628Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/jetstack/cert-manager-controller/manifests/v1.7.1","status":200,"method":"HEAD","latency":"67.714006ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:05.943444583Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/bitnami/external-dns/manifests/sha256:1a9664c161ed1d96445c7bc7bd0df5626ea841bdcdd6a88aa48bdd153c000170","status":200,"method":"GET","latency":"60.009352ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:05.947344167Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/openpolicyagent/gatekeeper/blobs/sha256:b260db4f6bf0bd81639b891f2fc469276d615e6d9a429d312984f78818969a5d","status":200,"method":"GET","latency":"640.756ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:06.051800528Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/jetstack/cert-manager-controller/manifests/sha256:51027a4cc4d30e197e3506daf3a4fa2d2a0bc2826469f8a87848dfd279e031c0","status":200,"method":"GET","latency":"108.369184ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:06.208639923Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/jetstack/cert-manager-controller/manifests/sha256:567f84c097de6bc4ce8ff14bc8a46f9ba08442ed1be6b337983344d247225fa5","status":200,"method":"GET","latency":"77.692783ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:06.230862847Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/bitnami/external-dns/blobs/sha256:ec1c193abc452b8ca0c210ccc712d6ce501fdef7256d2068373195f1db4e3a54","status":200,"method":"GET","latency":"101.036472ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:06.27467302Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/ingress-nginx/controller-chroot/manifests/sha256:c1c091b88a6c936a83bd7b098662760a87868d12452529bad0d178fb36147345","status":200,"method":"HEAD","latency":"19.256344ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:06.367422487Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"ghcr.io/xenitab/azad-kube-proxy:v0.0.47@sha256:38e0b439e4745715e70e2999af2e9a44cc9425e4e22e3cba2c0a8105927d2abb","type":"CREATE"}
{"time":"2024-06-04T06:21:06.571195246Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/ingress-nginx/controller-chroot/manifests/sha256:c1c091b88a6c936a83bd7b098662760a87868d12452529bad0d178fb36147345","status":200,"method":"GET","latency":"63.837539ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:06.669608248Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/fluxcd/helm-controller/manifests/v0.31.2","status":200,"method":"HEAD","latency":"104.925087ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:06.86101252Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/jetstack/cert-manager-controller/blobs/sha256:db7725ef729d74e24d51c93f831fa69b22747e67507f6bc2d7c981d16920ff35","status":200,"method":"GET","latency":"13.748024ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:07.37078246Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/fluxcd/helm-controller/manifests/sha256:8180a8cd8765d79b181a961000e051ab99dc50407740b3d3523cba98300936ff","status":200,"method":"GET","latency":"30.252873ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:07.378629555Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"ghcr.io/xenitab/azad-kube-proxy:v0.0.47@sha256:38e0b439e4745715e70e2999af2e9a44cc9425e4e22e3cba2c0a8105927d2abb","type":"UPDATE"}
{"time":"2024-06-04T06:21:07.448589472Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/ingress-nginx/controller-chroot/manifests/sha256:404043cd0073e4cafe4e68a785ae76b4a67f24d7a58d8a3487e915f24a2db0cd","status":200,"method":"GET","latency":"24.020872ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:07.613717664Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/jetstack/cert-manager-controller/blobs/sha256:efd4cd7e0f1ae7643a1325cf951a7da9b950e1cc02256dd951459584beb60ac7","status":200,"method":"GET","latency":"11.04978ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:07.667551835Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/jetstack/cert-manager-controller/blobs/sha256:2df365faf0e3007f983fadd7a65ba51d41b488eb2ed8fc70f4bf97043cfea560","status":200,"method":"GET","latency":"63.496167ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:07.68720452Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/fluxcd/helm-controller/manifests/sha256:bfc6e6a48d77060f5cfc7884d003e72803111ce6a6fa18cfd630df314d366037","status":200,"method":"GET","latency":"85.464462ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:07.768331249Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/ingress-nginx/controller-chroot/blobs/sha256:4d27e340724e3927caa65a8567563a2342d24428ed33a68cb107fdda73756533","status":200,"method":"GET","latency":"178.93825ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:07.769973703Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"docker.io/openpolicyagent/gatekeeper:v3.9.0@sha256:fe1ef3bb611456d6bf301f9f2a219306564ee9ad17269f78da59eb0afb6c956a","type":"CREATE"}
{"time":"2024-06-04T06:21:07.778697785Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/fluxcd/helm-controller/blobs/sha256:a5f4f333b754fc1a8ad0ab7288c31806980a0731b227837c9be9b7be2f103e62","status":200,"method":"GET","latency":"23.396705ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:08.000227005Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/fluxcd/helm-controller/blobs/sha256:63b65145d645c1250c391b2d16ebe53b3747c295ca8ba2fcb6b0cf064a4dc21c","status":200,"method":"GET","latency":"75.078944ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:08.368173221Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/ingress-nginx/controller-chroot/blobs/sha256:e64ed83561575574087a65fa0d85b3a1200bad4653b178a44e0090645b83d43a","status":200,"method":"GET","latency":"446.281712ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:08.369197455Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/fluxcd/helm-controller/blobs/sha256:4b0841908e08cb839e4b11ff5e6e09db471d2a7f649882a366b61defe23039e8","status":200,"method":"GET","latency":"442.64774ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:08.528753806Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"ghcr.io/xenitab/azad-kube-proxy@sha256:38e0b439e4745715e70e2999af2e9a44cc9425e4e22e3cba2c0a8105927d2abb","type":"CREATE"}
{"time":"2024-06-04T06:21:09.088581125Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/ingress-nginx/controller-chroot/blobs/sha256:fbba675851a00814faa00eda9fae6b040ab65ef5b3a9ac86ec3404b9cbb10a52","status":200,"method":"GET","latency":"1.087980324s","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:09.126768733Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/ingress-nginx/controller-chroot/blobs/sha256:230441d2bc1a61e0fa173a21e07bfd9d31de115b463636267b517ea69b1bffcb","status":200,"method":"GET","latency":"199.667632ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:09.565552271Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/jetstack/cert-manager-controller/blobs/sha256:76232995bb58358527282aedabb06b10c13369c35d463a02726173073aff5f8f","status":200,"method":"GET","latency":"1.962994938s","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:09.708960015Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/bitnami/external-dns/blobs/sha256:ddd2fb06d406b0b813d5f75e936a9363c62eeb0fbb1c598ae0857d6b71b9d586","status":200,"method":"GET","latency":"2.299263993s","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:10.183240385Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/bitnami/nginx/manifests/1.22.0-debian-11-r0","status":200,"method":"HEAD","latency":"640.202158ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:10.267927493Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/ingress-nginx/controller-chroot/manifests/sha256:c1c091b88a6c936a83bd7b098662760a87868d12452529bad0d178fb36147345","status":200,"method":"HEAD","latency":"724.800737ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:10.299891466Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/bitnami/external-dns/blobs/sha256:f8c1c832ce65f8c860a3388bf6681a4046a949475bc305b5a22e6deddf94abb5","status":200,"method":"GET","latency":"2.872028302s","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:10.307371406Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/bitnami/nginx/manifests/sha256:38bf6c296be111f9ad2123babf3047a156676c3b47293c258e1babf2e8259400","status":200,"method":"GET","latency":"5.990702ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:10.309778305Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"docker.io/openpolicyagent/gatekeeper:v3.9.0@sha256:fe1ef3bb611456d6bf301f9f2a219306564ee9ad17269f78da59eb0afb6c956a","type":"UPDATE"}
{"time":"2024-06-04T06:21:10.315873513Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"docker.io/openpolicyagent/gatekeeper:v3.9.0@sha256:fe1ef3bb611456d6bf301f9f2a219306564ee9ad17269f78da59eb0afb6c956a","type":"UPDATE"}
{"time":"2024-06-04T06:21:10.317574498Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"docker.io/openpolicyagent/gatekeeper@sha256:fe1ef3bb611456d6bf301f9f2a219306564ee9ad17269f78da59eb0afb6c956a","type":"CREATE"}
{"time":"2024-06-04T06:21:10.318578242Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/ingress-nginx/controller-chroot/blobs/sha256:a4eac395574239077f75cea8353fbfa02150a09543cc32327e24f66803534fea","status":200,"method":"GET","latency":"1.181018246s","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:10.319353695Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"docker.io/openpolicyagent/gatekeeper:v3.9.0@sha256:fe1ef3bb611456d6bf301f9f2a219306564ee9ad17269f78da59eb0afb6c956a","type":"UPDATE"}
{"time":"2024-06-04T06:21:10.321837256Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"docker.io/openpolicyagent/gatekeeper@sha256:fe1ef3bb611456d6bf301f9f2a219306564ee9ad17269f78da59eb0afb6c956a","type":"UPDATE"}
{"time":"2024-06-04T06:21:10.347884387Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/ingress-nginx/controller-chroot/blobs/sha256:0a8d00bc8d6ed1819b3f3c00a8d8aa9ab7c192f9c80e57625a503a529d797fe4","status":200,"method":"GET","latency":"1.151124559s","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:10.371966365Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/fluxcd/helm-controller/blobs/sha256:271e1ef8f68704966e7b13632fab235578b0340f1b2021f71b8bcdbd25a027a3","status":200,"method":"GET","latency":"2.447949118s","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:10.71347962Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/ingress-nginx/controller-chroot/blobs/sha256:f4bf85ec37424e94e1593ca5f5687e823c1eb39792fb47c58022735c1a9285e5","status":200,"method":"GET","latency":"2.71297492s","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:11.509909016Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"quay.io/jetstack/cert-manager-controller:v1.7.1@sha256:51027a4cc4d30e197e3506daf3a4fa2d2a0bc2826469f8a87848dfd279e031c0","type":"CREATE"}
{"time":"2024-06-04T06:21:11.587042017Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/bitnami/nginx/blobs/sha256:d5aaf494d47984f28e23764ac18719c555175dcf1aa8989bbe0d766455e90918","status":200,"method":"GET","latency":"47.240745ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:11.703521838Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/ingress-nginx/controller-chroot/blobs/sha256:791b46dac2e7ae607f5ed2adef3a175adb058737592d7fe933383d29ecdc46fe","status":200,"method":"GET","latency":"208.444602ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:11.739051181Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/ingress-nginx/controller-chroot/blobs/sha256:d27caff71b56fb4e593f9f4294aa00516ca92310d2caaeefd70d22f291186536","status":200,"method":"GET","latency":"8.447125ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:11.830127033Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/ingress-nginx/controller-chroot/blobs/sha256:210eb4ef167c8f5b5953d80a0e183e28a0445532e30745231781bc6f19bbcc05","status":200,"method":"GET","latency":"96.663835ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:11.834180156Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/bitnami/nginx/blobs/sha256:2974f71d01ecd67fb6c3d2123aa9187f06f0aee0d8554d4f8d3cff94528b2a4f","status":200,"method":"GET","latency":"3.834048ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:11.913151827Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/ingress-nginx/controller-chroot/blobs/sha256:15fcdc9f98a7d87bd9249c086b1b0c5845d6ea9274a0c0058c6ec74a1bf7b05f","status":200,"method":"GET","latency":"82.801048ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:11.925776243Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/bitnami/nginx/blobs/sha256:040d6f7df45c4ca28bded21d8c1b8e8ba8f14b3bd81e88beb6b71c1ce2d97483","status":200,"method":"GET","latency":"173.11458ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:11.966003376Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"quay.io/jetstack/cert-manager-controller:v1.7.1@sha256:51027a4cc4d30e197e3506daf3a4fa2d2a0bc2826469f8a87848dfd279e031c0","type":"UPDATE"}
{"time":"2024-06-04T06:21:11.975944669Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"quay.io/jetstack/cert-manager-controller@sha256:51027a4cc4d30e197e3506daf3a4fa2d2a0bc2826469f8a87848dfd279e031c0","type":"CREATE"}
{"time":"2024-06-04T06:21:11.992995039Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/ingress-nginx/controller-chroot/blobs/sha256:6f26eac30d443cc7e275a182f0518fe46c7ef4b6b98dc52ded4d1cdead0af81f","status":200,"method":"GET","latency":"421.932976ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:12.36065128Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/bitnami/nginx/blobs/sha256:2f205c6e7ae2f69fc2488fe9f2b84c7c9be1afe3be966300bf3e988a4664809d","status":200,"method":"GET","latency":"529.532772ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:12.909352288Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/bitnami/nginx/blobs/sha256:10a01e04b7dd97129a41b499825beaddd22ef063a2a9f3ae30d5c3f33a0991fd","status":200,"method":"GET","latency":"10.758356ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:13.087650031Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/bitnami/nginx/blobs/sha256:c8ad9148c174abaa60ad8608b4015b453460d97b67aaa0c76bd732adb940f725","status":200,"method":"GET","latency":"27.214503ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:15.640318837Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/bitnami/nginx/blobs/sha256:8d1893ebbb43008e20e78fc714de084bfb0fff08cfe50805b0e8503702b257a6","status":200,"method":"GET","latency":"10.846769ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:15.746923645Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"ghcr.io/fluxcd/helm-controller:v0.31.2@sha256:8180a8cd8765d79b181a961000e051ab99dc50407740b3d3523cba98300936ff","type":"CREATE"}
{"time":"2024-06-04T06:21:15.747791916Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/bitnami/nginx/blobs/sha256:34cde175a22cd59ab963eb156a4639cb65ffd1b1d77e7ba218f5f2f56c1d409c","status":200,"method":"GET","latency":"13.970538ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:15.749550274Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/bitnami/nginx/blobs/sha256:d52b53bf148f6be1d78067fa6f79c5e2a7d5da4d8e668fc792bfda74242040ec","status":200,"method":"GET","latency":"13.703623ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:15.826677763Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/bitnami/nginx/blobs/sha256:100cc416bdcdeda1e010dc0408abfbdb58b04b8a6185871d5ea923574cf1dbec","status":200,"method":"GET","latency":"47.512713ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:15.836139492Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/bitnami/nginx/blobs/sha256:3fc15e54809f67d93efb415da02e3e726ddcede6ae96c26c519922a5f84586d1","status":200,"method":"GET","latency":"3.037291ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:15.857095327Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/bitnami/nginx/blobs/sha256:c93535954a16a3e0cabc7709c608dfe36f1ea5628e0a6353aced2d77f6935698","status":200,"method":"GET","latency":"6.344028ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:15.860241366Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"ghcr.io/fluxcd/helm-controller:v0.31.2@sha256:8180a8cd8765d79b181a961000e051ab99dc50407740b3d3523cba98300936ff","type":"UPDATE"}
{"time":"2024-06-04T06:21:15.939010623Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/bitnami/nginx/blobs/sha256:4c9e4fd5a74e74241e4b03789c7606911185b251006bf516377617a472a7ba5a","status":200,"method":"GET","latency":"23.985856ms","ip":"10.100.31.13"}
{"time":"2024-06-04T06:21:16.004897551Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"ghcr.io/fluxcd/helm-controller@sha256:8180a8cd8765d79b181a961000e051ab99dc50407740b3d3523cba98300936ff","type":"CREATE"}
{"time":"2024-06-04T06:21:17.365835927Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"docker.io/bitnami/external-dns:0.13.1-debian-11-r19@sha256:1a9664c161ed1d96445c7bc7bd0df5626ea841bdcdd6a88aa48bdd153c000170","type":"CREATE"}
{"time":"2024-06-04T06:21:17.437821146Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"docker.io/bitnami/external-dns:0.13.1-debian-11-r19@sha256:1a9664c161ed1d96445c7bc7bd0df5626ea841bdcdd6a88aa48bdd153c000170","type":"UPDATE"}
{"time":"2024-06-04T06:21:17.454428717Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"docker.io/bitnami/external-dns@sha256:1a9664c161ed1d96445c7bc7bd0df5626ea841bdcdd6a88aa48bdd153c000170","type":"CREATE"}
{"time":"2024-06-04T06:21:20.243673313Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"docker.io/bitnami/nginx:1.22.0-debian-11-r0@sha256:38bf6c296be111f9ad2123babf3047a156676c3b47293c258e1babf2e8259400","type":"CREATE"}
{"time":"2024-06-04T06:21:20.294793547Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"docker.io/bitnami/nginx:1.22.0-debian-11-r0@sha256:38bf6c296be111f9ad2123babf3047a156676c3b47293c258e1babf2e8259400","type":"UPDATE"}
{"time":"2024-06-04T06:21:20.299507746Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"docker.io/bitnami/nginx@sha256:38bf6c296be111f9ad2123babf3047a156676c3b47293c258e1babf2e8259400","type":"CREATE"}
{"time":"2024-06-04T06:21:20.385874408Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"registry.k8s.io/ingress-nginx/controller-chroot@sha256:c1c091b88a6c936a83bd7b098662760a87868d12452529bad0d178fb36147345","type":"CREATE"}
{"time":"2024-06-04T06:21:20.787009127Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"registry.k8s.io/ingress-nginx/controller-chroot@sha256:c1c091b88a6c936a83bd7b098662760a87868d12452529bad0d178fb36147345","type":"UPDATE"}
{"time":"2024-06-04T06:21:21.099602615Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"registry.k8s.io/ingress-nginx/controller-chroot@sha256:c1c091b88a6c936a83bd7b098662760a87868d12452529bad0d178fb36147345","type":"UPDATE"}
{"time":"2024-06-04T06:21:21.295299872Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":42},"msg":"received image event","image":"registry.k8s.io/ingress-nginx/controller-chroot@sha256:c1c091b88a6c936a83bd7b098662760a87868d12452529bad0d178fb36147345","type":"UPDATE"}

So I guess spegel is working as intended?

@Versa78
Copy link
Author

Versa78 commented Jun 7, 2024

As I am upgrading nodes to a newer version (1.28) at the same time as I am updating spegel from 0.0.14 -> 0.0.22 could there be an issue with communication between the older spegel pods and the newer ones?

@phillebaba
Copy link
Member

In theory no there should not be an issue. They are both running OCI compliant registries. There are however no tests to verify that this actually works. I would say that I am 90% sure that this should work.

@Versa78
Copy link
Author

Versa78 commented Jun 7, 2024

Right now I am experiencing some kind of catch 22 where it cant pull from docker.io becuase it has exhausted the number of allowed pulls and therefor cant start Calico pods and spegel won't start because Calico is not up.

Warning NetworkNotReady 2m37s (x302 over 12m) kubelet network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

As I drained the older nodes I think spegel re-elected a new leader as can be seen by these logs I think

"time":"2024-06-07T10:08:52.516369943Z","level":"ERROR","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":135},"msg":"","err":"request closed for key: sha256:53de18a253cf4a58c790f85e4ac7a73b5449e1c6e826b70fe7ee4db875328fb3","path":"/v2/fluxcd/notification-controller/blobs/sha256:53de18a253cf4a58c790f85e4ac7a73b5449e1c6e826b70fe7ee4db875328fb3","status":404,"method":"GET","latency":"1.000600273s","ip":"10.100.154.221"}
{"time":"2024-06-07T10:09:26.057489451Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":33},"msg":"running scheduled image state update"}
{"time":"2024-06-07T10:10:07.275708805Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/ingress-nginx/controller-chroot/blobs/sha256:6f26eac30d443cc7e275a182f0518fe46c7ef4b6b98dc52ded4d1cdead0af81f","status":200,"method":"GET","latency":"30.379927ms","ip":"10.100.156.205"}
{"time":"2024-06-07T10:10:07.283352745Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/ingress-nginx/controller-chroot/blobs/sha256:15fcdc9f98a7d87bd9249c086b1b0c5845d6ea9274a0c0058c6ec74a1bf7b05f","status":200,"method":"GET","latency":"357.47µs","ip":"10.100.156.205"}
{"time":"2024-06-07T10:18:26.057754544Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":33},"msg":"running scheduled image state update"}
{"time":"2024-06-07T10:27:26.057258416Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":33},"msg":"running scheduled image state update"}
{"time":"2024-06-07T10:36:26.057539111Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":33},"msg":"running scheduled image state update"}
{"time":"2024-06-07T10:45:26.057218316Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":33},"msg":"running scheduled image state update"}
{"time":"2024-06-07T10:53:55.789161717Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/openpolicyagent/gatekeeper/manifests/v3.9.0","status":200,"method":"HEAD","latency":"1.066476ms","ip":"10.100.162.114"}
{"time":"2024-06-07T10:53:55.815065709Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/openpolicyagent/gatekeeper/manifests/sha256:3cab711e3bc3da16b7c605a57026480aaac728108ffce423219935abb474fbcf","status":200,"method":"GET","latency":"598.452µs","ip":"10.100.162.114"}
{"time":"2024-06-07T10:53:55.830165531Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":132},"msg":"","path":"/v2/openpolicyagent/gatekeeper/blobs/sha256:7694b49c90c35a1589df6b8314ad9a03d8a70adb2297398f8ab7d8f7d578a646","status":200,"method":"GET","latency":"350.098µs","ip":"10.100.162.114"}
{"time":"2024-06-07T10:54:26.056848019Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":33},"msg":"running scheduled image state update"}
{"time":"2024-06-07T10:55:12.383866239Z","level":"INFO","source":{"function":"k8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire.func1","file":"/go/pkg/mod/k8s.io/client-go@v0.28.8/tools/leaderelection/leaderelection.go","line":260},"msg":"successfully acquired lease spegel/spegel-leader-election\n"}
{"time":"2024-06-07T11:03:26.056835313Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":33},"msg":"running scheduled image state update"}
{"time":"2024-06-07T11:12:26.056997404Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":33},"msg":"running scheduled image state update"}
{"time":"2024-06-07T11:21:26.057445879Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":33},"msg":"running scheduled image state update"}

@phillebaba
Copy link
Member

Spegel relies on the CNI working so there is no real method to solve this.

@Versa78
Copy link
Author

Versa78 commented Jun 12, 2024

Ok, I've configured the pull of Calico image to use docker credentials and then spegel can hopefully solve any other images if needed.

@Versa78 Versa78 closed this as completed Jun 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@bittrance @phillebaba @Versa78