Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Version 0.4.0 error "Could not sync required resources" but version 0.3.4 works #279

Open
rwarford opened this issue Apr 7, 2024 · 5 comments
Open

Version 0.4.0 error "Could not sync required resources" but version 0.3.4 works #279

rwarford opened this issue Apr 7, 2024 · 5 comments

Comments

@rwarford
Copy link

rwarford commented Apr 7, 2024

I installed via Helm chart 2.0.4 (app version 0.3.4) and DNS resolution works correctly.
I uninstalled then installed Helm chart 2.1.0 (app version 0.4.0) and DNS resolution no longer works (dig reports SERVFAIL).
I've tried Helm chart 2.4.0 and get the same results.

Logs from app version 0.4.0 (note the Could not sync required resources error):

[INFO] plugin/k8s_gateway: Building k8s_gateway controller
[INFO] plugin/k8s_gateway: VirtualServer CRDs are not found. Not syncing VirtualServer resources.
[INFO] plugin/k8s_gateway: Starting k8s_gateway controller
[INFO] plugin/k8s_gateway: Waiting for controllers to sync
.:1053
[DEBUG] plugin/k8s_gateway: Request 1778848412669144676.1792364358248969471. has not matched any zones [XXXXXXX.com.]
W0407 20:23:30.501935       1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1alpha2.TLSRoute: the server could not find the requested resource (get tlsroutes.gateway.networking.k8s.io)
E0407 20:23:30.502068       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1alpha2.TLSRoute: failed to list *v1alpha2.TLSRoute: the server could not find the requested resource (get tlsroutes.gateway.networking.k8s.io)
[INFO] plugin/reload: Running configuration SHA512 = 321cf4ac2f3862a8000e9fb4f5c51a7019dc1d2e0bb1ef25da8ff1beb229fc8c43b57231f2e8bf971eb7d99f54d439ee893119d59937354704a0e61ab22d433f
CoreDNS-1.11.1+k8s_gateway-0.4.0
linux/amd64, go1.21.3, 22e389c
W0407 20:23:30.504228       1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1alpha2.GRPCRoute: the server could not find the requested resource (get grpcroutes.gateway.networking.k8s.io)
E0407 20:23:30.504300       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1alpha2.GRPCRoute: failed to list *v1alpha2.GRPCRoute: the server could not find the requested resource (get grpcroutes.gateway.networking.k8s.io)
[DEBUG] plugin/k8s_gateway: Adding index nginx-service.default for service nginx-service
[DEBUG] plugin/k8s_gateway: Adding index k8s-gateway-application.k8s-gateway-system for service k8s-gateway-application
[INFO] 127.0.0.1:59904 - 42794 "HINFO IN 1778848412669144676.1792364358248969471. udp 57 false 512" NXDOMAIN qr,rd,ra 132 0.014001293s
W0407 20:23:31.767988       1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1alpha2.GRPCRoute: the server could not find the requested resource (get grpcroutes.gateway.networking.k8s.io)
E0407 20:23:31.768041       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1alpha2.GRPCRoute: failed to list *v1alpha2.GRPCRoute: the server could not find the requested resource (get grpcroutes.gateway.networking.k8s.io)
W0407 20:23:31.934310       1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1alpha2.TLSRoute: the server could not find the requested resource (get tlsroutes.gateway.networking.k8s.io)
E0407 20:23:31.934360       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1alpha2.TLSRoute: failed to list *v1alpha2.TLSRoute: the server could not find the requested resource (get tlsroutes.gateway.networking.k8s.io)
W0407 20:23:33.888547       1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1alpha2.TLSRoute: the server could not find the requested resource (get tlsroutes.gateway.networking.k8s.io)
E0407 20:23:33.888610       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1alpha2.TLSRoute: failed to list *v1alpha2.TLSRoute: the server could not find the requested resource (get tlsroutes.gateway.networking.k8s.io)
W0407 20:23:34.946547       1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1alpha2.GRPCRoute: the server could not find the requested resource (get grpcroutes.gateway.networking.k8s.io)
E0407 20:23:34.946679       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1alpha2.GRPCRoute: failed to list *v1alpha2.GRPCRoute: the server could not find the requested resource (get grpcroutes.gateway.networking.k8s.io)
W0407 20:23:38.099568       1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1alpha2.TLSRoute: the server could not find the requested resource (get tlsroutes.gateway.networking.k8s.io)
E0407 20:23:38.099619       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1alpha2.TLSRoute: failed to list *v1alpha2.TLSRoute: the server could not find the requested resource (get tlsroutes.gateway.networking.k8s.io)
W0407 20:23:41.340906       1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1alpha2.GRPCRoute: the server could not find the requested resource (get grpcroutes.gateway.networking.k8s.io)
E0407 20:23:41.340959       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1alpha2.GRPCRoute: failed to list *v1alpha2.GRPCRoute: the server could not find the requested resource (get grpcroutes.gateway.networking.k8s.io)
W0407 20:23:49.403101       1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1alpha2.GRPCRoute: the server could not find the requested resource (get grpcroutes.gateway.networking.k8s.io)
E0407 20:23:49.403211       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1alpha2.GRPCRoute: failed to list *v1alpha2.GRPCRoute: the server could not find the requested resource (get grpcroutes.gateway.networking.k8s.io)
W0407 20:23:50.171164       1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1alpha2.TLSRoute: the server could not find the requested resource (get tlsroutes.gateway.networking.k8s.io)
E0407 20:23:50.171208       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1alpha2.TLSRoute: failed to list *v1alpha2.TLSRoute: the server could not find the requested resource (get tlsroutes.gateway.networking.k8s.io)
W0407 20:24:04.470328       1 reflector.go:535] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1alpha2.TLSRoute: the server could not find the requested resource (get tlsroutes.gateway.networking.k8s.io)
E0407 20:24:04.470372       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1alpha2.TLSRoute: failed to list *v1alpha2.TLSRoute: the server could not find the requested resource (get tlsroutes.gateway.networking.k8s.io)
[DEBUG] plugin/k8s_gateway: Computed Index Keys [nginx-service.default.XXXXXXX.com nginx-service.default]
[INFO] 10.42.232.128:51583 - 6720 "A IN nginx-service.default.XXXXXXX.com. udp 93 false 4096" - - 0 0.000086851s
[ERROR] plugin/errors: 2 nginx-service.default.XXXXXXX.com. A: plugin/k8s_gateway: Could not sync required resources

Logs from app version 0.3.4 (which is working correctly):

[INFO] plugin/k8s_gateway: Building k8s_gateway controller
[INFO] plugin/k8s_gateway: GatewayAPI CRDs are not found. Not syncing GatewayAPI resources.
[INFO] plugin/k8s_gateway: VirtualServer CRDs are not found. Not syncing VirtualServer resources.
[INFO] plugin/k8s_gateway: Starting k8s_gateway controller
[INFO] plugin/k8s_gateway: Waiting for controllers to sync
.:1053
[INFO] plugin/reload: Running configuration SHA512 = 321cf4ac2f3862a8000e9fb4f5c51a7019dc1d2e0bb1ef25da8ff1beb229fc8c43b57231f2e8bf971eb7d99f54d439ee893119d59937354704a0e61ab22d433f
CoreDNS-1.10.1+k8s_gateway-0.3.4
linux/amd64, go1.20.2, c982ce7
[DEBUG] plugin/k8s_gateway: Request 962279960104462924.4524425992165016576. has not matched any zones [XXXXXXX.com.]
[DEBUG] plugin/k8s_gateway: Adding index nginx-service.default for service nginx-service
[DEBUG] plugin/k8s_gateway: Adding index k8s-gateway-application.k8s-gateway-system for service k8s-gateway-application
[INFO] 127.0.0.1:44227 - 42749 "HINFO IN 962279960104462924.4524425992165016576. udp 56 false 512" NXDOMAIN qr,rd,ra 131 0.012402933s
[INFO] plugin/k8s_gateway: Synced all required resources
[DEBUG] plugin/k8s_gateway: Computed Index Keys [nginx-service.default.XXXXXXX.com nginx-service.default]
[DEBUG] plugin/k8s_gateway: Found 0 matching Ingress objects
[DEBUG] plugin/k8s_gateway: Found 1 matching Service objects
[DEBUG] plugin/k8s_gateway: Computed response addresses [10.100.11.105]
[INFO] 10.100.11.210:32824 - 3653 "A IN nginx-service.default.XXXXXXX.com. udp 93 false 4096" NOERROR qr,aa 138 0.000609899s
@koeppj
Copy link

koeppj commented May 8, 2024

I am experiencing the same issue.

@koeppj
Copy link

koeppj commented May 8, 2024 •
edited
Loading

More on this. Installed the experimental channel CRDs of the Gateway release 0.4.0 and issue resolved.

$kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/experimental-install.yaml

This installs the alpha CRDs for GPCRoute and TLSRoute. I think the plugin fails on the sync call if ALL referenced CRDs are not present (GPCRoute and TLSRoute are excluded in the standard channel).

@rwarford
Copy link
Author

rwarford commented May 8, 2024

More on this. Installed the experimental channel CRDs of the Gateway release 0.4.0 and issue resolved.

$kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/experimental-install.yaml

This installs the alpha CRDs for GPCRoute and TLSRoute. I think the plugin fails on the sync call if ALL referenced CRDs are not present (if is they are excluded in using resource type filter.

I think this must be the case. Seems like it should work with services and ingresses and just ignore routes if the CRDs aren't installed.

@gclawes gclawes mentioned this issue Jul 20, 2024
Closed
@Joker9944
Copy link

Maybe this can help anyone else coming looking for a solution. I only really needed the gateway to watch ingresses.networking.k8s.io. So I just restricted the ClusterRole with a Kustomize patch. This works around the issue at least for now.

https://github.com/Joker9944/k8s-config/blob/5f1ede42e9dbb8f12d9e642c6d7a357ead51a0cd/apps/nameserver-apps/blocky/helm-release.yaml#L215-L278

@larivierec
Copy link
Contributor

Maybe this can help anyone else coming looking for a solution. I only really needed the gateway to watch ingresses.networking.k8s.io. So I just restricted the ClusterRole with a Kustomize patch. This works around the issue at least for now.

https://github.com/Joker9944/k8s-config/blob/5f1ede42e9dbb8f12d9e642c6d7a357ead51a0cd/apps/nameserver-apps/blocky/helm-release.yaml#L215-L278

this is the right solution.
the chart should add rbac dynamically, however, it's static: (rbac)[https://github.com/ori-edge/k8s_gateway/blob/master/charts/k8s-gateway/templates/rbac.yaml#L28-L33]

another issue is you can't deactivate it to create your own RBAC with this chart.

LiquidPL added a commit to LiquidPL/infra that referenced this issue Oct 26, 2024
@LiquidPL
fix(external-dns): only allow controller to sync ingresses
3549a41 
otherwise this seems to break it because the cluster
does not have some experimental crds installed

see: ori-edge/k8s_gateway#279 (comment)
LiquidPL added a commit to LiquidPL/infra that referenced this issue Oct 26, 2024
@LiquidPL
fix(external-dns): only allow controller to sync ingresses
76ab46b 
otherwise this seems to break it because the cluster
does not have some experimental crds installed

see: ori-edge/k8s_gateway#279 (comment)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@larivierec @Joker9944 @koeppj @rwarford