Gateway API on EKS not working as expected #27493
Labels
feature/k8s-gateway-api
kind/bug
This is a bug in the Cilium logic.
kind/community-report
This was reported by a user in the Cilium community, eg via Slack.
sig/agent
Cilium agent related.
sig/datapath
Impacts bpf/ or low-level forwarding details, including map management and monitor messages.
Comments
Smana
added
kind/bug
|
I just tried again in order to try that with Amazon LInux instances (kernel version: 5.10.184-175.749.amzn2.x86_64). I got exactly the same behavior. The gateway is not properly configured, the status Programmed=False. |
ti-mo
added
sig/datapath
2 tasks
|
Same behaviour. Created dupe issue for this. Here is a resources and logs Gateway with status Service And some logs from cilium operator |
|
@Smana I find another workaround on this issue. You need to slightly tune policy that you using (afaik also written by me) |
2 tasks
|
@sergeyshevch Cool! I'm gonna give a try right now. Just curious: how did you find this option, I've never heard about it. |
|
Well, it doesn't seem to work on my side even though I defined the loadBalancerClass. apiVersion: v1
kind: Service
metadata:
annotations:
external-dns.alpha.kubernetes.io/hostname: echo-mycluster-0.cloud.ogenki.io
policies.kyverno.io/last-applied-patches: |
mutate-svc-annotations.mutate-cilium-gateway-echo-gateway.kyverno.io: added /metadata/annotations
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
labels:
io.cilium.gateway/owning-gateway: echo-gateway
name: cilium-gateway-echo-gateway
namespace: echo
spec:
allocateLoadBalancerNodePorts: true
clusterIP: 172.20.58.62
clusterIPs:
- 172.20.58.62
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
loadBalancerClass: service.k8s.aws/nlb
ports:
- name: port-80
nodePort: 30131
port: 80
type: LoadBalancerI have this error in the cilium-operator logs: cilium-operator-76c4f97d54-mtlbg cilium-operator level=error msg="Unable to create Service" controller=gateway error="Service \"cilium-gateway-echo-gateway\" is invalid: spec.loadBalancerClass: Invalid value: \"null\": may not change once set" resource=echo/echo-gateway subsys=gateway-controller |
|
I managed to get it work using only Classic LB. I had to configure the AWS LoadBalancer Controller accordingly with the option kubectl get svc -n echo cilium-gateway-echo-gateway -o yaml | head -n 10
apiVersion: v1
kind: Service
metadata:
annotations:
external-dns.alpha.kubernetes.io/hostname: echo-mycluster-0.cloud.ogenki.io
policies.kyverno.io/last-applied-patches: |
mutate-svc-annotations.mutate-cilium-gateway-echo-gateway.kyverno.io: added /metadata/annotations
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
...GATEWAY=$(kubectl get gateway echo-gateway -n echo -o jsonpath='{.status.addresses[0].value}')
curl http://$GATEWAY |jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 2491 100 2491 0 0 101k 0 --:--:-- --:--:-- --:--:-- 105k
{
"host": {
"hostname": "aa4ae59f99b1b4d20b74f35a06aa7ab7-228497968.eu-west-3.elb.amazonaws.com",
"ip": "::ffff:10.0.15.188",
"ips": []
},
"http": {
"method": "GET",
"baseUrl": "",
"originalUrl": "/",
...Well I don't know what's the recommendations here, as the Classic LB are deprecated, I guess I have to wait for this issue to be addressed. Note: It also works fine with the https example. |
|
@Smana It works with my workaround policy. But you need to apply policy first and then recreate the gateway (or just underlying service) |
|
@sergeyshevch thank you I just tried one more time and indeed that works fine. thank you! |
2 tasks
|
Did anyone managed to get http to https redirect working with the configuration mentioned in this issue? I tried the redirect http route and getting infinite redirect. Any ideas? |
|
@lucidprogrammer , I think you're facing this issue kubernetes-sigs/gateway-api#1185 |
2 tasks
2 tasks
bdalpe
added a commit
to bdalpe/cilium
that referenced
this issue
Mar 1, 2024
Related to the comment here: cilium#30038 (comment) If the Service contains either label indicating that it is managed by Cilium, return all nodes as LoadBalancer IPs, or filter the nodes based on an Annotation applied to the Service `io.cilium.nodeipam/match-labels`. Otherwise, leaves original Node IPAM LB behavior intact for other non-Cilium managed Services. Services for Gateways still need to have a mutating webhook apply the loadBalancerClass value, as there is not currently a way to do this with the Gateway config. See cilium#27493. Signed-off-by: Brendan Dalpe <bdalpe@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is there an existing issue for this?
What happened?
I'm not able to configure the gateway.
My Cilium configuration seems ok:
I followed this guide and when I create the gateway along with the httproute it creates a loadbalancer:
The HTTP route seems to be properly configured:
However, the gateway never gets ready:
Even though the curl command reaches the envoy service, it returns a 404:
Am I missing something on EKS? Note that I'm using a Kyverno workaround to set the annotations properly.
Cilium Version
1.14.0
Kernel Version
I'm not able to ssh to the instance right now but it's the latest bottlerocket AMI. I'll try to enable SSM.
Kubernetes Version
1.27
Sysdump
cilium-sysdump-20230814-183832.zip
Relevant log output
No response
Anything else?
I'm writing a blog post about Cilium with Gateway API on EKS. My code is here if you want to look at it.
Code of Conduct
The text was updated successfully, but these errors were encountered: