-
Notifications
You must be signed in to change notification settings - Fork 590
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Could not unmarshal config error #5676
Comments
@ksgnextuple What image and version of Gateway are you running? I believe the problem you're facing has been solved in #5638 in KIC 3.1.1 (if you're using an open source Kong) |
Hi @pmalek Kong gateway -> kong:3.6 Have used the helm installation -> helm install kong/kong --generate-name --set ingressController.installCRDs=false -n kong --create-namespace |
Upgrading to KIC 3.1.1 should fix your issue. |
Even after updating the image to -> kong/kubernetes-ingress-controller:3.1.1. Seeing the same error |
Any updates on this? |
@ksgnextuple Can you try following this guide https://docs.konghq.com/kubernetes-ingress-controller/latest/reference/troubleshooting/#dumping-generated-kong-configuration to get the configuration that failed to be applied? Specifically the one from This way we'll be able to progress with this knowing what config are we working with. |
I think the issue is resolved. I deleted the httproute and Gateway and recreated it after upgrading the KIC version to 3.1.1. Will get back here after few tests. |
Hmm, on further checking it's still the same, will try the troubleshooting documentation. |
kind: HTTPRoute
When I add the 2nd backendref get the error |
|
Are the Services in question both using the same selectors? Do they have the same endpoints if you check I (arbitrarily) tried this with two test Services that were identical in all but name, so I have
That yields
which is rejected with:
From the target list in the dumped config here, it roughly looks like that's the same pattern on your side. We presumably need to de-duplicate targets. On the error front, FTI-5584 was followed by FTI-5813 internally, though it looks like the first should have fixed all uniqueness constraint errors being unreadable--I'm not sure if FTI-5813 is just some other type of un-parseable error. We should probably just log the actual body or generate an Event in our own namespace with a text dump of those. |
Let me try it again, but when I actually do the kubectl argo rollout promote, things start to work. That is when the canary actually has an upstream. |
I'm also seeing this same issue in what appears to be the same use case as @ksgnextuple, using KIC + Gateway API to enable canary deployments with Argo rollouts. In my environment I have KIC 3.1.1, Kong Gateway 3.6.1.1, and k8s 1.28.
@rainest Following this Kong + Argo Rollouts example (step 5), yes. I'm not an expert on Argo Rollouts but as I understand it, the rollout controller will inject additional labels in to the Service's selector during a rollout. So during a rollout, the selectors will be different, but before a rollout has started or after it has completed the selectors will be the same. |
@ksgnextuple @congiv I've uploaded Can you try the affected rollout with it to confirm if they behave correctly? Is configuration accepted? Do you observe any unexpected distribution of traffic during the rollout? |
Thanks @rainest. I tried your image in my environment and it seems to be working. I don't see any error logs about being unable to update the routing config as I did before. I just tested quickly by having a Rollout with that increases traffic to the canary by 25% every 2 mins, and curling my service every 5s. Not that much data to go off for the weight, but it appears to be within reason for what I'd expect. I can set up a more involved test where I'm sending more traffic to a canary running over a longer period of time to get better info about the weight if that would help. The only thing that jumped out at me is that I was unable to visualize the progress of traffic shifting using prom metrics from Kong. I thought that I'd be able to look at the rate of |
That's expected. Backends are aggregated under a single route; distribution to the different endpoint sets is handled in the Kong upstream resource attached to that route via the Kong service. Unless you're tracking requests per upstream IP (which AFAIK isn't something that our stock metrics track) you won't see the split. |
#5817 is now merged, so going to go ahead and close this. It's not yet actually released, but will be included in 3.2 (or possibly another 3.1.x patch release--we don't have any immediate plans to release another, but might. |
Is there an existing issue for this?
Current Behavior
I have created a httproute object and integrating it with argo rollouts. Once I create the httproute and the argo rollout resource getting the below error in kong ingress controller container logs -
I have not created any plugins or consumers & there is only 1 httproute resource in the entire cluster. Below is the yaml of the resource
Expected Behavior
No response
Steps To Reproduce
No response
Kong Ingress Controller version
Kubernetes version
Anything else?
No response
The text was updated successfully, but these errors were encountered: