Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

grpcJsonTranscoder Upstream is causing all the Routes to be on a warning state #9160

Closed
Tracked by #9157
edubonifs opened this issue Feb 15, 2024 · 8 comments
Closed
Tracked by #9157
Assignees
Labels
Area: Envoy activities related to envoy proxy Prioritized Indicating issue prioritized to be worked on in RFE stream release/1.15 Type: Bug Something isn't working

Comments

@edubonifs
Copy link

edubonifs commented Feb 15, 2024

Gloo Edge Product

Open Source

Gloo Edge Version

1.15.x

Kubernetes Version

1.26

Describe the bug

If we apply an Upstream with grpcJsonTranscoder, every RouteTable will report a Warning state:

status:
  statuses:
    gloo-system:
      reportedBy: gloo
      state: Accepted
      subresourceStatuses:
        '*v1.Proxy.gateway-proxy_gloo-system':
          reason: "1 error occurred:\n\t* Route Error: ProcessingError. Reason: *grpcjson.plugin:
            invalid route. Route Name: service-route-0-matcher-0;
            Route Error: ProcessingError. Reason: *grpcjson.plugin: invalid route.
            Route Name: service-route-0-matcher-0;
            Route Error: ProcessingError. Reason: *grpcjson.plugin: invalid route.
            Route Name: service-route-0-matcher-0;
            Route Error: ProcessingError. Reason: *grpcjson.plugin: inv"
          reportedBy: gloo
          state: Rejected

Expected Behavior

It should not harm every RouteTable

Steps to reproduce the bug

Apply this Upstream
upstream.yaml.zip

Additional Environment Detail

No response

Additional Context

No response

┆Issue is synchronized with this Asana task by Unito

@edubonifs edubonifs added the Type: Bug Something isn't working label Feb 15, 2024
@DuncanDoyle
Copy link
Contributor

DuncanDoyle commented Mar 28, 2024

Do we happen to have a full reproducer of this one? I tried to apply the attached upstream.yaml to my GE 1.15.14 instance, but I didn't observe any errors .... Would be useful to have:

  • K8S service (targeted by the Upstream)
  • RouteTable
  • VirtualService
  • etc.

@DuncanDoyle
Copy link
Contributor

Full reproducer here (instructions in README): https://github.com/DuncanDoyle/ge-gloo-9160

@DuncanDoyle
Copy link
Contributor

Seems this issue started happening in Gloo Edge 1.14.3 ...... I can't reproduce it in 1.14.2 and lower .....

@sam-heilbron
Copy link
Contributor

See: #7477 (comment)

This seems all related to https://github.com/solo-io/gloo/milestone/40?closed=1

@nfuden
Copy link
Contributor

nfuden commented Apr 10, 2024

Discovered so far:
The error we are seeing is coming from
https://github.com/solo-io/gloo/blob/v1.14.x/projects/gloo/pkg/plugins/grpcjson/plugin.go#L146
Which for whatever reason is not correctly detecting that the routeaction is of type single: https://github.com/solo-io/gloo/blob/v1.14.x/projects/gloo/pkg/plugins/pluginutils/destination_upstreams.go#L35

This starts occuring due to this 1.14 backport which attempted to make migration from old to new apis seemless 7845a04

This backport makes it easier to comingle the two options and have a clean cut over but something about the case we are seeing is leading to these erroneous warning messages being spewed.

@DuncanDoyle DuncanDoyle added Area: Envoy activities related to envoy proxy Prioritized Indicating issue prioritized to be worked on in RFE stream labels Apr 12, 2024
@sync-by-unito sync-by-unito bot closed this as completed Apr 23, 2024
@nfuden nfuden reopened this Apr 23, 2024
@solo-io solo-io deleted a comment from sync-by-unito bot Apr 23, 2024
@davidjumani
Copy link
Contributor

This issue only arises when any route defined on the listener is not a routeAction (eg: redirectAction, directResponseAction). The plugin code expects / only supports a routeAction. If any other type of route is passed, it results in a nil routeAction that in turn results in the observed error. In the steps to reproduce this, the new VS uses a redirect action route that leads this issue.
A simple fix can be to log that the plugin can not work on any other type of route rather than return an error. While this does not solve the overarching issue, it is a quick stopgap to unblock the user

I've created a PR for the fix here

@davidjumani
Copy link
Contributor

This fix will be in 1.17.0-beta1, v1.16.11 and v1.15.26 when released

@nfuden
Copy link
Contributor

nfuden commented May 7, 2024

v1.15.18 and v1.16.9

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: Envoy activities related to envoy proxy Prioritized Indicating issue prioritized to be worked on in RFE stream release/1.15 Type: Bug Something isn't working
Projects
None yet
Development

No branches or pull requests

6 participants