Question/documentation: GRPC-JSON transcoder and grpc ext_authz filter (envoy control plane ext_authz ) integration

[2dev2]

Title: GRPC-JSON transcoder is ignoring ext_authz filter, can we have documentation/example updates in which both filter are working together

Description:

we are Using envoy latest (tag-v1.30.1 ) as front proxy (client submit request as http) and configured the grpc-json and ext_auth filter (envoy control plane Check https://github.com/envoyproxy/go-control-plane/blob/main/envoy/service/auth/v3/external_auth.pb.go ) , as Individual filter are working alone.
But once They are used together, means authentication delegate to ext_auth and once authentication done request submitted to grpc service then Its not working ( faced major 3 scenario)

1. request submitted to grpc service without authentication (call not reaching out to ext_auth filter)
2. 501 error code (grpc service Not implemented error)
3. 415 error code (mismatch json content-type and grpc service in upstream)
   We have gone through several issue/help posted for this: [ https://github.com/ext_authz is ignored for paths matching a grpc_json_transcoder filter #9929, https://github.com/Incorrect behavior with Routing based on JWT Token Dynamic Metadata #19910]

** Expectation:** in envoy ext_authz control plane authenticate and pass the CheckResponse to upstream grpc-service.
how to apply missing envoy-config or how we can resolve the 501 error code ( as auth is passing request to grpc but not in correct url )

Based on suggestion Tried Solution/steps :

for solving 1st issue ( without auth) : we have updated the filters order, grpc-json filter before the ext auth
Then we have got the second issue. { "code": 12, "message": "unknown method RPC_METHODE_NAME for service proto.package.RPC_SERVICE_NAME”, "details": [] } HTTP error code as (501 Not implemented error)

for solving second issue:
we tried multiple thing like
2.a). Trying not clearing the the route cache, clear_route_cache
2.b) setting header in ext_auth
LUA print
/ proto.package.RPC_SERVICE_NAME/RPC_METHODE_NAME?a=3
Envoy log prints
{"envoy.upstream.cluster":"grpc_cluster","http.request.method":"POST","client.local.address":"172.18.0.5:80","host.hostname":"a8dc1e8353de”,,”service.version":"1.28","envoy.route.name":null,"http.request.path”:”/http_reuest_url/xyz/abc?a=3&b=4","http.request.headers.authority":"local.api.xyz.com","http.request.start_time":"2024-04-20T15:58:31.357Z","http.request.headers.x_forwarded_proto":"http","http.request.duration":1,"http.response.response_code":501,"http.request.body.bytes":0,"http.response.body.bytes":199,"http.response.response_flag”:”-“,”http.request.headers.x_forwarded_for":null,"http.request.protocol":"HTTP/1.1","service.name":"envoy","http.request.headers.accept":"/",}

for solving 1st issue/2nd issue : When we added the one intermediate path mentioned on #9929 we have got the 415 http error code in client (grpc-message as content-type recieved to grpc was content-type json) through ext_auth

Listener envoy config:

envoy 1.30 listener config: [web_lb_grpc2 .txt](https://github.com/envoyproxy/envoy/files/15049407/web_lb_grpc2.txt)

Any extra documentation required to understand the issue.
Please help us to resolve the workflow and later i am feeling we can add one example for this common usecase to envoy example repo as end to end grpc integration in envoy.

[wbpcode]

cc @nareddyt

[nareddyt]

gRPC transcoder with ext_authz should work well, I believe I personally ran a similar configuration in the past.

Your config LGTM. I noticed the following:

• The router is only configured with gRPC routes and not HTTP routes. I believe this is OK, as the transcoder doesn't read route config and does it's own path matching on the HTTP request headers.
• The transcoder comes before ext_authz. This is also good, you should transcode requests before doing authz to prevent authz bypass.

To help further debug, please provide the following details:

1. What is the HTTP request your client is sending? I see POST /http_reuest_url/xyz/abc?a=3&b=4 with 0 body bytes, which I am suspicious about. gRPC transcoder might be expecting a GET request. Please share the full HTTP request (curl command would be nice) and the HTTP annotation on your gRPC service definition.
2. Run envoy with --log-level debug and share the application logs. gRPC transcoder outputs many logs which would help debug here.
3. Share your full Envoy config (including clusters) as text please, it's hard to parse the VS Code image.

2.a). Trying not clearing the the route cache, clear_route_cache

FYI gRPC transcoder will automatically clear the route cache once it translates the HTTP request unless match_incoming_request_route=true. Let's debug with the default behavior where match_incoming_request_route=false. Your route config looks correct for this case.

[2dev2]

@nareddyt Thanks for looking out, from long time we got stuck on this.
if possible please suggest further.

1. Attaching the client details

curl request here: curl --location 'local.api.com/sms_grpc/v1/sms/v1/api/v1/org/sendsms?message=hello&senderId=fg&number=6778&templateId=123'
--header 'X-API-KEY: abcdefgh'.

which is able to match the envoy grpc service and without envoy ext_auth and it gives response.

grpc service anotation: service SmsApiService {
rpc SendSmsExternalGETV1(SendSmsExternalGETV1Request) returns (SendSmsExternalGETV1Response){
option (google.api.http) = {
get: "/sms_grpc/v1/sms/v1/api/v1/org/sendsms"
};
}
}
client response in postman:

2 envoy debug log and screen shot where we are able to get the details its making POST API call from inside of ext_auth_filter,
envoy_debuglog.txt

3. i am running envoy with dynamic configuration sharing listener and cluster config:
   cds.txt
   lds.txt
   bootstrap_envoy_dynamic.txt

[nareddyt]

Thanks for providing the logs and configuration files, that gave me everything I needed to debug the issue. I'll walk you through it.

Logs show the request is following the request path correctly:

Client --> grpc transcoder --> ext_authz --> upstream cluster new_api_grpc_cluster

In fact, the HTTP 501 is from your own backend. Notice the response has header x-envoy-upstream-service-time which indicates it tried reaching out to your backend (upstream).

[2024-04-25 22:21:48.961][15][debug][http] [source/common/http/conn_manager_impl.cc:1838] [Tags: "ConnectionId":"0","StreamId":"18191014628947526340"] encoding headers via codec (end_stream=false):
':status', '501'
'content-type', 'application/json'
'x-envoy-upstream-service-time', '1'
'content-length', '167'
'date', 'Thu, 25 Apr 2024 22:21:48 GMT'
'server', 'envoy'
'x-request-id', '3241b8e0-c1eb-46d8-8f1d-fce751fbad69'

Obviously I don't have your backend to verify this. But you can try making the same request Envoy makes directly to your backend (host.docker.internal:8897, assuming you expose that container port) and observe the response. Request:

[2024-04-25 22:21:48.960][15][debug][router] [source/common/router/router.cc:738] [Tags: "ConnectionId":"0","StreamId":"18191014628947526340"] router decoding headers:
':scheme', 'http'
':method', 'POST'
':path', '/edugo.api.services.sms.v1.SmsApiService/SendSmsExternalGETV1?org_id=5b00abbab9'
':authority', 'local.api.com'
'x-api-key', 'abcdefgh'
'x-org-id', '5b00abbab9'
'x-forwarded-proto', 'http'
'postman-token', 'ec105acf-c79b-4ffd-930d-4a07d9480a79'
'te', 'trailers'
'x-envoy-auth-partial-body', 'false'
'x-envoy-original-method', 'GET'
'x-envoy-original-path', '/sms_grpc/v1/sms/v1/api/v1/org/sendsms?message=hello&senderId=fg&number=6778&templateId=123'
'x-request-id', '3241b8e0-c1eb-46d8-8f1d-fce751fbad69'
'content-type', 'application/grpc'
'x-envoy-expected-rq-timeout-ms', '60000'

I do have a guess for why your backend is sending back HTTP 501. Take a look at the HTTP path that envoy sends to your backend:

':path', '/edugo.api.services.sms.v1.SmsApiService/SendSmsExternalGETV1?org_id=5b00abbab9'

IIUC the query parameter ?org_id=5b00abbab9 in the path is not compliant with the gRPC over HTTP2 specification. All metadata you wish to forward to gRPC server should be as gRPC metadata, i.e. lowercase HTTP headers like x-api-key and x-org-id.

Why is Envoy sending this query parameter to the backend? It is because of the ext_authz response from your ext_authz server:

status {
}
ok_response {
  headers {
    header {
      key: "X-ORG-ID"
      value: "5b00abbab9"
    }
  }
  headers {
    header {
      key: ":scheme"
      value: "http"
    }
    keep_empty_value: true
  }
  headers {
    header {
      key: ":method"
      value: "POST"
    }
    keep_empty_value: true
  }
  headers {
    header {
      key: ":path"
      value: "/edugo.api.services.sms.v1.SmsApiService/SendSmsExternalGETV1"
    }
    keep_empty_value: true
  }
  headers {
    header {
      key: "x-forwarded-proto"
      value: "http"
    }
    keep_empty_value: true
  }
  headers {
    header {
      key: "postman-token"
      value: "ec105acf-c79b-4ffd-930d-4a07d9480a79"
    }
    keep_empty_value: true
  }
  headers {
    header {
      key: "te"
      value: "trailers"
    }
    keep_empty_value: true
  }
  headers {
    header {
      key: "x-envoy-auth-partial-body"
      value: "false"
    }
    keep_empty_value: true
  }
  headers {
    header {
      key: ":authority"
      value: "local.api.com"
    }
    keep_empty_value: true
  }
  headers {
    header {
      key: "x-envoy-original-method"
      value: "GET"
    }
    keep_empty_value: true
  }
  headers {
    header {
      key: "x-envoy-original-path"
      value: "/sms_grpc/v1/sms/v1/api/v1/org/sendsms?message=hello&senderId=fg&number=6778&templateId=123"
    }
    keep_empty_value: true
  }
  headers {
    header {
      key: "x-request-id"
      value: "3241b8e0-c1eb-46d8-8f1d-fce751fbad69"
    }
    keep_empty_value: true
  }
  headers {
    header {
      key: "content-type"
      value: "application/grpc"
    }
    keep_empty_value: true
  }
  headers_to_remove: "apikey"
  query_parameters_to_set {
    key: "org_id"
    value: "5b00abbab9"
  }
}
dynamic_metadata {
  fields {
    key: "x-edu-org-id"
    value {
      string_value: "5b00abbab9"
    }
  }
}

The following part should be removed, as it is not compliant with gRPC over HTTP2:

  query_parameters_to_set {
    key: "org_id"
    value: "5b00abbab9"
  }

In general, i don't recommend you rewrite the entire request via the ext_authz response. Just add on the headers you need, like x-org-id, and omit the rest of the fields.

[nareddyt]

TLDR - there is no issue with gRPC JSON transcoder or ext_authz filter integration. It is due to malformed ext_authz response causing the user's backend to respond with 501.

[2dev2]

i have removed query_parameters_to_set and make path as grpc compliant and its working fine. Thanks a lot, for giving the details insight and suggestion.

We have used query params to use this feature of grpc-gateway: https://github.com/googleapis/googleapis/blob/master/google/api/http.proto#L223
Means if we pass some field as query params and they are mentioned/member of in body proto struct, it will be filled with passed query data.