How to Include GRPC Status Codes in Envoy Access Logs?

[TanOfAllCodes]

Hello, I'm relatively new to Istio and I would like a feature where the istio-proxy logs are able to show the GRPC status codes.
Now my dilemma here is that Envoy does not make it clear as to how to add the GRPC status codes to the Format String - HTTP and TCP are documented.

https://www.envoyproxy.io/docs/envoy/latest/configuration/access_log#format-rules
In this link ^ format string I want to be able to use GRPC status codes.

https://preliminary.istio.io/docs/tasks/telemetry/logs/access-log/
This link tells me that I need the change the values file but I am unsure of how the values file should be like, and how the config map should be edited.

How to Include GRPC Status Codes in Envoy Access Logs?

Is there a simple way to do this?

The end goal is to use the GRPC status code in the logging and monitoring setup - the requirement from the team is to have the status code displayed in the logs.

Config File included for reference

https://raw.githubusercontent.com/istio/istio/master/install/kubernetes/helm/istio/templates/configmap.yaml

Describe alternatives you've considered

Affected product area (please put an X in all that apply)

[X] Configuration Infrastructure
[X] Docs
[X] Installation
[X] Networking
[X] Performance and Scalability
[X] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

Additional context

[mandarjog]

@douglas-reid @kyessenov this is related to the grpc resp code metric.

[TanOfAllCodes]

@mandarjog @douglas-reid @kyessenov How do I use the GRPC response code metric in this particular case?

[hanoisteve]

Have you asked on the envoyproxy.slack.com channel?

[TanOfAllCodes]

@hanoisteve I dont have access to it...

[hanoisteve]

Ok, I asked them. I will see what they say. I agree GRPC support is high priority.

[TanOfAllCodes]

@hanoisteve @crockeo @junr03 @mattklein123
I guess this issue relates to
envoyproxy/envoy#5682

How do I apply this into my configuration?

[samush]

Hello, may be it will help you:

My environment: openshift(3.11)+istio(maistra project)+app(front http + backend grpc microservices)

I was looking for configuration options to return http response code relative to grpc status code and here that works for me:

istio-system project:
#here i found resource kind metric, but in Istio docs thay talk about Instances
$ oc get metrics -n istio-system --no-headers
requestcount 3d <------------interest for me
requestduration 42d
requestsize 42d
...and so on.....

requestcount manifest:
oc get metrics -n istio-system requestcount -o yaml
....
spec:
dimensions:
...
reporter: conditional((context.reporter.kind | "inbound") == "outbound", "source",
"destination")
request_protocol: api.protocol | context.protocol | "unknown"
response_code: response.code | 200 <------------interest for me
response_flags: context.proxy_error_code | "-"
....

after checking docs i found list of attributes that can produce envoy:
https://istio.io/docs/reference/config/policy-and-telemetry/attribute-vocabulary/

i append this field to spec.demensions
oc get metrics -n istio-system requestcount -o yaml
....
spec:
dimensions:
...
reporter: conditional((context.reporter.kind | "inbound") == "outbound", "source",
"destination")
request_protocol: api.protocol | context.protocol | "unknown"
response_code: response.code | 200
response_flags: context.proxy_error_code | "-"
response_grpc_status: response.grpc_status | "no_status" <------------here

after that in mixer logs some warning and error logs happend , but after that i also edit one more thing

$ oc get handlers -n istio-system
NAME AGE
doublehandler 4d
kubernetesenv 42d
prometheus 42d <--------------here

oc edit handlers prometheus -n istio-system
....
spec:
compiledAdapter: prometheus
params:
metrics:
- instance_name: requestcount.metric.istio-system
kind: COUNTER
label_names:
- reporter
- source_app
- source_principal
- source_workload
- source_workload_namespace
- source_version
- destination_app
- destination_principal
- destination_workload
- destination_workload_namespace
- destination_version
- destination_service
- destination_service_name
- destination_service_namespace
- request_protocol
- response_code
- response_grpc_status <-----------------append this field
- response_flags
- permissive_response_code
- permissive_response_policyid
- connection_security_policy
name: requests_total
- buckets:

after that in prometheus server in istio-system project i get new label for metric
istio_requests_total{response_grpc_status=~".*"}

after that we change metrics response_code demension with this expression
oc edit metrics requestcount -n istio-system -o yaml
...
#we want to map grpc status code to http code
#when grpc_status_code="0" conditional expression return allways false
#then we cat string "0" with random string "resp"
#import to remember that response_grpc_status return string

and response_code expects int64

..
response_code: >-
conditional(((response.grpc_status | "unknown")+"resp"=="0resp"), 200,
(conditional((response.grpc_status | "unknown") ==
"1",499,(conditional((response.grpc_status | "unknown") ==
"2",500,(conditional((response.grpc_status | "unknown")
=="3",400,(conditional((response.grpc_status | "unknown")
=="4",504,(conditional((response.grpc_status | "unknown")
=="5",404,(conditional((response.grpc_status | "unknown")
=="6",409,(conditional((response.grpc_status | "unknown")
=="7",403,(conditional((response.grpc_status | "unknown")
=="8",429,(conditional((response.grpc_status | "unknown")
=="9",400,(conditional((response.grpc_status | "unknown")
=="10",409,(conditional((response.grpc_status | "unknown")
=="11",400,(conditional((response.grpc_status | "unknown")
=="12",501,(conditional((response.grpc_status | "unknown")
=="12",501,(conditional((response.grpc_status | "unknown")
=="13",500,(conditional((response.grpc_status | "unknown")
=="14",503,(conditional((response.grpc_status | "unknown")
=="15",500,(response.code | 200))))))))))))))))))))))))))))))))))

mapping grpc to http we found here
https://github.com/grpc/grpc/blob/master/doc/http-grpc-status-mapping.md

this all work for us, hope it help you

Istio

Attribute Vocabulary

Describes the base attribute vocabulary used for policy and control.

GitHub

grpc/grpc

The C based gRPC (C++, Python, Ruby, Objective-C, PHP, C#) - grpc/grpc

[istio-policy-bot]

🚧 This issue or pull request has been closed due to not having had activity from an Istio team member since 2019-07-08. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions.

Created by the issue and PR lifecycle manager.