gRPC call via nginx ingress missing metadata

[RagingPuppies]

Hi,
I've been struggling with this for couple of days now,
while using nginx-ingress with http\https calls i was able to get all the http headers been passing trough the nginx-ingress-controller (just did a simple post and echo the received headers)
But while using gRPC calls and supplying a metadata as following (using bloomRPC) :
{"test_header":"value123"}
I'm unable to receive it in the pod side. the flow works but all metadata (aka headers) are gone, Is there a special configuration needed to forward the metadata via the nginx-ingress-controller?

using the same call directly to pod with nodeport works and all metadata is valid.
and if there is no special option i should activate, is there a way to debug this in the controller side? i can only see the calls in the log but not the headers.

i'm using the following ingress service yml (Rancher2)(Image: rancher/nginx-ingress-controller:nginx-0.25.1-rancher1)

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
field.cattle.io/creatorId: u-zmb65
field.cattle.io/ingressState:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: GRPC
nginx.ingress.kubernetes.io/ssl-redirect: "true"
creationTimestamp: "2020-04-07T13:47:11Z"
generation: 1
labels:
cattle.io/creator: norman
name:
namespace: ingress-test
resourceVersion: "1950374"
selfLink:
uid: 0ecb2cc1-8fe9-474b-8d37-863a2e5c8f8d
spec:
rules:

• host:
  http:
  paths:
  • backend:
    serviceName:
    servicePort:
    path: /
    tls:
• hosts:
  secretName:
  status:
  loadBalancer:
  ingress:
  • ip:
  • ip:

Thanks a lot

[leonjalfon1]

Your issue seems to be that your custom headers are not passed to the application by the nginx proxy (ingress controller). The issue is not directly related to gRPC, it's just related to the nginx proxy configuration.

You have several options to fix it:

1) Pass the custom headers using ingress annotations

To add those custom headers you can use the annotation "nginx.org/proxy-pass-headers" (more info here: https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/#request-uri-header-manipulation)

2) Pass the custom headers using a configmap

Another approach is to configure the nginx ingress controller via a ConfigMap to pass a custom list of headers to the upstream server: https://kubernetes.github.io/ingress-nginx/examples/customization/custom-headers/

3) Set the custom headers from the ingress resource

You can see an example here: https://linkerd.io/2/tasks/using-ingress/#nginx

[RagingPuppies]

Thank you @leonjalfon1 , but this did not resolved the issue,
the annotation: proxy-pass-headers is related to http request, only modules starts with grpc_* are related to GRPC requests, as i mentioned the HTTP test passed with no further configurations and the reason was that i did not used the same header name.
my issue here was:
after setting debug on the ingress-controller with - --v=5
I received the following message:

client sent invalid header: "header_name" while reading client request headers

nginx set the header as invalid due to the "_" and does not pass it forward..
adding the following to the ingress solves the issue:
nginx.ingress.kubernetes.io/server-snippet: |
underscores_in_headers on;
ignore_invalid_headers on;

Thanks for the help!

[psanders]

@RagingPuppies

Thanks a lot! That help me with the issue.