New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gRPC servers cannot use dynamic HPACK #2405

Closed
stevewolter opened this Issue Apr 23, 2018 · 8 comments

Comments

Projects
None yet
4 participants
@stevewolter

stevewolter commented Apr 23, 2018

NGINX Ingress controller version: 0.13.0
Kubernetes version (use kubectl version): v1.8.8-gke.0
Environment: GKE

What happened:

I ran a golang gRPC server behind an nginx ingress. Many (but not all) RPCs fail with 502. nginx logs show "upstream sent invalid http2 table index: 64 while reading response header from upstream".

What you expected to happen:

No error.

How to reproduce it (as minimally and precisely as possible):

(since this is an upstream bug, I'm not including full repro)

Anything else we need to know:

This is a bug in nginx (https://trac.nginx.org/nginx/ticket/1538#ticket), I'm posting a bug here for reference.

@aledbf aledbf added the bug label Apr 23, 2018

@aledbf

This comment has been minimized.

Member

aledbf commented Apr 23, 2018

@stevewolter thank you for the report

@aledbf

This comment has been minimized.

Member

aledbf commented Apr 24, 2018

Closing. Please check response in nginx ticket

@aledbf aledbf closed this Apr 24, 2018

@mumoshu

This comment has been minimized.

mumoshu commented May 9, 2018

@stevewolter @aledbf Hi. You concluded this to be a bug in nginx. JFYI but it seems like one in grpc-go.

nginx explicitly announces that it does not support dynamic header compression by sending the SETTINGS_HEADER_TABLE_SIZE value set to 0, see ​here. Any attempt of an upstream server to use indexes from the dynamic range is a bug in the upstream server
https://trac.nginx.org/nginx/ticket/1538#ticket

We already have an issue in grpc/grpc-go#1928, and the PR is in-p at grpc/grpc-go#2045

That being said, using a grpc-go built from the PR head would solve this issue.

Anyway, this is a bug outside of ingress-nginx. I just wanted to leave a note 😉

@stevewolter

This comment has been minimized.

stevewolter commented May 14, 2018

Thanks for the pointers, much appreciated.

@aledbf

This comment has been minimized.

Member

aledbf commented Aug 10, 2018

There is a workaround that fixes this issues https://trac.nginx.org/nginx/ticket/1397#comment:13
(I hope NGINX 1.15.3 will include the fix)

@keatz55

This comment has been minimized.

keatz55 commented Sep 5, 2018

Any updates on this? I am trying to use ArgoCD behind some ingress rules and this is preventing me from using the ArgoCD CLI which depends on GRPC.

@aledbf

This comment has been minimized.

Member

aledbf commented Sep 5, 2018

@keatz55 please use 0.19.0. That version already uses NGINX 1.15.3

@keatz55

This comment has been minimized.

keatz55 commented Sep 5, 2018

@aledbf I see that the change made it into the 1.15.3 release and I am using 0.19.0, however, I am still getting the following error while trying to use GRPC:

[error] 1787#1787: *403612 upstream sent invalid http2 table index: 64 while reading response header from upstream
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment