Nginx is supported out of box

[jmzwcn]

Nginx is supported out of box, the config is very simple!

server {
    listen 80;
    
    ssl_certificate /etc/fullchain.cer;
    ssl_certificate_key /etc/iyou.city.key;

    index index.html index.htm;
    
    location / {
        try_files $uri $uri/ /index.html;
    }

    # api
    location /zwan {
        grpc_set_header Content-Type application/grpc;
        grpc_pass grpc://zwan:50051;
    }
}

Keypoints:

1. https: Ensure grpc-web send http2 request.
2. grpc_set_header Content-Type application/grpc in nginx.conf;
3. mode is grpcweb when generate-js:

More example, please refer to: https://github.com/emart-io/cross/blob/main/pwa/nginx.conf

[sampajano]

@jmzwcn Thanks for sharing! Glad to hear it works on Nginx! :)

Sharing a few related links i found here:

Announcements:
https://www.nginx.com/blog/nginx-1-13-10-grpc/

API:
http://nginx.org/en/docs/http/ngx_http_grpc_module.html

Code:
https://github.com/nginx/nginx/blob/master/src/http/modules/ngx_http_grpc_module.c

[steeve]

Awesome! Is there any info for grpc-web support ?

[sampajano]

@steeve hi :) there's no official support for Nginx from grpc-web side. But if the above mentioned modules work, then it'll be your freedom to use :)

[mkonir0]

I can't get it running 🤔 , our grpc-java service is throwing the following exception
StatusRuntimeException: INTERNAL: gRPC frame header malformed: reserved bits not zero

we have the following setup:
grpc-web -> nginx -> grpc-java service

request from grpc-web looks like this

curl 'http://localhost:9091/com.personal.test.PrivateService/SaveCustomer' \
  -H 'Accept: application/grpc-web-text' \
  -H 'Accept-Language: en-GB,en-US;q=0.9,en;q=0.8' \
  -H 'Content-Type: application/grpc-web-text' \
  -H 'Origin: https://test.com' \
  -H 'Proxy-Connection: keep-alive' \
  -H 'Sec-Fetch-Dest: empty' \
  -H 'Sec-Fetch-Mode: cors' \
  -H 'Sec-Fetch-Site: cross-site' \
  -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36 (Cypress)' \
  -H 'X-Cypress-Is-XHR-Or-Fetch: xhr' \
  -H 'X-Grpc-Web: 1' \
  -H 'X-User-Agent: grpc-web-javascript/0.1' \
  -H 'sec-ch-ua;' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: ""' \
  --data-raw 'AAAAACgKJmdpdmVuIGEgc2ltcGxlIGhpZXJhcmNoeSB3aXRoIGN1c3RvbWVy' \
  --compressed

nginx(running in docker) has following config

events { }

http {
    server {
        listen 9091 http2;
        location / {
          grpc_pass grpc://host.docker.internal:9090;
          grpc_set_header Content-Type "application/grpc";
          grpc_set_header Proxy-Connection "";
          grpc_set_header x-forwarded-proto http;
          grpc_set_header grpc-accept-encoding identity;
    }
  }
}

grpc-java service(running on port 9090) is throwing following:

{"ts":"2023-09-11T19:57:16.186+02","logLevel":"DEBUG","thread":"grpc-nio-worker-ELG-3-5","logger":"io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler","message":"[id: 0xb46b7519, L:/127.0.0.1:9090 - R:/127.0.0.1:59453] INBOUND DATA: streamId=1 padding=0 endStream=true length=84 bytes=4141414141446f4b476d6470646d567549484e305a58416764326c306143427759584a68625756305a584a7a4567304b42485a68624445534257467a5a475a6e..."}

{"ts":"2023-09-11T19:57:16.189+02","logLevel":"WARN","thread":"grpc-nio-worker-ELG-3-5","logger":"io.grpc.netty.shaded.io.grpc.netty.NettyServerStream","message":"Exception processing message","stacktrace":"i.g.StatusRuntimeException: INTERNAL: gRPC frame header malformed: reserved bits not zero\n\tat io.grpc.Status.asRuntimeException(Status.java:529)\n\tat i.g.i.MessageDeframer.processHeader(MessageDeframer.java:382)\n\tat i.g.i.MessageDeframer.deliver(MessageDeframer.java:272)\n\tat i.g.i.MessageDeframer.request(MessageDeframer.java:162)\n\tat i.g.i.AbstractStream$TransportState$1RequestRunnable.run(AbstractStream.java:233)\n\tat i.g.n.s.i.g.n.NettyServerStream$TransportState$1.run(NettyServerStream.java:188)\n\tat i.g.n.s.i.n.u.c.AbstractEventExecutor.runTask$$$capture(AbstractEventExecutor.java:174)\n\tat i.g.n.s.i.n.u.c.AbstractEventExecutor.runTask(AbstractEventExecutor.java)\n\tat i.g.n.s.i.n.u.c.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)\n\tat i.g.n.s.i.n.u.c.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)\n\tat i.g.n.s.i.n.c.n.NioEventLoop.run(NioEventLoop.java:566)\n\tat i.g.n.s.i.n.u.c.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat i.g.n.s.i.n.u.i.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat i.g.n.s.i.n.u.c.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.lang.Thread.run(Thread.java:833)\n"}

Seems like nginx is not able to translate the grpc-web request to valid http2 header frame binary format - it's missing the reserved bits (INBOUND DATA: streamId=1 padding=0 endStream=true length=84 bytes=4141414....)

[jmzwcn]

Don't use curl request, it don't follow grpc-web spec, PLEASE use grpc-web client send request. About how to request, you can refer to grpc-web doc.

[jmzwcn]

If you are not sure to how to grpc_set_header, please use example's setting.

[mkonir0]

I'm using grpc-web. Mentioned curl is just copied request generated by grpc-web (taken from browser)

this is simplified version of what I'm doing in client

        //client generated from proto file
        client = new ServiceClient('http://localhost:9091')
        //CustomerRequest class is generated from proto file
        const req = new CustomerRequest();
        req.setCustomerName('example');
        if (customerParameters.size > 0) {
            customerParameters.forEach((v, k) => req.getCustomerParameters().set(k, v));
        }
        client.saveCustomer(req, {}, (err, response) => {
            if (err) {
                return reject(err);
            }

            return resolve(response);
        });

[jmzwcn]

Two doubts, but need yourself confirm. The first "grpc_set_header" is much more than example, the second is grpc-web client and service are based on the same version proto generated?

You can refer to full example: https://github.com/emart-io/cross/tree/main/pwa

[mkonir0]

the first "grpc_set_header" is much more than example

I started with that example but since it was not working I tried adding other stuff

grpc-web client and service are based on the same version proto generated

yes it is

sorry for the radio silence
I decided to abandon the Nginx proxy solution and instead chose to use Envoy as a sidecar(not ideall but it works)

[MrFoxPro]

@jmzwcn hey, are you sure that this actually works for grpc-web?
I spent a lot of trying to configure Nginx to proxy grpc, but didn't achieved it. I described my issue in details on StackOverflow: https://stackoverflow.com/questions/77353121/unable-to-configure-nginx-as-local-proxy-for-https-grpc

[jmzwcn]

Please notice Keypoints in issue.

[MrFoxPro]

Please notice Keypoints in issue.

So using

location / {
  grpc_set_header Content-Type application/grpc;
  grpc_pass grpc://localhost:3002;
}

doesn't do anything, error is the same.
And again, original grpc-web client produces shitty javascript that doesn't work in any modern environment (#1242)
I tried ts-proto/protobuf-ts that abide to follow grpc specs, and nginx doesn't allow messages.

Anyway, there are no mentions of grpc-web on nginx side, so, wether ts-proto/protobuf-ts produce incompatable/incorrect grpc-web requests, or you have different environment that I can't reproduce.

[jmzwcn]

By the above metioned, it doesn't come to grpc process, please resolve other problem firstly.

[Levyks]

@MrFoxPro have you ever got around on getting @protobuf-ts to work with a GRPC server using nginx as a proxy? I'm trying to do the same with no success

[MrFoxPro]

@Levyks just dropped this bs

[visortelle]

@jmzwcn hi 👋

We are considering moving from Envoy to Nginx for grpc-web, but not sure if it works for our use case.

grpc-web docs says:

Server-side Streaming RPCs (example) (NOTE: Only when grpcwebtext mode is used.)

The configuration you provide uses mode=grpcweb. Does it mean that at this moment, server-side streaming isn't achievable with grpc-web + Nginx?

Thank you.

[sampajano]

@jmzwcn Very interesting! Thanks so much for the info! :)

[sampajano]

I'd want to make a PR to mention this fact!

But I'd like one extra confirmation here.. (since i don't really run an nginx server)

If this is confirmed to work by anyone else, i'd change our documentation to reflect that! 😃

[zhumazhenis]

@jmzwcn Thanks for sharing the example. It works for me. By following above example, I could set up for both NGINX Server, and Kubernetes NGINX Ingress.

Provided below Kubernetes NGINX Ingress config.

nginx.ingress.kubernetes.io/configuration-snippet: |
      grpc_set_header Content-Type "application/grpc";

@sampajano I think now we can change the doc. Thanks!

[sampajano]

Oh fantastic! Thanks for the great news! Will make the doc change! 😃

[sampajano]

FYI here is the document update :)
#1403

Thanks all for contributing to the discussion! 😊

[vcache]

I've just tried the proposed configuration. Although it works almost out-of-box, it seems far from a production-ready grade.

First of all, after some tinkering around I was able to run the "hello world" example. But there were a few non-obvious moments. CORS headers are must have option. Those work fine - https://github.com/emart-io/cross/blob/main/pwa/nginx.conf#L20-L26.

Another thing, ensure that you use the latest Nginx. I've tried with 1.18.0 and it didn't work due to the bug which was fixed in 1.19+. Finally, it worked with nginx version 1.25.5.

What is serious is that nginx doesn't handle correctly the so-called "trailers" ("correctly" in the sense of gRPC/gRPC-Web interop, ofc), which compromises the whole scheme. I am wondering why no one else spotted this problem. To my guess, people only tried it with nodejs, where this is not the case.

So, the servers, that are written in any other language (I've tested only with C++ and Python, but I bet that Go/Java are affected too), require the support of trailers, which is a basic requirement for gRPC. Browsers (so as gRPC-Web) don't support them, which is a problem.

By default, the following error will appear on the server side:

D0509 12:45:15.702537737   52699 server.cc:1794]                       Failed call creation: UNKNOWN:Missing :te header {grpc_status:2}

This dirty hack fixes it:

        location / {
            grpc_set_header Content-Type application/grpc;
            grpc_set_header TE trailers;  # <-- this one
            grpc_pass grpc://localhost:50051;
        }

Although it works with the hack, I have no idea how it will perform when the server decides to send trailing headers in responses. I bet it will jam the browser's HTTP parser and cause a network error. I haven't researched yet the condition for emitting trailing headers. Maybe when the data stream is long enough or in the presence of another stream. Or maybe it will never happen :).

There is a code in Envoy, that handles trailing headers correctly - https://github.com/envoyproxy/envoy/blob/a7d3789c3eeda314c6a96be72485eb89a672a268/source/extensions/filters/http/grpc_web/grpc_web_filter.cc#L324-L365 - it moves them from "trailing area" into a body, if I understood it right. Nothing similar persists in nginx's ngx_http_grpc_module.c, so, it cannot be considered as a full-fledged replacement (which I found very sad and disappointing).

All in all, despite the fact that solution works, one should use it will care.

[sampajano]

@vcache wow. Thanks for much for giving this a try and digging into the details!

There is a code in Envoy, that handles trailing headers correctly - https://github.com/envoyproxy/envoy/blob/a7d3789c3eeda314c6a96be72485eb89a672a268/source/extensions/filters/http/grpc_web/grpc_web_filter.cc#L324-L365 - it moves them from "trailing area" into a body, if I understood it right. Nothing similar persists in nginx's ngx_http_grpc_module.c, so, it cannot be considered as a full-fledged replacement (which I found very sad and disappointing).

Yes.. It's indeed necessary for grpc-web to work fully when trailer is moved into the body, along with a bit denoting the beginning of a trailer frame:

https://github.com/envoyproxy/envoy/blob/7b7ab06de635f830529cc54a807073a2db2c4c38/source/extensions/filters/http/grpc_web/grpc_web_filter.cc#L57

And is parsed in grpc-web:

grpc-web/javascript/net/grpc/web/grpcwebstreamparser.js

Line 276 in e91e540

TRAILER: 0x80, // expecting a trailer frame

So the lack of this function would unfortunately indicate that the support is incomplete (if it works at all)..

[timostamm]

The issue is that you will never know if your call succeeded, because gRPC encodes the RPC status in a HTTP trailer field.

Besides explicitly allowing for HTTP 1.1, this is the crucial difference between gRPC and gRPC-Web: The latter attaches the HTTP trailers to the response body, because web browsers do not give access to HTTP trailers.

Why does gRPC use trailers?

Because the protocol allows to send and receive multiple messages in a request. Each individual message is prefixed by 5 bytes. One byte is for flags like compression, the remaining 4 bytes contain the message length. So a server could be responding with one message, but run into an error assembling the second message. Since response headers are already sent, it isn't possible to use the HTTP status or a header field for the error code. This encoding is used for all RPCs, streaming or unary. If the error occurs early, a server may send trailers as headers instead, and omit the response body. But since this behavior optional, it's unreliable for clients rely on it, even for unary RPCs.

How gRPC-Web encodes trailers

The message prefixes in the body have one byte reserved for flags. gRPC-Web uses one bit to designate a message as trailers. Trailers are encoded like HTTP 1.1 headers, but keys must be lower-case. ngx_http_grpc_module clearly does not implement this transcoding, envoy's gRPC-Web filter does.

[sampajano]

@timostamm Oh wow.. thanks for sharing this great context knowledge and crisp clear explanations!

[rizdaputra]

It doesnt work for me, whenever i run i get

Unexpected error for sayHello: code = 2, message = "Incomplete response"

I run the react application as client and this is my nginx config

http {

  server {
    listen 8080;
    http2 on;

    root   /usr/share/nginx/html;
    index  index.html index.htm;

    add_header 'Access-Control-Allow-Origin' '*' always;
    add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS' always;
    add_header 'Access-Control-Allow-Headers' 'Content-Type, Authorization, x-grpc-web, x-user-agent' always;
    add_header 'Access-Control-Expose-Headers' 'grpc-status, grpc-message' always;

    location / {
        try_files $uri /index.html;
    }

    # this is the grpc setting
    location /helloworld.Greeter/ {
        grpc_set_header Content-Type application/grpc;
        grpc_set_header X-Forwarded-For $remote_addr;
        grpc_pass grpc://172.17.0.1:9090;
    }
  }
}

events {}

the server is the one in the example folder the hello world example and running locally in 9090
can someone help me? I am stuck

[jmzwcn]

ensure your 3 keypoints.

1.https: Ensure grpc-web send http2 request.
2.grpc_set_header Content-Type application/grpc in nginx.conf;
3.mode is grpcweb when generate-js:

[rizdaputra]

ahhh i was using grpcwebtext for the output, thank you soooo much this works well now

[sampajano]

@jmzwcn Wow thank you so much for providing the pro-tips!! 🙏

@rizdaputra Very happy it's working now! 😃