@grpc/grpc-js not working with aws ALB #2093
Comments
|
Do you think you could get a wireshark-compatible TCP dump of the interaction that results in this error ( |
I need to run this command inside my container on aws, right? |
|
Actually, I think a dump from the client would be more useful. Since the problem happens with different gRPC servers, it seems to be a problem with the client talking to the ALB. I want to see exactly what happens in that interaction. |
|
Sorry for the delay, I already had deleted all my tests from aws so needed to create everything again. I ran this command ( |
|
Can you please share the raw pcap file? I would like to look at it in Wireshark to see how it parses the HTTP/2 data from the actual bytes that went over the wire. |
|
That capture shows that the server is responding with the error HTTP/1.1 400 Bad Request. Something is misconfigured and the front end your client is talking to isn't handling HTTP/2. And the client doesn't understand an HTTP/1.1 response to an HTTP/2 request, so that's why the error is "Protocol error". |
|
Ok, somehow in my communication chain (client->alb->server->alb->client) the request is being "converted" to HTTP/1.1, right? I didn't understand if the problem is in the communication between my client and the ALB or between the ALB and my server. Was you able to determine that in your data analysis? |
|
You gave me a dump of the communication between the client and the ALB. It shows how the ALB responded to the request. It does not show anything about the ALB communicating with the server or anything about "converting" the request because those are not things that happen in the client to ALB communication path. I also just took another look at that dump log, and I noticed that the client is making a request to the method I think the primary problem here is that the ALB can respond to an HTTP/2 request with an HTTP/1.1 response at all. I believe this is a bug in the ALB. |
|
I didn't use the Hello World client for the dump log because when you requested it I already had deleted all my tests from AWS. So in order to make it easier I used other Grpc service that already was uploaded to ECR. Now I understood the thing you said about the communication, but I don't think it is only an ALB problem because when I used a python client and server in my tests it worked. I will recreate de scenario using the python Hello World and put the tcpdump here. |
|
When you do that, can you please also capture a dump with the Node Hello World client so that we can do a 1-to-1 comparison? |
|
I've been looking into this with @g-sartori and we found the problem. ALB ensures that only HTTPS listeners can foward requests to a gRPC target group, but we were consuming the server like it did'nt had any TLS/SSL in front of it. So we changed the way we created the client and it worked. We were creating the client like this var client = new hello_proto.Greeter(target, grpc.credentials.createInsecure())And we changed to this var client = new hello_proto.Greeter(target, grpc.credentials.createSsl())In this documentation https://aws.amazon.com/pt/blogs/aws/new-application-load-balancer-support-for-end-to-end-http-2-and-grpc/ they do the same thing for the python hello-word example using |
|
Hi There, I am also facing the same issues with ALB, my Application is running on AWS ECS, and i have configured GRPS protocol in Target group, Health check is showing healthy but non of the request is going to the Application server. Error: 400 Bad Request. |
I'm having the same issue and have been banging my head against the wall for far too long. When I write the node-based client using this library, I can hit the api server locally using insecure mode. But if I try to hit the remote API server using ssl, it drops the connection. Both clients are effectively doing the same thing. Why does the connection get dropped for the node client? |
It may be that you guys need to especify the Here is an example: return new hello_proto.Greeter(target, grpc.credentials.createSsl(),
{
'grpc.ssl_target_name_override': 'my-certificate.com'
})I had to do this when i used the direct dns name of the load balancer or a custom domain name setup in route53 that was direfent from the certificate domain name |
|
Hm, I tried a variety of combinations with that override and still to no avail. The main difference is that the URL I'm requesting is tied to a wildcard certificate. I tried overriding the target name to be the wildcard name, and a few other things, but still nothing. URL example: Certificate: To further isolate this, I was able to call the remote service using Works without any issues, however, |
|
Not sure if this is helpful, but here is a debug trace: |
|
If you are using Node.js server-side grpc-web, then it defaults to HTTP/2. However, AWS ALB can be configured for HTTP/1.1 Setting Is there a way to force HTTP/1.1? Our browser clients work fine at the moment, but this will break with Node.js. |
|
Actually HTTP/1.1 won't work, as GRPC relies on HTTP/2 Here's the AWS setting for request vs. configured protocol It should work, so this is some sort of issue with the Node.js implementation interacting with AWS Application Load Balancer (ALB). Perhaps you must be forced to configure as "HTTP2" on the ALB to support GRPC fully. This would mean you can't mix HTTP/1.1 with HTTP/2 on that specific ALB, and would be forced to use TLS/HTTPS as well. |
|
Hi @g-sartori , @cabulafhy, Can you please help with gRPC server-side code as well, if it's working for you? |
|
with AWS ALB I am getting 14 unavailable errors. But with IP and port, it is working as expected. Also, I have a wildcard certificate on AWS ALB for my target group. |
Problem description
I was trying to use grpc with ALB, considering this documentation from AWS: https://aws.amazon.com/pt/blogs/aws/new-application-load-balancer-support-for-end-to-end-http-2-and-grpc/
I created the Load balancer, target group, cluster, service and task definition using an ECR image with this hello-world server example from grpc repo. I tried dynamic and static proto generation and both returned this error:
After that to make sure the error wasn't in my hello-world server I changed the ECR image to another grpc-server(using @grpc/grpc-jt) that I already have working on aws, and the error persists.
To finish I decided to test using the python hello world example, and it worked, so I believe the @grpc/grpc-js have some bad interaction with ALB, but I don't know exactly how debug this
Reproduction steps
Environment
Additional context
I tried generate a cert with openssl in order to no use grcp.createInsecure, but it changed nothin
The text was updated successfully, but these errors were encountered: