Quarkus with GRPC server, behind a istio service mesh throws error #39769

priyanknarvekar · 2024-03-28T12:07:28Z

Describe the bug

Quarkus with new GRPC server, behind a istio service mesh throws error
Without the Service mesh it works fine.

__  ____  __  _____   ___  __ ____  ______ 
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
2024-03-28 11:49:54,476 WARN  [io.qua.config] (main) Unrecognized configuration key "quarkus.log.console.json" was provided; it will be ignored; verify that the dependency extension for this configuration is set or that you did not make a typo
2024-03-28 11:49:55,196 INFO  [io.qua.grp.run.GrpcServerRecorder] (main) Registering gRPC reflection service
2024-03-28 11:49:55,249 INFO  [io.qua.grp.run.GrpcServerRecorder] (main) Starting new Vert.x gRPC server ...
2024-03-28 11:49:55,331 INFO  [io.quarkus] (main) grpc-demo 1.0.0-SNAPSHOT on JVM (powered by Quarkus 3.3.0) started in 1.438s. Listening on: http://0.0.0.0:8080
2024-03-28 11:49:55,332 INFO  [io.quarkus] (main) Profile prod activated. 
2024-03-28 11:49:55,332 INFO  [io.quarkus] (main) Installed features: [cdi, grpc-server, kubernetes, resteasy-reactive, smallrye-context-propagation, vertx]
2024-03-28 11:51:29,587 ERROR [io.qua.mut.run.MutinyInfrastructure] (vert.x-eventloop-thread-0) Mutiny had to drop the following exception: java.lang.IllegalStateException: You must set the Content-Length header to be the total size of the message body BEFORE sending any data if you are not using HTTP chunked encoding.
	at io.vertx.core.http.impl.Http1xServerResponse.write(Http1xServerResponse.java:764)
	at io.vertx.core.http.impl.Http1xServerResponse.write(Http1xServerResponse.java:325)
	at io.vertx.core.http.impl.Http1xServerResponse.write(Http1xServerResponse.java:67)
	at io.vertx.grpc.server.impl.GrpcServerResponseImpl.writeMessage(GrpcServerResponseImpl.java:242)
	at io.vertx.grpc.server.impl.GrpcServerResponseImpl.writeMessage(GrpcServerResponseImpl.java:112)
	at io.vertx.grpc.common.impl.WriteStreamAdapter.write(WriteStreamAdapter.java:47)
	at io.vertx.grpc.server.impl.GrpcServiceBridgeImpl$ServerCallImpl.sendMessage(GrpcServiceBridgeImpl.java:137)
	at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onNext(ServerCalls.java:380)
	at io.smallrye.mutiny.subscription.Subscribers$CallbackBasedSubscriber.onItem(Subscribers.java:79)
	at io.smallrye.mutiny.operators.multi.MultiMapOp$MapProcessor.onItem(MultiMapOp.java:50)
	at io.smallrye.mutiny.subscription.MultiSubscriber.onNext(MultiSubscriber.java:61)
	at io.smallrye.mutiny.operators.multi.processors.UnicastProcessor.drainWithDownstream(UnicastProcessor.java:107)
	at io.smallrye.mutiny.operators.multi.processors.UnicastProcessor.drain(UnicastProcessor.java:138)
	at io.smallrye.mutiny.operators.multi.processors.UnicastProcessor.onNext(UnicastProcessor.java:204)
	at io.quarkus.grpc.stubs.ServerCalls$1.onNext(ServerCalls.java:163)
	at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:262)
	at io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33)
	at io.quarkus.grpc.runtime.supports.context.GrpcRequestContextGrpcInterceptor$1.onMessage(GrpcRequestContextGrpcInterceptor.java:60)
	at io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33)
	at io.quarkus.grpc.ExceptionHandler.onMessage(ExceptionHandler.java:27)
	at io.quarkus.grpc.runtime.supports.context.GrpcDuplicatedContextGrpcInterceptor$ListenedOnDuplicatedContext.lambda$onMessage$1(GrpcDuplicatedContextGrpcInterceptor.java:136)
	at io.quarkus.grpc.runtime.supports.context.GrpcDuplicatedContextGrpcInterceptor$ListenedOnDuplicatedContext.invoke(GrpcDuplicatedContextGrpcInterceptor.java:115)
	at io.quarkus.grpc.runtime.supports.context.GrpcDuplicatedContextGrpcInterceptor$ListenedOnDuplicatedContext.onMessage(GrpcDuplicatedContextGrpcInterceptor.java:136)
	at io.vertx.grpc.server.impl.GrpcServiceBridgeImpl$ServerCallImpl$1.handleMessage(GrpcServiceBridgeImpl.java:97)
	at io.vertx.grpc.common.impl.ReadStreamAdapter.lambda$init$0(ReadStreamAdapter.java:30)
	at io.vertx.grpc.common.impl.GrpcReadStreamBase.handleMessage(GrpcReadStreamBase.java:207)
	at io.vertx.grpc.common.impl.GrpcReadStreamBase.lambda$init$3(GrpcReadStreamBase.java:89)
	at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:255)
	at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:134)
	at io.vertx.grpc.common.impl.GrpcReadStreamBase.handle(GrpcReadStreamBase.java:125)
	at io.vertx.grpc.common.impl.GrpcReadStreamBase.handle(GrpcReadStreamBase.java:39)
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:264)
	at io.vertx.core.http.impl.HttpEventHandler.handleChunk(HttpEventHandler.java:51)
	at io.vertx.core.http.impl.Http1xServerRequest.onData(Http1xServerRequest.java:551)
	at io.vertx.core.http.impl.Http1xServerRequest.lambda$pendingQueue$1(Http1xServerRequest.java:132)
	at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:255)
	at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:134)
	at io.vertx.core.http.impl.Http1xServerRequest.handleContent(Http1xServerRequest.java:146)
	at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:76)
	at io.vertx.core.impl.DuplicatedContext.execute(DuplicatedContext.java:153)
	at io.vertx.core.http.impl.Http1xServerConnection.onContent(Http1xServerConnection.java:199)
	at io.vertx.core.http.impl.Http1xServerConnection.handleOther(Http1xServerConnection.java:182)
	at io.vertx.core.http.impl.Http1xServerConnection.handleMessage(Http1xServerConnection.java:169)
	at io.vertx.core.net.impl.ConnectionBase.read(ConnectionBase.java:158)
	at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:153)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.vertx.core.http.impl.Http1xOrH2CHandler.end(Http1xOrH2CHandler.java:61)
	at io.vertx.core.http.impl.Http1xOrH2CHandler.channelRead(Http1xOrH2CHandler.java:38)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1583)

Removing the istio label, saw a the grpc reponses

grpcurl -vv --plaintext  grpc-demo.default.svc.cluster.local:80 list
grpc.health.v1.Health
helloworld.Greeter

Expected behavior

Should send appropriate response, when service mesh is enabled as well.

Actual behavior

Throws Exception, on the client side no response/timeout.

How to Reproduce?

Create a Getting started quarkus grpc project based on https://quarkus.io/guides/grpc-getting-started (I have one at https://github.com/priyanknarvekar/grpc-demo)
deploy it to k8s with istio deployed.
try any grpc requests to the deployed service, I had a vpn into the cluster, so just doing grpcurl or postman requests fail, programatic Request from a client application fail as well - grpcurl -vv --plaintext grpc-demo.default.svc.cluster.local:80 list

Output of `uname -a` or `ver`

Docker Runtime - eclipse-temurin:21-jdk-jammy

Output of `java -version`

eclipse-temurin:21-jdk-jammy

Quarkus version or git rev

3.3.0

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

quarkus 3.4.1 (using quarkus build Apache Maven 3.8.7 local Java version: 17.0.9)

Additional information

No response

The text was updated successfully, but these errors were encountered:

quarkus-bot · 2024-03-28T12:07:31Z

/cc @alesj (grpc), @cescoffier (grpc), @geoand (kubernetes), @iocanel (kubernetes)

cescoffier · 2024-04-01T15:59:50Z

It seems that the service mesh does not send the content-length header and does not use "chunked" encoding. I do not have experience with service meshes, so could you provide guidelines for getting them running locally?

Your example uses the reflection service; I suspect you also tried with a real service and saw the same behavior.

priyanknarvekar · 2024-04-04T02:18:53Z

this appears to provide guidance to setup MiniKube and thereafter install istio.

Yeah I had seen that with the real service as well.

keitaf · 2024-04-26T17:35:37Z

I encountered the same issue. It was due to the Istio protocol selection, which translates HTTP/2 to HTTP/1.1.
https://istio.io/latest/docs/ops/configuration/traffic-management/protocol-selection/

Changing the port name of the Service from http to grpc solved the issue.

I think it'd be nicer if Quarkus gRPC can show some warning when it received requests in HTTP/1.1.

cescoffier · 2024-04-28T11:47:59Z

@alesj WDYT?

alesj · 2024-04-29T05:25:24Z

@alesj WDYT?

You mean wrt this:

I think it'd be nicer if Quarkus gRPC can show some warning when it received requests in HTTP/1.1.

Sure, why not.

cescoffier · 2024-04-29T05:45:21Z

@alesj we already have a method checking if the request is a gRPC request. We can add the check there.

Note that, in theory, it is possible to transform gRPC frames on http/1.1. But we do not support that.

alesj · 2024-04-29T07:55:11Z

we already have a method checking if the request is a gRPC request. We can add the check there.

Ah yes. But this is for Vert.x gRPC only.
Is that enough? What about io.grpc ?

cescoffier · 2024-05-02T06:20:46Z

I would ignore io.grpc (grpc-java). We should even considering switching the defaults (with the right amount of documentation, breaking change description...)

alesj · 2024-05-03T10:59:46Z

I would ignore io.grpc (grpc-java). We should even considering switching the defaults (with the right amount of documentation, breaking change description...)

Yeah, we should do that -- when was the official plan?

We can add the check there.

What would the check look like? Or what to check for in the headers?

cescoffier · 2024-05-03T11:39:49Z

The official plan was: after the LTS, so right about... now 😝

cescoffier · 2024-05-03T11:40:57Z

About the check, if you get the request object (which I believe you do) you can check the protocol and reject http/1 and http/1.1 requests.

alesj · 2024-05-03T12:07:02Z

The official plan was: after the LTS, so right about... now 😝

Uf ... let me finish my Observability blog / docs ... and then I can add something about this change.

alesj · 2024-05-03T14:11:19Z

About the check, if you get the request object (which I believe you do) you can check the protocol and reject http/1 and http/1.1 requests.

Like this?

    private static boolean isGrpc(RoutingContext rc) {
        HttpServerRequest request = rc.request();
        HttpVersion version = request.version();
        if (HttpVersion.HTTP_1_0.equals(version) || HttpVersion.HTTP_1_1.equals(version)) {
            LOGGER.debugf("Expecting %s, received %s - not a gRPC request", HttpVersion.HTTP_2, version);
            return false;
        }
        String header = request.getHeader("content-type");
        return header != null && GRPC_CONTENT_TYPE.matcher(header.toLowerCase(Locale.ROOT)).matches();
    }

cescoffier · 2024-05-03T17:59:34Z

Yes!

priyanknarvekar added the kind/bug Something isn't working label Mar 28, 2024

quarkus-bot bot added area/grpc gRPC area/kubernetes labels Mar 28, 2024

alesj mentioned this issue May 4, 2024

Ignore non-HTTP_2 requests #40454

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Quarkus with GRPC server, behind a istio service mesh throws error #39769

Quarkus with GRPC server, behind a istio service mesh throws error #39769

priyanknarvekar commented Mar 28, 2024

quarkus-bot bot commented Mar 28, 2024

cescoffier commented Apr 1, 2024

priyanknarvekar commented Apr 4, 2024

keitaf commented Apr 26, 2024

cescoffier commented Apr 28, 2024

alesj commented Apr 29, 2024

cescoffier commented Apr 29, 2024

alesj commented Apr 29, 2024

cescoffier commented May 2, 2024

alesj commented May 3, 2024

cescoffier commented May 3, 2024

cescoffier commented May 3, 2024

alesj commented May 3, 2024

alesj commented May 3, 2024 •

edited

Loading

cescoffier commented May 3, 2024

Quarkus with GRPC server, behind a istio service mesh throws error #39769

Quarkus with GRPC server, behind a istio service mesh throws error #39769

Comments

priyanknarvekar commented Mar 28, 2024

Describe the bug

Expected behavior

Actual behavior

How to Reproduce?

Output of uname -a or ver

Output of java -version

Quarkus version or git rev

Build tool (ie. output of mvnw --version or gradlew --version)

Additional information

quarkus-bot bot commented Mar 28, 2024

cescoffier commented Apr 1, 2024

priyanknarvekar commented Apr 4, 2024

keitaf commented Apr 26, 2024

cescoffier commented Apr 28, 2024

alesj commented Apr 29, 2024

cescoffier commented Apr 29, 2024

alesj commented Apr 29, 2024

cescoffier commented May 2, 2024

alesj commented May 3, 2024

cescoffier commented May 3, 2024

cescoffier commented May 3, 2024

alesj commented May 3, 2024

alesj commented May 3, 2024 • edited Loading

cescoffier commented May 3, 2024

Output of `uname -a` or `ver`

Output of `java -version`

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

alesj commented May 3, 2024 •

edited

Loading