Http2: Writes starving reads?

[tommyulfsparre]

Expected behavior

Running the reproducer code the client should succeed the majority of RPCs without hitting the deadline (100ms).

Actual behavior

Using gRPC or servicetalk the reproducer will fail most RPC while using OkHttp does not. The commonality between gRPC and servicetalk is that they are both built on Netty, although with different implementations.

This issue was initial raised here: grpc/grpc-java#8912 and contains some more information; It was speculated that it might be due to writes starving reads and seeing that OkHttp uses separate dedicated threads for reading and writing would explain why it does not exhibit the same behavior here.

Steps to reproduce

Run the reproducer with either -Dcom.grpc.example.transport=netty or -Dcom.grpc.example.transport=servicetalk system property set.

Minimal yet complete reproducer code (or URL to code)

https://github.com/tommyulfsparre/repro-netty-http2

Netty version

4.1.72.Final

JVM version (e.g. java -version)

Java 11

OS version (e.g. uname -a)

---

Does this sound plausible and if so is there anything that can be tweak in Netty to alleviate this?

[normanmaurer]

Any idea @ejona86 ?

[ejona86]

I looked into it and confirmed the behavior. But I haven't determined why ioRatio isn't avoiding this problem.

[franz1981]

@ejona86 the benchmark/test is strange (no warmup, just 1K input buffers) and I am not personally confident how the protocol is supposed to work.
I remember that others eg Jetty has developed and hoc scheduling of writes/reads to save HOL to happen ie Eat What You Kill - named as a jetty blog post (if that's the issue that's happening, maybe not).
Can you confirm that despite how this test is written, this is a real issue that can affect production-level usages?
@tommyulfsparre as well
@vietj we are interested on this because of grpc

[tommyulfsparre]

Can you confirm that despite how this test is written, this is a real issue that can affect production-level usages

@franz1981 How would one know if a timeout happens because of this (potential) issue or any other reason that can lead to a timeout? Is there instrumentation that is available that one could hook into? Here is some additional context: grpc/grpc-java#8912 (comment).

The code merely tries to show that during certain, fabricated, conditions Netty seem to fare worse than other transport (OkHttp in this case). If this is something that actually happens in practice or if it's even a Netty issue, I don't know yet.