Elevated server closed the stream without sending trailers after enabling oghttp2

[milton0825]

Title: Elevated server closed the stream without sending trailers after enabling oghttp2

Description:
We are seeing elevated level of server closed the stream without sending trailers in grpc-go after enabling oghttp2 via
envoy.reloadable_features.http2_use_oghttp2.

Repro steps:

1. Enable oghttp2 by setting envoy.reloadable_features.http2_use_oghttp2 to true.
2. Run GRPC service with Envoy version of 20c7368afa9d686a109f9601ae1b9b6028b74b0a.

Admin and Stats Output:

Include the admin output for the following endpoints: /stats,
/clusters, /routes, /server_info. For more information, refer to the
admin endpoint documentation.

Note: If there are privacy concerns, sanitize the data prior to
sharing.

Config:

Include the config used to configure Envoy.

Logs:

Include the access logs and the Envoy logs.

Note: If there are privacy concerns, sanitize the data prior to
sharing.

Call Stack:

If the Envoy binary is crashing, a call stack is required.
Please refer to the Bazel Stack trace documentation.

[milton0825]

cc: @birenroy

[birenroy]

Do you have any log output from a grpc-go client? That would help us with debugging.

[milton0825]

Captured the following logs in go-grpc
service-gogrpc-debug.csv

[milton0825]

So it seems like the GRPC client is not expecting a data frame with END_STREAM flag set. I am wondering where in envoy (or oghttp2) could be touching the flag.

[birenroy]

/assign @birenroy

[birenroy]

I wasn't able to find any server closed the stream without sending trailers in the logs you attached. Are you still able to reproduce the problem?

It would also be helpful to know the failure rate before and after enabling oghttp2. Are we talking 0.1%? 1%? 10%?

[milton0825]

Those are the logs from grpc-go client after I enabled:

GRPC_GO_LOG_VERBOSITY_LEVEL=99
GRPC_GO_LOG_SEVERITY_LEVEL=info

I can still see server closed the stream without sending trailers error message returned from grpc-go in the app.

The failure rate was ~0.001% and increase to the level of 0.1%.

[milton0825]

@birenroy any other information you need from me?

[birenroy]

Sorry for the delay. An increase from 0.001% to 0.1% might be difficult to reproduce, and therefore difficult to track down, especially if the reproduction involves running a complete gRPC service through an Envoy.

I am working on some cleanup to the HTTP/2 codec in #32378; I'd like to revisit this after that PR.

[birenroy]

@milton0825 Now that #32378 has been merged, would you be able to try enabling the oghttp2 feature again?

[milton0825]

It seems like #32378 is just code clean up. Is it going to fix the issue?

[birenroy]

It simplifies the flow of HTTP/2 protocol events through the Envoy codec layer. At the very least, it will make the issue easier to investigate if we can reproduce the bug.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.