Assertion failed: self.slab.is_empty()

[mzabaluev]

A panic caused by a failed assertion in h2 0.1.26 has been observed in input-output-hk/jormungandr#881.

The assertion self.slab.is_empty() fails under some circumstances in the Drop impl for proto::streams::store::Store. The diagnostic gives me no idea whether this could be an application problem or how to mitigate it.

[seanmonstar]

The assertion is for internal state. If I remember correctly, we assume that all streams will have been dropped before the whole thing shuts down, otherwise we may have left a stream in an unfinished state.

Maybe the assumption is just bad.

[mzabaluev]

Shall the application (or a higher level library like tower-grpc) then take care of dropping every stream handle?

[seanmonstar]

No, I think we assumed this because the store is inside an arc, and all streams have a reference, so ideally all streams have been dropped and cleaned up before the arc drops. It's a bug in h2.

[fridge-dev]

Duhza#2283 from Discord told me to paste this here. I've observed this issue with my gRPC client which uses tonic. Here is my stacktrace: https://pastebin.com/P829ZSB0?TTL=6months

My bin crate was called rust_client which had a workspace crate called grpc. The test setup was as follows:

1. gRPC server application has a server-streaming RPC. Implementation is to send back 1MM messages (payload is just random stuff)
2. gRPC client application (written in rust, using tonic) connects to the server and calls the stream RPC.
3. gRPC client app polls from the stream inside a loop until it receives either Err(_) or Ok(None) (this is grpc::grpc_client::poll_stream_until_complete in the stack trace).
4. Upon running the server app and client app, I ctrl+c my server. The client app has the resulting panic seen here.

Hope this helps!