multi-thread lsquic server crashed because header stream was released before connection closed(Q043).

[jazune]

I got a crash in maybe_conn_flush_headers_stream function because the header stream was abnormal released when I carried out my stress test on my multi-thread server.（lsquc version is 2.20.0 with multi-thread safe patch）

I found the header stream was released (with AddressSanitizer compile) in service_streams function because (stream->sm_qflags & SMQF_FREE_STREAM) was true.

Since maybe_conn_flush_headers_stream will be executed in every full_conn_ci_tick, Should we not release header stream before connection closed?

It seems header stream flag SMQF_FREE_STREAM was set but conn was not closed, and after the next full_conn_ci_tick, it crashed.

[dtikhonov]

@jazune, did this turn out not to be an lsquic bug?

[jazune]

It seems lsquic bug.
I use AddressSanitizer to analyse, and I found the header stream is allocated in lsquic, released in lsquic, and reused also in lsquic.
In fact， application will not use header stream directly.

I fix this by:

headers_on_close (lsquic_stream_t *stream, struct lsquic_stream_ctx *ctx)
{
struct headers_stream *hs = (struct headers_stream *) ctx;.
LSQ_DEBUG("stream closed");
++ hs->hs_stream = NULL;

}

@@ -3099,6 +3099,8 @@ maybe_conn_flush_headers_stream (struct full_conn *conn)
if (conn->fc_flags & FC_HTTP)
{
stream = lsquic_headers_stream_get_stream(conn->fc_pub.u.gquic.hs);
++ if (stream == NULL)
++ return;
if (lsquic_stream_has_data_to_flush(stream))
(void) lsquic_stream_flush(stream);
}

But I still don't know why full_conn_ci_tick was executed when header stream was closed

[dtikhonov]

It does not happen in single-threaded code that a connection is ticked after the headers stream is destroyed. I incline to think that it is more likely that the bug is in your multi-threaded logic.

[dtikhonov]

We have been able to reproduce this issue. Fix is forthcoming.

[litespeedtech]

Fixed in 2.27.3 -- closing.