Skip to content

2.31.0.0-b271

2.31.0.0-b271: [#32647] DistributedTracing: Fix bugs in backend shutdown path
@IshanChhangani IshanChhangani tagged this 18 Jul 09:44 
Summary:
1) Heap use-after-free at process exit.
When a backend dies via ereport(FATAL) mid-query (pg_terminate_backend, tserver
shutdown), the interrupted query's spans are still on the static otel_scope_stack.
At exit(), OpenTelemetry's thread-local context stack is freed first; postgres's backend
local stack's (OtelScopeStack) destructor then runs
~Scope -> ~Token -> RuntimeContext::Detach against the freed
stack, which ASAN/TSAN report as heap-use-after-free.
This produced the sanitizer failure wave across kill/shutdown/abort
tests (pg_auto_analyze-test, PgRecursiveAbortTest, pg_op_processing_order-test, ...).

Fix: clear the scope stack at the top of YBCCleanupDistTrace(), which runs from the
ProcKill on_shmem_exit callback -- well before TLS teardown, so tokens detach while
the thread-local stack is still alive.

2) PgClientSession leak from slowed backend exit.
YBOnPostgresBackendShutdown ran YBCDestroyPgGate() (which closes the tserver
connection) before YBCCleanupDistTrace(). Cleanup shuts down the TracerProvider,
whose BatchSpanProcessor performs a synchronous OTLP flush .
The tserver's CheckSessionShutdown probe checks the disconnected
backend exactly once, 50 ms after the connection closes, and does not retry; the
backend was still alive flushing, so its PgClientSession was never reaped and lived
to the 60 s pg_client_session_expiration_ms. Observed as
PgSharedMemTest.ConnectionShutdown failing deterministically on arm-release
(all 19 sessions leaked), and as spurious client connection failures.

Fix: run YBCCleanupDistTrace() before YBCDestroyPgGate() so the OTLP flush
completes while the connection is still open, and the backend exits promptly
after the connection closes.

Test Plan:
Running the following tests with distributed tracing enabled (collector flag set and each query run with a valid traceparent) should reproduce this bug.
PgSharedMemTest.ConnectionShutdown
XClusterDDLReplicationTest.SingleDDLQueueHandler
pg_auto_analyze-test
pg_backends-test

Reviewers: asaha

Reviewed By: asaha

Subscribers: yql

Differential Revision: https://phorge.dev.yugabyte.com/D55619
Assets 2
Loading