Skip to content

Fix session logger use-after-free during EP teardown under VERBOSE logging#28274

Merged
tianleiwu merged 3 commits intomainfrom
copilot/fix-session-logger-access-violation
May 1, 2026
Merged

Fix session logger use-after-free during EP teardown under VERBOSE logging#28274
tianleiwu merged 3 commits intomainfrom
copilot/fix-session-logger-access-violation

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Apr 29, 2026

Description

Move owned_session_logger_ declaration before execution_providers_ in InferenceSession so the logger outlives EPs during member destruction.

C++ destroys members in reverse declaration order. Previously:

~owned_session_logger_   (L905)  // logger freed
     ↓
~execution_providers_    (L745)  // EP teardown logs via dangling pointer → crash

After this change:

~execution_providers_    (L750)  // EP teardown logs safely ✅
     ↓
~owned_session_logger_   (L744)  // logger freed, no remaining users

Motivation and Context

Plugin EPs receive an OrtLogger* via OrtEpFactory::CreateEp(). During session destruction, EP teardown callbacks (e.g., ReleaseNodeComputeInfos) may log through this pointer. Because owned_session_logger_ was declared after execution_providers_, the logger was already freed when EPs destructed — a use-after-free that crashes deterministically under VERBOSE logging.

Affects all Plugin EPs that log in any teardown path. Reproduced with OpenVINO Plugin EP via webnn_graph_impl_fuzzer at VERBOSE level.

Fixes #28234

Tests Added

Added regression tests in onnxruntime/test/framework/inference_session_test.cc:

  • LoggingOnDestroyExecutionProvider — A mock EP that logs via its stored logger pointer in its destructor. If the logger has been freed, this triggers a use-after-free (detected by ASan or as a segfault).
  • SessionLoggerOutlivesEPsOnDestruction — Creates a session with VERBOSE logging and the mock EP, then destroys the session. Verifies that the logger was valid during EP teardown and that the teardown log message was captured.
  • SessionLoggerOutlivesEPsWithMultipleEPs — Same scenario with two mock EPs (distinct type names) to confirm all registered EPs can safely log during teardown.

Verification

Confirmed the tests are effective regression tests:

Scenario Result
With fix (logger declared before EPs) Both tests pass ✅
Without fix (logger declared after EPs, original bug) SessionLoggerOutlivesEPsOnDestruction crashes with Segmentation fault (exit code 139) — use-after-free ❌

This proves the member declaration order is the critical factor, and the tests will catch any future regression that reorders these members.

Copilot AI changed the title [WIP] Fix session logger destruction order to prevent use-after-free Fix session logger use-after-free during EP teardown under VERBOSE logging Apr 29, 2026
Copilot AI requested a review from tianleiwu April 29, 2026 17:16
…execution_providers_

Move owned_session_logger_ declaration before execution_providers_ in
InferenceSession so the logger outlives EPs during destruction. C++ destroys
members in reverse declaration order, so previously the logger was destroyed
before EPs, causing a dangling pointer when EP teardown callbacks tried to
log (deterministic crash under VERBOSE logging).

Agent-Logs-Url: https://github.com/microsoft/onnxruntime/sessions/3f374444-9d0d-4b35-ab2a-a2231bbfd994

Co-authored-by: tianleiwu <30328909+tianleiwu@users.noreply.github.com>
@tianleiwu tianleiwu force-pushed the copilot/fix-session-logger-access-violation branch from 1d561d1 to 8ecb2c2 Compare April 30, 2026 04:10
@tianleiwu tianleiwu marked this pull request as ready for review April 30, 2026 04:10
hariharans29
hariharans29 previously approved these changes Apr 30, 2026
@tianleiwu tianleiwu enabled auto-merge (squash) April 30, 2026 07:44
Comment thread onnxruntime/test/framework/inference_session_test.cc
Copy link
Copy Markdown
Contributor

@adrianlizarraga adrianlizarraga left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The added tests fail for WASM debug build:

[ RUN ] InferenceSessionTests.SessionLoggerOutlivesEPsOnDestruction
1: unknown file: Failure
1: C++ exception with description "/mnt/vss/_work/onnxruntime/onnxruntime/onnxruntime/core/session/inference_session.cc:604 void onnxruntime::InferenceSession::ConstructorCommon(const SessionOptions &, const Environment &) session_env.EnvCreatedWithGlobalThreadPools() was false. When the session is not configured to use per session threadpools, the env must be created with the the CreateEnvWithGlobalThreadPools API.
1: " thrown in the test body.
1:
1: [ FAILED ] InferenceSessionTests.SessionLoggerOutlivesEPsOnDestruction (1 ms)

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses a crash caused by a session logger lifetime issue during execution provider (EP) teardown (use-after-free under VERBOSE logging), by adjusting InferenceSession member destruction ordering and adding regression tests to validate safe EP logging during session destruction.

Changes:

  • Reorders InferenceSession members so the owned session logger is intended to outlive execution_providers_ during destruction.
  • Adds new InferenceSession tests with a mock EP that logs during its destructor to catch regressions.
  • Updates Azure Pipelines plugin CUDA stages (step ordering and removal of an artifact-merge stage).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
tools/ci_build/github/azure-pipelines/stages/plugin-linux-cuda-stage.yml Reorders a plugin build variables step relative to Python/setup steps.
tools/ci_build/github/azure-pipelines/stages/plugin-cuda-packaging-stage.yml Removes the Linux x64/aarch64 artifact merge stage.
onnxruntime/test/framework/inference_session_test.cc Adds regression tests using a mock EP that logs in its destructor.
onnxruntime/core/session/inference_session.h Moves owned_session_logger_ earlier to change destruction order relative to EPs.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread onnxruntime/core/session/inference_session.h
Comment thread onnxruntime/test/framework/inference_session_test.cc
@tianleiwu tianleiwu merged commit 5f7e9d0 into main May 1, 2026
165 of 173 checks passed
@tianleiwu tianleiwu deleted the copilot/fix-session-logger-access-violation branch May 1, 2026 02:33
tianleiwu added a commit that referenced this pull request May 3, 2026
…gging (#28274)

### Description

Move `owned_session_logger_` declaration before `execution_providers_`
in `InferenceSession` so the logger outlives EPs during member
destruction.

C++ destroys members in reverse declaration order. Previously:

```
~owned_session_logger_   (L905)  // logger freed
     ↓
~execution_providers_    (L745)  // EP teardown logs via dangling pointer → crash
```

After this change:

```
~execution_providers_    (L750)  // EP teardown logs safely ✅
     ↓
~owned_session_logger_   (L744)  // logger freed, no remaining users
```

### Motivation and Context

Plugin EPs receive an `OrtLogger*` via `OrtEpFactory::CreateEp()`.
During session destruction, EP teardown callbacks (e.g.,
`ReleaseNodeComputeInfos`) may log through this pointer. Because
`owned_session_logger_` was declared after `execution_providers_`, the
logger was already freed when EPs destructed — a use-after-free that
crashes deterministically under VERBOSE logging.

Affects all Plugin EPs that log in any teardown path. Reproduced with
OpenVINO Plugin EP via `webnn_graph_impl_fuzzer` at VERBOSE level.

Fixes #28234

### Tests Added

Added regression tests in
`onnxruntime/test/framework/inference_session_test.cc`:

- **`LoggingOnDestroyExecutionProvider`** — A mock EP that logs via its
stored logger pointer in its destructor. If the logger has been freed,
this triggers a use-after-free (detected by ASan or as a segfault).
- **`SessionLoggerOutlivesEPsOnDestruction`** — Creates a session with
VERBOSE logging and the mock EP, then destroys the session. Verifies
that the logger was valid during EP teardown and that the teardown log
message was captured.
- **`SessionLoggerOutlivesEPsWithMultipleEPs`** — Same scenario with two
mock EPs (distinct type names) to confirm all registered EPs can safely
log during teardown.

### Verification

Confirmed the tests are effective regression tests:

| Scenario | Result |
|----------|--------|
| **With fix** (logger declared before EPs) | Both tests pass ✅ |
| **Without fix** (logger declared after EPs, original bug) |
`SessionLoggerOutlivesEPsOnDestruction` crashes with **Segmentation
fault** (exit code 139) — use-after-free ❌ |

This proves the member declaration order is the critical factor, and the
tests will catch any future regression that reorders these members.

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: tianleiwu <30328909+tianleiwu@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Session logger destroyed before EP teardown callbacks — use-after-free under VERBOSE logging

5 participants