Skip to content

Fix NVTX profiling: restore missing node_compute_range_.Begin() call#28273

Merged
tianleiwu merged 2 commits intomainfrom
copilot/fix-nvtx-profiling-error
May 1, 2026
Merged

Fix NVTX profiling: restore missing node_compute_range_.Begin() call#28273
tianleiwu merged 2 commits intomainfrom
copilot/fix-nvtx-profiling-error

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Apr 29, 2026

Description

Restores the missing node_compute_range_.Begin() call in KernelScope's constructor. The corresponding End() in the destructor was kept, so every kernel execution hits:

ORT_ENFORCE(is_begin_called_, "End must be called after calling Begin.");

Motivation and Context

PR #26846 refactored sequential_executor.cc into RAII scope classes but dropped the node_compute_range_.Begin() call during the move. Any build with --enable_nvtx_profile fails at runtime on the first kernel execution.

The Begin() call on node_compute_range_ was accidentally removed in
PR #26846 (commit 879ec03) while the corresponding End() call in the
KernelScope destructor was kept. This caused the runtime error
"End must be called after calling Begin" when building with
--enable_nvtx_profile.

Agent-Logs-Url: https://github.com/microsoft/onnxruntime/sessions/9daf7cf0-e1be-4647-bce7-ccda27743b96

Co-authored-by: tianleiwu <30328909+tianleiwu@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix NVTX profiling failure with End must be called after calling Begin Fix NVTX profiling: restore missing node_compute_range_.Begin() call Apr 29, 2026
Copilot AI requested a review from tianleiwu April 29, 2026 17:14
@tianleiwu
Copy link
Copy Markdown
Contributor

@copilot Please run an experiment to verify the issue indeed exist, and this fix actualy fix the issue.

Copy link
Copy Markdown
Contributor Author

Copilot AI commented Apr 30, 2026

@copilot Please run an experiment to verify the issue indeed exist, and this fix actualy fix the issue.

I verified this through code analysis since this environment doesn't have CUDA/NVTX hardware for a runtime test. Here's the evidence:

Bug exists in base commit (8a77597):

$ git show 8a77597:onnxruntime/core/framework/sequential_executor.cc | grep -n "node_compute_range_"
334:        node_compute_range_(MakeString(...)    // constructed
400:    node_compute_range_.End();                 // End() called in destructor  
466:  profile::NvtxRangeCreator node_compute_range_;  // member declaration

No node_compute_range_.Begin() exists anywhere — it was dropped during the RAII refactor in PR #26846.

RangeCreatorBase::End() (nvtx_profile.h:70-71) enforces:

void End() {
    ORT_ENFORCE(is_begin_called_, "End must be called after calling Begin.");

So the first KernelScope destructor will always crash with ORT_ENFORCE failure because is_begin_called_ is false (initialized in constructor at line 48).

Fix (commit 64897f7) adds node_compute_range_.Begin() at line 367, which is executed in the KernelScope constructor before the destructor's End() can run — restoring the proper Begin→End lifecycle.

Copy link
Copy Markdown
Member

@yuslepukhin yuslepukhin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@tianleiwu tianleiwu marked this pull request as ready for review May 1, 2026 02:35
@tianleiwu tianleiwu merged commit 76655a0 into main May 1, 2026
90 checks passed
@tianleiwu tianleiwu deleted the copilot/fix-nvtx-profiling-error branch May 1, 2026 02:36
tianleiwu added a commit that referenced this pull request May 3, 2026
…28273)

### Description

Restores the missing `node_compute_range_.Begin()` call in
`KernelScope`'s constructor. The corresponding `End()` in the destructor
was kept, so every kernel execution hits:

```
ORT_ENFORCE(is_begin_called_, "End must be called after calling Begin.");
```

### Motivation and Context

PR #26846 refactored `sequential_executor.cc` into RAII scope classes
but dropped the `node_compute_range_.Begin()` call during the move. Any
build with `--enable_nvtx_profile` fails at runtime on the first kernel
execution.

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: tianleiwu <30328909+tianleiwu@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

NVTX profiling fails with End must be called after calling Begin

3 participants