Skip to content

fixing GIL deadlock in mqpu async kernel compilation#4430

Merged
sacpis merged 2 commits intoNVIDIA:mainfrom
sacpis:fix_gil_deadlock
May 1, 2026
Merged

fixing GIL deadlock in mqpu async kernel compilation#4430
sacpis merged 2 commits intoNVIDIA:mainfrom
sacpis:fix_gil_deadlock

Conversation

@sacpis
Copy link
Copy Markdown
Collaborator

@sacpis sacpis commented May 1, 2026

The test test_mpi_mqpu.py was hanging where PyGILState_Ensure() on the worker waits for the main thread to release the GIL, but the main thread is in asyncResult.get() holding the GIL.

addPythonSignalInstrumentation now skips installing the per-pass PyGILState_Ensure based instrumentation when called from a thread that doesn't currently hold the GIL.

@sacpis
fixing GIL deadlock in mqpu async kernel compilation
6673b59 
Signed-off-by: Sachin Pisal <spisal@nvidia.com>
@sacpis sacpis requested a review from 1tnguyen May 1, 2026 03:32
1tnguyen
1tnguyen approved these changes May 1, 2026
View reviewed changes
Copy link
Copy Markdown
Collaborator

@1tnguyen 1tnguyen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @sacpis!

@sacpis sacpis enabled auto-merge May 1, 2026 03:49
@sacpis sacpis added this pull request to the merge queue May 1, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks May 1, 2026
@sacpis sacpis added this pull request to the merge queue May 1, 2026
@mitchdz mitchdz removed this pull request from the merge queue due to a manual request May 1, 2026
@sacpis sacpis enabled auto-merge May 1, 2026 13:29
@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 1, 2026 •
edited
Loading

CI Summary — ❌ failed

Run #25228687707 · trigger merge_group · ✅ 12 · ⏩ 0 · ❌ 1 · ⛔ 0

❌ Failed or cancelled
Job Result Link
binaries ❌ failure view
Top-level jobs (13)
Job Result
binaries ❌ failure
build_and_test ✅ success
config_devdeps ✅ success
config_source_build ✅ success
config_wheeldeps ✅ success
devdeps ✅ success
docker_image ✅ success
gen_code_coverage ✅ success
metadata ✅ success
python_metapackages ✅ success
python_wheels ✅ success
source_build ✅ success
wheeldeps ✅ success
All sub-jobs (184) — every matrix leg, with links
Job Status Link
Build and test (amd64, clang16, openmpi) / Dev environment (Debug) ✅ success view
Build and test (amd64, clang16, openmpi) / Dev environment (Python) ✅ success view
Build and test (amd64, gcc11, openmpi) / Dev environment (Debug) ✅ success view
Build and test (amd64, gcc11, openmpi) / Dev environment (Python) ✅ success view
Build and test (amd64, gcc12, openmpi) / Dev environment (Debug) ✅ success view
Build and test (amd64, gcc12, openmpi) / Dev environment (Python) ✅ success view
Build and test (arm64, clang16, openmpi) / Dev environment (Debug) ✅ success view
Build and test (arm64, clang16, openmpi) / Dev environment (Python) ✅ success view
Build and test (arm64, gcc11, openmpi) / Dev environment (Debug) ✅ success view
Build and test (arm64, gcc11, openmpi) / Dev environment (Python) ✅ success view
Build and test (arm64, gcc12, openmpi) / Dev environment (Debug) ✅ success view
Build and test (arm64, gcc12, openmpi) / Dev environment (Python) ✅ success view
CI Summary ❔ in_progress view
Configure build (devdeps) ✅ success view
Configure build (source_build) ✅ success view
Configure build (wheeldeps) ✅ success view
Create CUDA Quantum installer (amd64, 12.6) / Build CUDA Quantum assets ❌ failure view
Create CUDA Quantum installer (amd64, 12.6) / Minimal OpenMPI installation ✅ success view
Create CUDA Quantum installer (amd64, 12.6) / Prepare cache clean-up ✅ success view
Create CUDA Quantum installer (amd64, 12.6) / Prepare validation ✅ success view
Create CUDA Quantum installer (amd64, 12.6) / Staging ⏩ skipped view
Create CUDA Quantum installer (amd64, 12.6) / Validate installer ⏩ skipped view
Create CUDA Quantum installer (amd64, 13.0) / Build CUDA Quantum assets ✅ success view
Create CUDA Quantum installer (amd64, 13.0) / Minimal OpenMPI installation ✅ success view
Create CUDA Quantum installer (amd64, 13.0) / Prepare cache clean-up ✅ success view
Create CUDA Quantum installer (amd64, 13.0) / Prepare validation ✅ success view
Create CUDA Quantum installer (amd64, 13.0) / Staging ⏩ skipped view
Create CUDA Quantum installer (amd64, 13.0) / Validate installer (debian:12) ✅ success view
Create CUDA Quantum installer (amd64, 13.0) / Validate installer (fedora:42) ✅ success view
Create CUDA Quantum installer (amd64, 13.0) / Validate installer (opensuse/leap:15.5) ✅ success view
Create CUDA Quantum installer (amd64, 13.0) / Validate installer (ubuntu:22.04) ✅ success view
Create CUDA Quantum installer (arm64, 12.6) / Build CUDA Quantum assets ✅ success view
Create CUDA Quantum installer (arm64, 12.6) / Minimal OpenMPI installation ✅ success view
Create CUDA Quantum installer (arm64, 12.6) / Prepare cache clean-up ✅ success view
Create CUDA Quantum installer (arm64, 12.6) / Prepare validation ✅ success view
Create CUDA Quantum installer (arm64, 12.6) / Staging ⏩ skipped view
Create CUDA Quantum installer (arm64, 12.6) / Validate installer (redhat/ubi9:9.6) ✅ success view
Create CUDA Quantum installer (arm64, 12.6) / Validate installer (ubuntu:22.04) ✅ success view
Create CUDA Quantum installer (arm64, 13.0) / Build CUDA Quantum assets ✅ success view
Create CUDA Quantum installer (arm64, 13.0) / Minimal OpenMPI installation ✅ success view
Create CUDA Quantum installer (arm64, 13.0) / Prepare cache clean-up ✅ success view
Create CUDA Quantum installer (arm64, 13.0) / Prepare validation ✅ success view
Create CUDA Quantum installer (arm64, 13.0) / Staging ⏩ skipped view
Create CUDA Quantum installer (arm64, 13.0) / Validate installer (ubuntu:22.04) ✅ success view
Create Docker images (amd64) / Documentation ✅ success view
Create Docker images (amd64) / Metadata ✅ success view
Create Docker images (amd64) / Prepare cache clean-up ✅ success view
Create Docker images (amd64) / Staging ⏩ skipped view
Create Docker images (amd64) / Validation ✅ success view
Create Docker images (amd64) / cuda-quantum (release) ✅ success view
Create Docker images (amd64) / cuda-quantum-dev (debug) ✅ success view
Create Docker images (amd64) / open-mpi ⏩ skipped view
Create Docker images (arm64) / Documentation ⏩ skipped view
Create Docker images (arm64) / Metadata ✅ success view
Create Docker images (arm64) / Prepare cache clean-up ✅ success view
Create Docker images (arm64) / Staging ⏩ skipped view
Create Docker images (arm64) / Validation ✅ success view
Create Docker images (arm64) / cuda-quantum (release) ✅ success view
Create Docker images (arm64) / cuda-quantum-dev (debug) ✅ success view
Create Docker images (arm64) / open-mpi ⏩ skipped view
Create Python metapackages / Build Python metapackages ✅ success view
Create Python metapackages / Test Python metapackages (12.6, 3.11) ✅ success view
Create Python metapackages / Test Python metapackages (12.6, 3.13) ✅ success view
Create Python metapackages / Test Python metapackages (13.0, 3.11) ✅ success view
Create Python metapackages / Test Python metapackages (13.0, 3.13) ✅ success view
Create Python metapackages / Test Python metapackages (3.11) ✅ success view
Create Python metapackages / Test Python metapackages (3.13) ✅ success view
Create Python wheels (amd64, 3.11, 12.6) / Build Python 3.11 wheel ✅ success view
Create Python wheels (amd64, 3.11, 12.6) / Prepare validation ✅ success view
Create Python wheels (amd64, 3.11, 12.6) / Staging ⏩ skipped view
Create Python wheels (amd64, 3.11, 12.6) / Validate wheel (debian:12) ✅ success view
Create Python wheels (amd64, 3.11, 12.6) / Validate wheel (debian:12, --user) ✅ success view
Create Python wheels (amd64, 3.11, 12.6) / Validate wheel (fedora:42) ✅ success view
Create Python wheels (amd64, 3.11, 12.6) / Validate wheel (fedora:42, --user) ✅ success view
Create Python wheels (amd64, 3.11, 12.6) / Validate wheel (redhat/ubi8:8.10) ✅ success view
Create Python wheels (amd64, 3.11, 12.6) / Validate wheel (redhat/ubi8:8.10, --user) ✅ success view
Create Python wheels (amd64, 3.11, 12.6) / Validate wheel (redhat/ubi9:9.6) ✅ success view
Create Python wheels (amd64, 3.11, 12.6) / Validate wheel (redhat/ubi9:9.6, --user) ✅ success view
Create Python wheels (amd64, 3.11, 12.6) / Validate wheel (ubuntu:22.04) ✅ success view
Create Python wheels (amd64, 3.11, 12.6) / Validate wheel (ubuntu:22.04, --user) ✅ success view
Create Python wheels (amd64, 3.11, 13.0) / Build Python 3.11 wheel ✅ success view
Create Python wheels (amd64, 3.11, 13.0) / Prepare validation ✅ success view
Create Python wheels (amd64, 3.11, 13.0) / Staging ⏩ skipped view
Create Python wheels (amd64, 3.11, 13.0) / Validate wheel (debian:12) ✅ success view
Create Python wheels (amd64, 3.11, 13.0) / Validate wheel (debian:12, --user) ✅ success view
Create Python wheels (amd64, 3.11, 13.0) / Validate wheel (fedora:42) ✅ success view
Create Python wheels (amd64, 3.11, 13.0) / Validate wheel (fedora:42, --user) ✅ success view
Create Python wheels (amd64, 3.11, 13.0) / Validate wheel (redhat/ubi8:8.10) ✅ success view
Create Python wheels (amd64, 3.11, 13.0) / Validate wheel (redhat/ubi8:8.10, --user) ✅ success view
Create Python wheels (amd64, 3.11, 13.0) / Validate wheel (redhat/ubi9:9.6) ✅ success view
Create Python wheels (amd64, 3.11, 13.0) / Validate wheel (redhat/ubi9:9.6, --user) ✅ success view
Create Python wheels (amd64, 3.11, 13.0) / Validate wheel (ubuntu:22.04) ✅ success view
Create Python wheels (amd64, 3.11, 13.0) / Validate wheel (ubuntu:22.04, --user) ✅ success view
Create Python wheels (amd64, 3.13, 12.6) / Build Python 3.13 wheel ✅ success view
Create Python wheels (amd64, 3.13, 12.6) / Prepare validation ✅ success view
Create Python wheels (amd64, 3.13, 12.6) / Staging ⏩ skipped view
Create Python wheels (amd64, 3.13, 12.6) / Validate wheel (fedora:42) ✅ success view
Create Python wheels (amd64, 3.13, 12.6) / Validate wheel (fedora:42, --user) ✅ success view
Create Python wheels (amd64, 3.13, 13.0) / Build Python 3.13 wheel ✅ success view
Create Python wheels (amd64, 3.13, 13.0) / Prepare validation ✅ success view
Create Python wheels (amd64, 3.13, 13.0) / Staging ⏩ skipped view
Create Python wheels (amd64, 3.13, 13.0) / Validate wheel (fedora:42) ✅ success view
Create Python wheels (amd64, 3.13, 13.0) / Validate wheel (fedora:42, --user) ✅ success view
Create Python wheels (arm64, 3.11, 12.6) / Build Python 3.11 wheel ✅ success view
Create Python wheels (arm64, 3.11, 12.6) / Prepare validation ✅ success view
Create Python wheels (arm64, 3.11, 12.6) / Staging ⏩ skipped view
Create Python wheels (arm64, 3.11, 12.6) / Validate wheel (debian:12) ✅ success view
Create Python wheels (arm64, 3.11, 12.6) / Validate wheel (debian:12, --user) ✅ success view
Create Python wheels (arm64, 3.11, 12.6) / Validate wheel (fedora:42) ✅ success view
Create Python wheels (arm64, 3.11, 12.6) / Validate wheel (fedora:42, --user) ✅ success view
Create Python wheels (arm64, 3.11, 12.6) / Validate wheel (redhat/ubi8:8.10) ✅ success view
Create Python wheels (arm64, 3.11, 12.6) / Validate wheel (redhat/ubi8:8.10, --user) ✅ success view
Create Python wheels (arm64, 3.11, 12.6) / Validate wheel (redhat/ubi9:9.6) ✅ success view
Create Python wheels (arm64, 3.11, 12.6) / Validate wheel (redhat/ubi9:9.6, --user) ✅ success view
Create Python wheels (arm64, 3.11, 12.6) / Validate wheel (ubuntu:22.04) ✅ success view
Create Python wheels (arm64, 3.11, 12.6) / Validate wheel (ubuntu:22.04, --user) ✅ success view
Create Python wheels (arm64, 3.11, 13.0) / Build Python 3.11 wheel ✅ success view
Create Python wheels (arm64, 3.11, 13.0) / Prepare validation ✅ success view
Create Python wheels (arm64, 3.11, 13.0) / Staging ⏩ skipped view
Create Python wheels (arm64, 3.11, 13.0) / Validate wheel (debian:12) ✅ success view
Create Python wheels (arm64, 3.11, 13.0) / Validate wheel (debian:12, --user) ✅ success view
Create Python wheels (arm64, 3.11, 13.0) / Validate wheel (fedora:42) ✅ success view
Create Python wheels (arm64, 3.11, 13.0) / Validate wheel (fedora:42, --user) ✅ success view
Create Python wheels (arm64, 3.11, 13.0) / Validate wheel (redhat/ubi8:8.10) ✅ success view
Create Python wheels (arm64, 3.11, 13.0) / Validate wheel (redhat/ubi8:8.10, --user) ✅ success view
Create Python wheels (arm64, 3.11, 13.0) / Validate wheel (redhat/ubi9:9.6) ✅ success view
Create Python wheels (arm64, 3.11, 13.0) / Validate wheel (redhat/ubi9:9.6, --user) ✅ success view
Create Python wheels (arm64, 3.11, 13.0) / Validate wheel (ubuntu:22.04) ✅ success view
Create Python wheels (arm64, 3.11, 13.0) / Validate wheel (ubuntu:22.04, --user) ✅ success view
Create Python wheels (arm64, 3.13, 12.6) / Build Python 3.13 wheel ✅ success view
Create Python wheels (arm64, 3.13, 12.6) / Prepare validation ✅ success view
Create Python wheels (arm64, 3.13, 12.6) / Staging ⏩ skipped view
Create Python wheels (arm64, 3.13, 12.6) / Validate wheel (fedora:42) ✅ success view
Create Python wheels (arm64, 3.13, 12.6) / Validate wheel (fedora:42, --user) ✅ success view
Create Python wheels (arm64, 3.13, 13.0) / Build Python 3.13 wheel ✅ success view
Create Python wheels (arm64, 3.13, 13.0) / Prepare validation ✅ success view
Create Python wheels (arm64, 3.13, 13.0) / Staging ⏩ skipped view
Create Python wheels (arm64, 3.13, 13.0) / Validate wheel (fedora:42) ✅ success view
Create Python wheels (arm64, 3.13, 13.0) / Validate wheel (fedora:42, --user) ✅ success view
Gen code coverage (amd64, clang16) / gen_code_coverage ✅ success view
Load dependencies (amd64, clang16) / Caching ✅ success view
Load dependencies (amd64, clang16) / Finalize ✅ success view
Load dependencies (amd64, clang16) / Metadata ✅ success view
Load dependencies (amd64, gcc11) / Caching ✅ success view
Load dependencies (amd64, gcc11) / Finalize ✅ success view
Load dependencies (amd64, gcc11) / Metadata ✅ success view
Load dependencies (amd64, gcc12) / Caching ✅ success view
Load dependencies (amd64, gcc12) / Finalize ✅ success view
Load dependencies (amd64, gcc12) / Metadata ✅ success view
Load dependencies (arm64, clang16) / Caching ✅ success view
Load dependencies (arm64, clang16) / Finalize ✅ success view
Load dependencies (arm64, clang16) / Metadata ✅ success view
Load dependencies (arm64, gcc11) / Caching ✅ success view
Load dependencies (arm64, gcc11) / Finalize ✅ success view
Load dependencies (arm64, gcc11) / Metadata ✅ success view
Load dependencies (arm64, gcc12) / Caching ✅ success view
Load dependencies (arm64, gcc12) / Finalize ✅ success view
Load dependencies (arm64, gcc12) / Metadata ✅ success view
Load source build cache (amd64, 12.6) / Caching ✅ success view
Load source build cache (amd64, 12.6) / Finalize ✅ success view
Load source build cache (amd64, 12.6) / Metadata ✅ success view
Load source build cache (amd64, 13.0) / Caching ✅ success view
Load source build cache (amd64, 13.0) / Finalize ✅ success view
Load source build cache (amd64, 13.0) / Metadata ✅ success view
Load source build cache (arm64, 12.6) / Caching ✅ success view
Load source build cache (arm64, 12.6) / Finalize ✅ success view
Load source build cache (arm64, 12.6) / Metadata ✅ success view
Load source build cache (arm64, 13.0) / Caching ✅ success view
Load source build cache (arm64, 13.0) / Finalize ✅ success view
Load source build cache (arm64, 13.0) / Metadata ✅ success view
Load wheel dependencies (amd64, 12.6) / Caching ✅ success view
Load wheel dependencies (amd64, 12.6) / Finalize ✅ success view
Load wheel dependencies (amd64, 12.6) / Metadata ✅ success view
Load wheel dependencies (amd64, 13.0) / Caching ✅ success view
Load wheel dependencies (amd64, 13.0) / Finalize ✅ success view
Load wheel dependencies (amd64, 13.0) / Metadata ✅ success view
Load wheel dependencies (arm64, 12.6) / Caching ✅ success view
Load wheel dependencies (arm64, 12.6) / Finalize ✅ success view
Load wheel dependencies (arm64, 12.6) / Metadata ✅ success view
Load wheel dependencies (arm64, 13.0) / Caching ✅ success view
Load wheel dependencies (arm64, 13.0) / Finalize ✅ success view
Load wheel dependencies (arm64, 13.0) / Metadata ✅ success view
Prepare cache clean-up ✅ success view
Retrieve PR info ✅ success view
⚠️ Required checks (29/35) — 6 missing — declared in .github/required-checks.yml for merge_group
Required check Status Link
Build and test (arm64, gcc11, openmpi) / Dev environment (Debug) ✅ success view
Build and test (arm64, gcc11, openmpi) / Dev environment (Python) ✅ success view
Build and test (arm64, gcc12, openmpi) / Dev environment (Debug) ✅ success view
Build and test (arm64, gcc12, openmpi) / Dev environment (Python) ✅ success view
Create CUDA Quantum installer (amd64, 12.6) / Build CUDA Quantum assets ❌ failure view
Create CUDA Quantum installer (amd64, 12.6) / Minimal OpenMPI installation ✅ success view
Create CUDA Quantum installer (amd64, 12.6) / Validate installer (debian:12) ❔ missing
Create CUDA Quantum installer (amd64, 12.6) / Validate installer (fedora:42) ❔ missing
Create CUDA Quantum installer (amd64, 12.6) / Validate installer (opensuse/leap:15.5) ❔ missing
Create CUDA Quantum installer (amd64, 12.6) / Validate installer (redhat/ubi9:9.6) ❔ missing
Create CUDA Quantum installer (amd64, 12.6) / Validate installer (ubuntu:22.04) ❔ missing
Create CUDA Quantum installer (arm64, 12.6) / Build CUDA Quantum assets ✅ success view
Create CUDA Quantum installer (arm64, 12.6) / Minimal OpenMPI installation ✅ success view
Create CUDA Quantum installer (arm64, 12.6) / Validate installer (redhat/ubi9:9.6) ✅ success view
Create CUDA Quantum installer (arm64, 12.6) / Validate installer (ubuntu:22.04) ✅ success view
Create Docker images (amd64) / Documentation ✅ success view
Create Docker images (amd64) / Validation ✅ success view
Create Docker images (arm64) / Validation ✅ success view
Create Python metapackages / Build Python metapackages ✅ success view
Create Python metapackages / Test Python metapackages (3.11) ✅ success view
Create Python metapackages / Test Python metapackages (3.13) ✅ success view
Create Python metapackages / Test Python metapackages (12.6, 3.11) ✅ success view
Create Python metapackages / Test Python metapackages (12.6, 3.13) ✅ success view
Create Python wheels (amd64, 3.11, 12.6) / Validate wheel (debian:12, --user) ✅ success view
Create Python wheels (amd64, 3.11, 12.6) / Validate wheel (debian:12) ✅ success view
Create Python wheels (amd64, 3.11, 12.6) / Validate wheel (fedora:42, --user) ✅ success view
Create Python wheels (amd64, 3.11, 12.6) / Validate wheel (fedora:42) ✅ success view
Create Python wheels (amd64, 3.11, 12.6) / Validate wheel (redhat/ubi8:8.10, --user) ✅ success view
Create Python wheels (amd64, 3.11, 12.6) / Validate wheel (redhat/ubi8:8.10) ✅ success view
Create Python wheels (arm64, 3.11, 12.6) / Validate wheel (debian:12, --user) ✅ success view
Create Python wheels (arm64, 3.11, 12.6) / Validate wheel (debian:12) ✅ success view
Create Python wheels (arm64, 3.11, 12.6) / Validate wheel (fedora:42, --user) ✅ success view
Create Python wheels (arm64, 3.11, 12.6) / Validate wheel (fedora:42) ✅ success view
Create Python wheels (arm64, 3.11, 12.6) / Validate wheel (redhat/ubi8:8.10, --user) ✅ success view
Create Python wheels (arm64, 3.11, 12.6) / Validate wheel (redhat/ubi8:8.10) ✅ success view

@sacpis sacpis added this pull request to the merge queue May 1, 2026
Merged via the queue into NVIDIA:main with commit 034de2f May 1, 2026
91 checks passed
@sacpis sacpis deleted the fix_gil_deadlock branch May 1, 2026 22:36
github-actions Bot pushed a commit that referenced this pull request May 1, 2026
@cuda-quantum-bot
Cleaning up docs preview for PR #4430.
7beb849
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@1tnguyen 1tnguyen 1tnguyen approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@sacpis @1tnguyen