prevent AOTI library collisions between delegates by shoumikhin · Pull Request #21287 · pytorch/executorch

shoumikhin · 2026-07-23T16:53:35Z

Summary:
The CUDA backend writes each compiled AOTInductor library to a temporary file,
named from its content key plus the process id, before loading it. Two identical
CUDA partitions can share the same content key, so both delegates computed the
same path; loading the second library overwrote the first while it was still in
use, and symbol lookup could then crash.

Append a per-process atomic counter to the temporary file name so every delegate
in a process gets a distinct path. The file is still removed when the delegate is
destroyed, as before; only the name changes.

Differential Revision: D113322459

pytorch-bot · 2026-07-23T16:53:40Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21287

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 23 Pending

As of commit 538cc90 with merge base 96e87e3 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

meta-codesync · 2026-07-23T16:53:53Z

@shoumikhin has exported this pull request. If you are a Meta employee, you can view the originating Diff in D113322459.

github-actions · 2026-07-23T16:54:44Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Copilot

Pull request overview

This PR fixes a correctness issue in the CUDA backend’s AOTInductor temporary shared-library handling by ensuring each delegate instance in a process writes to a distinct temporary .so path, preventing overwrites and subsequent symbol-lookup crashes when multiple identical partitions share the same so_blob_key.

Changes:

Add a per-process std::atomic<uint64_t> counter to disambiguate temporary .so filenames for delegates that share the same content key.
Update the temp library naming scheme from {key}{pid}.so to {key}{pid}_{counter}.so, while keeping the existing cleanup behavior on delegate destruction.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Summary: The CUDA backend writes each compiled AOTInductor library to a temporary file, named from its content key plus the process id, before loading it. Two identical CUDA partitions can share the same content key, so both delegates computed the same path; loading the second library overwrote the first while it was still in use, and symbol lookup could then crash. Append a per-process atomic counter to the temporary file name so every delegate in a process gets a distinct path. The file is still removed when the delegate is destroyed, as before; only the name changes. Differential Revision: D113322459

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Summary: The CUDA backend writes each compiled AOTInductor library to a temporary file, named from its content key plus the process id, before loading it. Two identical CUDA partitions can share the same content key, so both delegates computed the same path; loading the second library overwrote the first while it was still in use, and symbol lookup could then crash. Append a per-process atomic counter to the temporary file name so every delegate in a process gets a distinct path. The file is still removed when the delegate is destroyed, as before; only the name changes. Differential Revision: D113322459

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

shoumikhin · 2026-07-23T21:53:43Z

+    filesystem::path so_path = temp_dir /
+        (so_blob_key + to_string(get_process_id()) + "_" +
+         to_string(so_file_counter.fetch_add(1, std::memory_order_relaxed)) +
+         ".so");


Fixed. so_blob_key is now kept out of the filename entirely — it already selected the blob via get_data() above, so it has no role in the path. The temp file is now executorch_cuda_<pid>_<counter>.so, where the pid gives cross-process uniqueness and the atomic counter gives per-process uniqueness. This removes the delimiter-ambiguity/collision for variable-length (legacy) keys.

Summary: The CUDA backend writes each compiled AOTInductor library to a temporary file, named from its content key plus the process id, before loading it. Two identical CUDA partitions can share the same content key, so both delegates computed the same path; loading the second library overwrote the first while it was still in use, and symbol lookup could then crash. Append a per-process atomic counter to the temporary file name so every delegate in a process gets a distinct path. The file is still removed when the delegate is destroyed, as before; only the name changes. Differential Revision: D113322459

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Summary: The CUDA backend writes each compiled AOTInductor library to a temporary file, named from its content key plus the process id, before loading it. Two identical CUDA partitions can share the same content key, so both delegates computed the same path; loading the second library overwrote the first while it was still in use, and symbol lookup could then crash. Append a per-process atomic counter to the temporary file name so every delegate in a process gets a distinct path. The file is still removed when the delegate is destroyed, as before; only the name changes. Differential Revision: D113322459

Copilot AI review requested due to automatic review settings July 23, 2026 16:53

meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 23, 2026

meta-codesync Bot added the meta-exported label Jul 23, 2026

meta-codesync Bot had a problem deploying to cadence July 23, 2026 16:54 Failure

Copilot started reviewing on behalf of shoumikhin July 23, 2026 16:54 View session