Elide redundant same-device copies in PropagateDevicePass (#20914)#20914
Merged
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20914
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 44fea49 with merge base e500fee ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
Contributor
|
@Conarnar has exported this pull request. If you are a Meta employee, you can view the originating Diff in D111992997. |
This PR needs a
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves PropagateDevicePass by removing redundant host/device transfer pairs that occur when two delegated regions execute back-to-back on the same device, avoiding an unnecessary device→host→device round-trip while preserving true cross-device transfers.
Changes:
- Add a third phase to
PropagateDevicePassthat detects and elides_h2d_copy(_d2h_copy(x))whenxis already on the same target device. - Add targeted tests that construct two consecutive delegates with per-op target devices and verify redundant same-device copies are removed while cross-device copies remain.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
exir/passes/propagate_device_pass.py |
Adds an optimization pass phase to remove redundant same-device D2H→H2D copy chains after per-boundary copy insertion. |
exir/tests/test_propagate_device_pass.py |
Adds a custom partitioner to force per-add delegation and two tests validating same-device elision and cross-device preservation. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
meta-codesync
Bot
changed the title
Conarnar
added a commit
to Conarnar/executorch
that referenced
this pull request
Jul 14, 2026
) Summary: PropagateDevicePass inserts H2D/D2H copies per delegate boundary independently (an H2D before every device delegate input, a D2H after every device delegate output). When two delegates run consecutively on the *same* device, this leaves a redundant device -> host -> device round-trip between them: the first delegate's output is copied to host and immediately copied back to the same device for the second delegate. This adds a third phase to PropagateDevicePass that elides `_h2d_copy(_d2h_copy(x))` when the tensor before the D2H already lives on the device the H2D copies back to. The device comparison is the correctness guard: a genuine cross-device move (e.g. cuda:0 -> host -> cuda:1) has differing devices and is preserved. A D2H that still feeds other (host) consumers keeps those users and is removed only once it becomes dead. The phase runs only when enable_non_cpu_memory_planning is set (i.e. when copies were actually inserted). This closes the "redundant copies between back-to-back delegates on the same device" limitation from the device-aware export work. Differential Revision: D111992997
Conarnar
force-pushed
the
export-D111992997
branch
from
July 14, 2026 21:15
9e4ab31 to
ce30c8b
Compare
Conarnar
added a commit
to Conarnar/executorch
that referenced
this pull request
Jul 14, 2026
) Summary: PropagateDevicePass inserts H2D/D2H copies per delegate boundary independently (an H2D before every device delegate input, a D2H after every device delegate output). When two delegates run consecutively on the *same* device, this leaves a redundant device -> host -> device round-trip between them: the first delegate's output is copied to host and immediately copied back to the same device for the second delegate. This adds a third phase to PropagateDevicePass that elides `_h2d_copy(_d2h_copy(x))` when the tensor before the D2H already lives on the device the H2D copies back to. The device comparison is the correctness guard: a genuine cross-device move (e.g. cuda:0 -> host -> cuda:1) has differing devices and is preserved. A D2H that still feeds other (host) consumers keeps those users and is removed only once it becomes dead. The phase runs only when enable_non_cpu_memory_planning is set (i.e. when copies were actually inserted). This closes the "redundant copies between back-to-back delegates on the same device" limitation from the device-aware export work. Differential Revision: D111992997
Conarnar
force-pushed
the
export-D111992997
branch
from
July 14, 2026 21:25
ce30c8b to
7c91a10
Compare
meta-codesync
Bot
changed the title
Conarnar
force-pushed
the
export-D111992997
branch
from
July 14, 2026 23:14
7c91a10 to
b2d7712
Compare
Conarnar
force-pushed
the
export-D111992997
branch
from
July 14, 2026 23:20
b2d7712 to
596c42e
Compare
Conarnar
force-pushed
the
export-D111992997
branch
from
July 14, 2026 23:26
596c42e to
e07e470
Compare
meta-codesync
Bot
changed the title
Conarnar
added a commit
to Conarnar/executorch
that referenced
this pull request
Jul 14, 2026
) Summary: PropagateDevicePass inserts H2D/D2H copies per delegate boundary independently (an H2D before every device delegate input, a D2H after every device delegate output). When two delegates run consecutively on the *same* device, this leaves a redundant device -> host -> device round-trip between them: the first delegate's output is copied to host and immediately copied back to the same device for the second delegate. This adds a third phase to PropagateDevicePass that elides `_h2d_copy(_d2h_copy(x))` when the tensor before the D2H already lives on the device the H2D copies back to. The device comparison is the correctness guard: a genuine cross-device move (e.g. cuda:0 -> host -> cuda:1) has differing devices and is preserved. A D2H that still feeds other (host) consumers keeps those users and is removed only once it becomes dead. The phase runs only when enable_non_cpu_memory_planning is set (i.e. when copies were actually inserted). This closes the "redundant copies between back-to-back delegates on the same device" limitation from the device-aware export work. Differential Revision: D111992997
Conarnar
force-pushed
the
export-D111992997
branch
from
July 14, 2026 23:31
e07e470 to
0ae723a
Compare
Conarnar
added a commit
to Conarnar/executorch
that referenced
this pull request
Jul 15, 2026
) Summary: PropagateDevicePass inserts H2D/D2H copies per delegate boundary independently (an H2D before every device delegate input, a D2H after every device delegate output). When two delegates run consecutively on the *same* device, this leaves a redundant device -> host -> device round-trip between them: the first delegate's output is copied to host and immediately copied back to the same device for the second delegate. This adds a third phase to PropagateDevicePass that elides `_h2d_copy(_d2h_copy(x))` when the tensor before the D2H already lives on the device the H2D copies back to. The device comparison is the correctness guard: a genuine cross-device move (e.g. cuda:0 -> host -> cuda:1) has differing devices and is preserved. A D2H that still feeds other (host) consumers keeps those users and is removed only once it becomes dead. The phase runs only when enable_non_cpu_memory_planning is set (i.e. when copies were actually inserted). This closes the "redundant copies between back-to-back delegates on the same device" limitation from the device-aware export work. Differential Revision: D111992997
Conarnar
force-pushed
the
export-D111992997
branch
from
July 15, 2026 01:11
0ae723a to
ec7ba46
Compare
Conarnar
added a commit
to Conarnar/executorch
that referenced
this pull request
Jul 15, 2026
) Summary: PropagateDevicePass inserts H2D/D2H copies per delegate boundary independently (an H2D before every device delegate input, a D2H after every device delegate output). When two delegates run consecutively on the *same* device, this leaves a redundant device -> host -> device round-trip between them: the first delegate's output is copied to host and immediately copied back to the same device for the second delegate. This adds a third phase to PropagateDevicePass that elides `_h2d_copy(_d2h_copy(x))` when the tensor before the D2H already lives on the device the H2D copies back to. The device comparison is the correctness guard: a genuine cross-device move (e.g. cuda:0 -> host -> cuda:1) has differing devices and is preserved. A D2H that still feeds other (host) consumers keeps those users and is removed only once it becomes dead. The phase runs only when enable_non_cpu_memory_planning is set (i.e. when copies were actually inserted). This closes the "redundant copies between back-to-back delegates on the same device" limitation from the device-aware export work. Differential Revision: D111992997
Conarnar
force-pushed
the
export-D111992997
branch
from
July 15, 2026 01:12
ec7ba46 to
4d2005a
Compare
Gasoonjia
approved these changes
Jul 15, 2026
Conarnar
added a commit
to Conarnar/executorch
that referenced
this pull request
Jul 15, 2026
) Summary: PropagateDevicePass inserts H2D/D2H copies per delegate boundary independently (an H2D before every device delegate input, a D2H after every device delegate output). When two delegates run consecutively on the *same* device, this leaves a redundant device -> host -> device round-trip between them: the first delegate's output is copied to host and immediately copied back to the same device for the second delegate. This adds a third phase to PropagateDevicePass that elides `_h2d_copy(_d2h_copy(x))` when the tensor before the D2H already lives on the device the H2D copies back to. The device comparison is the correctness guard: a genuine cross-device move (e.g. cuda:0 -> host -> cuda:1) has differing devices and is preserved. A D2H that still feeds other (host) consumers keeps those users and is removed only once it becomes dead. The phase runs only when enable_non_cpu_memory_planning is set (i.e. when copies were actually inserted). This closes the "redundant copies between back-to-back delegates on the same device" limitation from the device-aware export work. Reviewed By: Gasoonjia Differential Revision: D111992997
Conarnar
force-pushed
the
export-D111992997
branch
from
July 15, 2026 17:07
4d2005a to
27d4f20
Compare
Conarnar
added a commit
to Conarnar/executorch
that referenced
this pull request
Jul 15, 2026
) Summary: PropagateDevicePass inserts H2D/D2H copies per delegate boundary independently (an H2D before every device delegate input, a D2H after every device delegate output). When two delegates run consecutively on the *same* device, this leaves a redundant device -> host -> device round-trip between them: the first delegate's output is copied to host and immediately copied back to the same device for the second delegate. This adds a third phase to PropagateDevicePass that elides `_h2d_copy(_d2h_copy(x))` when the tensor before the D2H already lives on the device the H2D copies back to. The device comparison is the correctness guard: a genuine cross-device move (e.g. cuda:0 -> host -> cuda:1) has differing devices and is preserved. A D2H that still feeds other (host) consumers keeps those users and is removed only once it becomes dead. The phase runs only when enable_non_cpu_memory_planning is set (i.e. when copies were actually inserted). This closes the "redundant copies between back-to-back delegates on the same device" limitation from the device-aware export work. Reviewed By: Gasoonjia Differential Revision: D111992997
Conarnar
force-pushed
the
export-D111992997
branch
from
July 15, 2026 17:07
27d4f20 to
6510082
Compare
Conarnar
added a commit
to Conarnar/executorch
that referenced
this pull request
Jul 15, 2026
) Summary: PropagateDevicePass inserts H2D/D2H copies per delegate boundary independently (an H2D before every device delegate input, a D2H after every device delegate output). When two delegates run consecutively on the *same* device, this leaves a redundant device -> host -> device round-trip between them: the first delegate's output is copied to host and immediately copied back to the same device for the second delegate. This adds a third phase to PropagateDevicePass that elides `_h2d_copy(_d2h_copy(x))` when the tensor before the D2H already lives on the device the H2D copies back to. The device comparison is the correctness guard: a genuine cross-device move (e.g. cuda:0 -> host -> cuda:1) has differing devices and is preserved. A D2H that still feeds other (host) consumers keeps those users and is removed only once it becomes dead. The phase runs only when enable_non_cpu_memory_planning is set (i.e. when copies were actually inserted). This closes the "redundant copies between back-to-back delegates on the same device" limitation from the device-aware export work. Reviewed By: Gasoonjia Differential Revision: D111992997
Conarnar
force-pushed
the
export-D111992997
branch
from
July 15, 2026 17:43
6510082 to
8422b8e
Compare
) Summary: PropagateDevicePass inserts H2D/D2H copies per delegate boundary independently (an H2D before every device delegate input, a D2H after every device delegate output). When two delegates run consecutively on the *same* device, this leaves a redundant device -> host -> device round-trip between them: the first delegate's output is copied to host and immediately copied back to the same device for the second delegate. This adds a third phase to PropagateDevicePass that elides `_h2d_copy(_d2h_copy(x))` when the tensor before the D2H already lives on the device the H2D copies back to. The device comparison is the correctness guard: a genuine cross-device move (e.g. cuda:0 -> host -> cuda:1) has differing devices and is preserved. A D2H that still feeds other (host) consumers keeps those users and is removed only once it becomes dead. The phase runs only when enable_non_cpu_memory_planning is set (i.e. when copies were actually inserted). This closes the "redundant copies between back-to-back delegates on the same device" limitation from the device-aware export work. Reviewed By: Gasoonjia Differential Revision: D111992997
Conarnar
force-pushed
the
export-D111992997
branch
from
July 15, 2026 17:44
8422b8e to
44fea49
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
PropagateDevicePass inserts H2D/D2H copies per delegate boundary
independently (an H2D before every device delegate input, a D2H after every
device delegate output). When two delegates run consecutively on the same
device, this leaves a redundant device -> host -> device round-trip between
them: the first delegate's output is copied to host and immediately copied
back to the same device for the second delegate.
This adds a third phase to PropagateDevicePass that elides
_h2d_copy(_d2h_copy(x))when the tensor before the D2H already lives on thedevice the H2D copies back to. The device comparison is the correctness guard:
a genuine cross-device move (e.g. cuda:0 -> host -> cuda:1) has differing
devices and is preserved. A D2H that still feeds other (host) consumers keeps
those users and is removed only once it becomes dead. The phase runs only when
enable_non_cpu_memory_planning is set (i.e. when copies were actually
inserted).
This closes the "redundant copies between back-to-back delegates on the same
device" limitation from the device-aware export work.
Reviewed By: Gasoonjia
Differential Revision: D111992997