CUDA delegate: a device-resident input is copied with a host memcpy, crashing the no-copy export mode

[shoumikhin]

Problem

If you export a model so the caller supplies GPU tensors, running it crashes the process.

ExecuTorch can export a CUDA model two ways. By default it inserts copies at the delegate boundary, so
you pass ordinary CPU tensors and the runtime moves the data. You can also turn those copies off, which
tags the method's inputs as device memory and makes the caller responsible for passing GPU tensors:

program = lowered.to_executorch(
    ExecutorchBackendConfig(
        propagate_device_config=PropagateDeviceConfig(
            skip_h2d_for_method_inputs=True,
            skip_d2h_for_method_outputs=True,
        ),
        enable_non_cpu_memory_planning=True,
    )
)

That is the documented way to avoid a redundant copy when your data is already on the GPU. Running such a
program segfaults.

Why

Setting an input goes through copy_tensor_data, which is an unconditional host copy
(runtime/core/exec_aten/util/tensor_util_portable.cpp):

std::memcpy(
    t_dst.mutable_data_ptr(), t_src.const_data_ptr(), t_src.nbytes());

Called from Method::set_input (runtime/executor/method.cpp:1247).

In this mode the destination is device memory, which is not mapped into the host address space, so the
copy faults. Stack:

Module::execute
  Method::set_input
    copy_tensor_data
      memcpy          <- faults on a device pointer

The no-copy path appears to assume set_input will skip the copy for a device-resident input. It does
not.

Reproducing

import torch
from executorch.backends.cuda.cuda_partitioner import CudaPartitioner
from executorch.exir import ExecutorchBackendConfig, to_edge_transform_and_lower
from executorch.exir.passes.propagate_device_config import PropagateDeviceConfig
from executorch.extension.pybindings import portable_lib


class Net(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.register_buffer("scale", torch.full((64,), 1.5))

    def forward(self, x):
        return torch.relu(x * self.scale) * 2.0


model = Net().eval().cuda()
example = (torch.randn(4, 64, device="cuda"),)
lowered = to_edge_transform_and_lower(
    torch.export.export(model, example), partitioner=[CudaPartitioner([])]
)
program = lowered.to_executorch(
    ExecutorchBackendConfig(
        propagate_device_config=PropagateDeviceConfig(
            skip_h2d_for_method_inputs=True,
            skip_d2h_for_method_outputs=True,
        ),
        enable_non_cpu_memory_planning=True,
    )
)
open("model.pte", "wb").write(program.buffer)

module = portable_lib._load_for_executorch("model.pte")
module.forward([torch.randn(4, 64, device="cuda")])  # segfault

The same crash happens from a C++ application that builds its input over a cudaMalloc pointer, so this
is in the runtime rather than the Python bindings.

Observed on x86_64 with CUDA 13.0 and on an aarch64 GPU device with CUDA 13.0. The default export mode,
where the copies are inserted, is unaffected.

Suggested fix

copy_tensor_data should not assume both pointers are host memory. Two options:

Dispatch on where the memory lives. Ask the allocator or the tensor which device backs the pointer and
use a device copy when either side is not host memory. This keeps a single entry point for callers.

Skip the copy when the destination is already device memory. In the no-copy mode the caller has
deliberately placed the data where the delegate wants it, so the correct action at the boundary is to
adopt the pointer rather than copy it. That also removes the transfer the mode exists to avoid, so it is
probably the intended behaviour.

Either way the mode currently has no working runtime path, so a test that exports with these flags and
runs with a device pointer would keep it working once fixed.

Note

An earlier version of this report also described a constants-loading fault, based on this log line:

weights_blob '<hash>' not found or update fn is null

That was wrong and I have retracted it. The hash is sha256(""), an empty blob, so the lookup misses
because a partition has no constants of its own, and the message is expected at info level. A model that
emits the line runs correctly. My earlier failures were a delegate that had not been loaded and, in one
case, an input shape that did not match the exported program.

[shoumikhin]

Correction to this report

Further testing shows the first of the two issues above was misdiagnosed, and I would rather correct it
than leave a misleading report standing.

The log line is expected, not a fault

I reported that the backend logs

weights_blob '<hash>' not found or update fn is null

and concluded that constants were never loaded, so the kernel read unset memory. The message is real,
but it is not the cause, and it is not an error.

The hash in the message is sha256(""):

sha256 of empty input:  e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
hash in the message:    e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

An empty blob. The lookup misses because there is nothing to look up, which is the correct state for a
partition that has no constants of its own. Logging it at info level is consistent with that.

A model runs with this line present

The decisive evidence is a program that emits the same line and produces correct output. One partitioned
across both the TensorRT and CUDA delegates, run from an installed wheel:

registered backends: ['CudaBackend', 'OpenvinoBackend', 'TensorRTBackend', 'XnnpackBackend']
[cuda_backend] weights_blob 'e3b0c44...' not found or update fn is null
RAN, output (64, 64), all finite, mean 0.734548

So the message does not prevent execution and does not indicate corrupted constants.

What my earlier failures actually were

Two separate causes, neither the one I named:

A backend that was not registered. The failure I saw as error: 0x20 was the delegate library not
being loaded at all, so the program's TensorRT partition had no backend to run on. Loading it fixed that,
and the log line above was present both before and after, which is what showed it was not the variable.

An input shape mismatch. The failure I saw as error: 0x10 was mine: the program was exported for
one input shape and I passed another. The runtime said so plainly:

Attempted to resize a static tensor. Expected shape (64, 64), but received (4, 64).

What still stands

The second issue in the original report is unaffected: a program exported to skip the host-to-device
copies expects device-resident inputs, and Method::set_input performs a host copy on that pointer,
which segfaults. That reproduces from Python and from a C++ application, and remains a real defect.

What I got wrong methodologically

I treated an info-level log line adjacent to a failure as the explanation for it, without testing whether
the line appeared on a successful run. It does. A message that accompanies both success and failure
explains neither, and I should have established that before filing.