TensorRT 10.14+ multi-GPU build regression with Cask-backed ops

## Summary

Starting with TensorRT **10.14.1**, `IBuilder::buildSerializedNetwork()` can fail when building a second engine after switching to a different CUDA device in the same process.

The failure is specific to Cask-backed operators; the minimal repro uses `MaxPool`:

```text
Error Code: 9: Skipping tactic ... Cask Pooling Runner Execute Failure
Error Code: 10: Internal Error (Could not find any implementation for node MaxPool_0.
In computeCosts at .../optimizer.cpp:4265)
```

The same-device sequence succeeds. Building either device alone succeeds. Building each engine in a separate process succeeds.

The regression is present in **10.14.1.48.post1 through 10.16.1.11** and is not present in **10.13.3.9** or **11.2.1.2**.

## Environment

* TensorRT: `tensorrt-cu13`

  * 10.14.1.48.post1, 10.15.1.29, 10.16.0.72, 10.16.1.11: broken
  * 10.13.2.6, 10.13.3.9: OK
  * 11.2.1.2: OK
* GPUs:

  * Quadro RTX 4000, SM 7.5
  * NVIDIA RTX 4000 SFF Ada Generation, SM 8.9
* Driver: 580.142
* OS: AlmaLinux 9.8, kernel 6.12.103-1.el9.x86_64
* Python: 3.11

## Minimal reproducer

Requires two CUDA devices.

```
#!/usr/bin/env python3
import ctypes
import sys

import onnx
import tensorrt as trt
from onnx import TensorProto, helper


def make_model() -> bytes:
    x = helper.make_tensor_value_info("x", TensorProto.FLOAT, [1, 1, 8, 8])
    node = helper.make_node(
        "MaxPool", ["x"], ["y"], name="MaxPool_0",
        kernel_shape=[3, 3], pads=[1, 1, 1, 1], strides=[1, 1],
    )
    y = helper.make_tensor_value_info("y", TensorProto.FLOAT, [1, 1, 8, 8])
    graph = helper.make_graph([node], "tiny_maxpool", [x], [y])
    model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", 20)])
    model.ir_version = 9
    onnx.checker.check_model(model)
    return model.SerializeToString()


def build(model: bytes, cudart: ctypes.CDLL, device: int, label: str) -> bool:
    assert cudart.cudaSetDevice(device) == 0
    print(f"======={label} on {device=}", file=sys.stderr)
    logger = trt.Logger(trt.Logger.INFO)
    builder = trt.Builder(logger)
    config = builder.create_builder_config()
    config.set_memory_pool_limit(
        trt.MemoryPoolType.WORKSPACE, 1 << 30
    )

    network = builder.create_network(
        1 << int(trt.NetworkDefinitionCreationFlag.STRONGLY_TYPED)
    )
    parser = trt.OnnxParser(network, logger)
    assert parser.parse(model, path="/tmp")
    engine = builder.build_serialized_network(network, config)
    print("=======Failed" if engine is None else "Succeed", file=sys.stderr)


def main():
    print(f"======={trt.__version__=}", file=sys.stderr)
    cudart = ctypes.CDLL("libcudart.so")
    count = ctypes.c_int()
    cudart.cudaGetDeviceCount(ctypes.byref(count))
    print(f"======={count.value=}", file=sys.stderr)
    if count.value < 2:
        return 1

    model = make_model()

    build(model, cudart, 0, "same device 1/2")
    build(model, cudart, 0, "same device 2/2") # succeeds

    build(model, cudart, 0, "different device 1/2")
    build(model, cudart, 1, "different device 2/2") # fails after device switch

    return 0

if __name__ == "__main__":
    main()

```
**Full traceback / output** (TensorRT 10.16.1.11):
```
=======trt.__version__='10.16.1.11'
=======count.value=2
=======same device 1/2 on device=0
[09/05/2026-19:32:08] [TRT] [I] [MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 47, GPU 99 (MiB)
[09/05/2026-19:32:08] [TRT] [I] ----------------------------------------------------------------
[09/05/2026-19:32:08] [TRT] [I] Input filename:   /tmp
[09/05/2026-19:32:08] [TRT] [I] ONNX IR version:  0.0.9
[09/05/2026-19:32:08] [TRT] [I] Opset version:    20
[09/05/2026-19:32:08] [TRT] [I] Producer name:    
[09/05/2026-19:32:08] [TRT] [I] Producer version: 
[09/05/2026-19:32:08] [TRT] [I] Domain:           
[09/05/2026-19:32:08] [TRT] [I] Model version:    0
[09/05/2026-19:32:08] [TRT] [I] Doc string:       
[09/05/2026-19:32:08] [TRT] [I] ----------------------------------------------------------------
[09/05/2026-19:32:08] [TRT] [I] BuilderFlag::kTF32 is set but hardware does not support TF32. Disabling TF32.
[09/05/2026-19:32:08] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +91, GPU +2, now: CPU 335, GPU 101 (MiB)
[09/05/2026-19:32:08] [TRT] [I] BuilderFlag::kTF32 is set but hardware does not support TF32. Disabling TF32.
[09/05/2026-19:32:08] [TRT] [I] Local timing cache in use. Profiling results in this builder pass will not be stored.
[09/05/2026-19:32:09] [TRT] [I] Detected 1 inputs and 1 output network tensors.
[09/05/2026-19:32:09] [TRT] [I] Total Host Persistent Memory: 4160 bytes
[09/05/2026-19:32:09] [TRT] [I] Total Device Persistent Memory: 0 bytes
[09/05/2026-19:32:09] [TRT] [I] Max Scratch Memory: 0 bytes
[09/05/2026-19:32:09] [TRT] [I] Total Activation Memory: 0 bytes
[09/05/2026-19:32:09] [TRT] [I] Total Weights Memory: 0 bytes
[09/05/2026-19:32:09] [TRT] [I] Engine generation completed in 0.412562 seconds.
[09/05/2026-19:32:09] [TRT] [I] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 0 MiB, GPU 1 MiB
Succeed
=======same device 2/2 on device=0
[09/05/2026-19:32:09] [TRT] [I] ----------------------------------------------------------------
[09/05/2026-19:32:09] [TRT] [I] Input filename:   /tmp
[09/05/2026-19:32:09] [TRT] [I] ONNX IR version:  0.0.9
[09/05/2026-19:32:09] [TRT] [I] Opset version:    20
[09/05/2026-19:32:09] [TRT] [I] Producer name:    
[09/05/2026-19:32:09] [TRT] [I] Producer version: 
[09/05/2026-19:32:09] [TRT] [I] Domain:           
[09/05/2026-19:32:09] [TRT] [I] Model version:    0
[09/05/2026-19:32:09] [TRT] [I] Doc string:       
[09/05/2026-19:32:09] [TRT] [I] ----------------------------------------------------------------
[09/05/2026-19:32:09] [TRT] [I] BuilderFlag::kTF32 is set but hardware does not support TF32. Disabling TF32.
[09/05/2026-19:32:09] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU -89, GPU +2, now: CPU 383, GPU 103 (MiB)
[09/05/2026-19:32:09] [TRT] [I] BuilderFlag::kTF32 is set but hardware does not support TF32. Disabling TF32.
[09/05/2026-19:32:09] [TRT] [I] Local timing cache in use. Profiling results in this builder pass will not be stored.
[09/05/2026-19:32:09] [TRT] [I] Detected 1 inputs and 1 output network tensors.
[09/05/2026-19:32:09] [TRT] [I] Total Host Persistent Memory: 4160 bytes
[09/05/2026-19:32:09] [TRT] [I] Total Device Persistent Memory: 0 bytes
[09/05/2026-19:32:09] [TRT] [I] Max Scratch Memory: 0 bytes
[09/05/2026-19:32:09] [TRT] [I] Total Activation Memory: 0 bytes
[09/05/2026-19:32:09] [TRT] [I] Total Weights Memory: 0 bytes
[09/05/2026-19:32:09] [TRT] [I] Engine generation completed in 0.171036 seconds.
[09/05/2026-19:32:09] [TRT] [I] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 0 MiB, GPU 1 MiB
Succeed
=======different device 1/2 on device=0
[09/05/2026-19:32:09] [TRT] [I] ----------------------------------------------------------------
[09/05/2026-19:32:09] [TRT] [I] Input filename:   /tmp
[09/05/2026-19:32:09] [TRT] [I] ONNX IR version:  0.0.9
[09/05/2026-19:32:09] [TRT] [I] Opset version:    20
[09/05/2026-19:32:09] [TRT] [I] Producer name:    
[09/05/2026-19:32:09] [TRT] [I] Producer version: 
[09/05/2026-19:32:09] [TRT] [I] Domain:           
[09/05/2026-19:32:09] [TRT] [I] Model version:    0
[09/05/2026-19:32:09] [TRT] [I] Doc string:       
[09/05/2026-19:32:09] [TRT] [I] ----------------------------------------------------------------
[09/05/2026-19:32:09] [TRT] [I] BuilderFlag::kTF32 is set but hardware does not support TF32. Disabling TF32.
[09/05/2026-19:32:09] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU -89, GPU +2, now: CPU 383, GPU 103 (MiB)
[09/05/2026-19:32:09] [TRT] [I] BuilderFlag::kTF32 is set but hardware does not support TF32. Disabling TF32.
[09/05/2026-19:32:09] [TRT] [I] Local timing cache in use. Profiling results in this builder pass will not be stored.
[09/05/2026-19:32:09] [TRT] [I] Detected 1 inputs and 1 output network tensors.
[09/05/2026-19:32:09] [TRT] [I] Total Host Persistent Memory: 4160 bytes
[09/05/2026-19:32:09] [TRT] [I] Total Device Persistent Memory: 0 bytes
[09/05/2026-19:32:09] [TRT] [I] Max Scratch Memory: 0 bytes
[09/05/2026-19:32:09] [TRT] [I] Total Activation Memory: 0 bytes
[09/05/2026-19:32:09] [TRT] [I] Total Weights Memory: 0 bytes
[09/05/2026-19:32:09] [TRT] [I] Engine generation completed in 0.174508 seconds.
[09/05/2026-19:32:09] [TRT] [I] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 0 MiB, GPU 1 MiB
Succeed
=======different device 2/2 on device=1
[09/05/2026-19:32:09] [TRT] [I] ----------------------------------------------------------------
[09/05/2026-19:32:09] [TRT] [I] Input filename:   /tmp
[09/05/2026-19:32:09] [TRT] [I] ONNX IR version:  0.0.9
[09/05/2026-19:32:09] [TRT] [I] Opset version:    20
[09/05/2026-19:32:09] [TRT] [I] Producer name:    
[09/05/2026-19:32:09] [TRT] [I] Producer version: 
[09/05/2026-19:32:09] [TRT] [I] Domain:           
[09/05/2026-19:32:09] [TRT] [I] Model version:    0
[09/05/2026-19:32:09] [TRT] [I] Doc string:       
[09/05/2026-19:32:09] [TRT] [I] ----------------------------------------------------------------
[09/05/2026-19:32:09] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU -58, GPU +0, now: CPU 413, GPU 169 (MiB)
[09/05/2026-19:32:09] [TRT] [I] Local timing cache in use. Profiling results in this builder pass will not be stored.
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xca723f3a6d5c572a due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x1b13dea2697d92fc due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x5f085e81bccfdfd4 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xadf1751d9d52bc14 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x4805490eb56c6142 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xd4b9db5d6f22d997 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x4fc018d7ee59fb45 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x39a96deeea00c898 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x5154ad0527677a6f due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x80cf0fe785959355 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xe8f436d3bcc38289 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x31efd02f5d31b9ba due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x6abffa4236113b26 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x5faf4a0a8a5670ed due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x62f9478381204a04 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xbc27f0a63be7eb30 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xb59f9cfb90407c92 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x96dbfe2b0fabc076 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xc5cd6987d3ba33c8 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x9e9d43eab89ab154 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x9e04eb8087eb1de8 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x6cfdc01ca6767e28 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x1022263884083147 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x7213e6ffae85b15b due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x388b14a64cc134ea due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xbbe2a17f60d27137 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x6193a97a80c05353 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xf987a1a777e5f6d6 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xc17f7d05d2cb7b56 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x8cfd5f7bc7e409c2 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xbf50b5fd61a339a9 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xcd8bd446648b42ea due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x18649bf933394065 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x664b53018051029a due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xef31670ae7f6188e due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xf660830175fd03a3 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xc6ba2cdc89fee151 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x1cd68f7b324808fb due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x0b7e8a3b56789516 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x2bc9717f957e700e due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x618e02d8db64989d due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xb33a917a9f2c32a9 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x78dfe6d3496f83b0 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xb716083cd692488b due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xdfebc8d71bf5fa7c due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xe74c45c0759b786b due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x8382d5c464539e87 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xaec8628e8180bced due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xd76bac5638836f8a due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xfa211b1cdd504de0 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xe9d01a2a900075cb due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x789b2859f2e03e79 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0xbd3963b8ccd084c6 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x2c7251cbae30cf74 due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] Error Code: 9: Skipping tactic 0x22fb1bb4a70e340d due to exception Cask Pooling Runner Execute Failure In execute at /_src/runtime/gpu/cask/poolingRunner.cpp:324
[09/05/2026-19:32:09] [TRT] [E] IBuilder::buildSerializedNetwork: Error Code 10: Internal Error (Could not find any implementation for node MaxPool_0. In computeCosts at /_src/optimizer/common/tactic/optimizer.cpp:4265)
=======Failed
```

## Reproduction matrix

| TensorRT             | Result   |
| -------------------- | -------- |
| 10.12.0.36 (`cu12`)  | OK       |
| 10.13.2.6            | OK       |
| 10.13.3.9            | OK       |
| **10.14.1.48.post1** | **FAIL** |
| 10.15.1.29           | FAIL     |
| 10.16.0.72           | FAIL     |
| 10.16.1.11           | FAIL     |
| 11.2.1.2             | OK       |

## Isolation

* Reproduces with FP32; not FP16-specific.
* Reproduces with both 3×3 and 5×5 MaxPool.
* `Relu` with the same tensor shape does not reproduce.
* Failure follows the **device switch**, independent of device order.
* `cudaMemGetInfo()` shows ~20 GB free on the second GPU before the build.
* Workspace limits of 1 GB and 4 GB produce the same result.
* `cudaDeviceReset()` does not resolve it.
* `IBuilderConfig.reset()` does not resolve it.
* Custom `profile_stream` does not resolve it.
* Restricting/disabling tactic sources does not resolve it.
* Dedicated OS threads do not resolve it.
* Explicit CUDA driver contexts (`cuCtxCreate`/`cuCtxSetCurrent`) do not resolve it.
* Building each engine in a separate OS process succeeds.

A real production model also shows the same pattern with `Cask convolution execution` tactic failures, suggesting the issue may affect multiple Cask-backed operators.

## Expected behavior

`buildSerializedNetwork()` should succeed when building an engine for a different CUDA device after a previous engine was built in the same process.

## Suspected cause

The behavior is consistent with process-global Cask state being initialized or cached for the first device and not correctly refreshed when the active device changes, particularly across SM 7.5 → SM 8.9.

This is a hypothesis; the minimal repro should allow the issue to be investigated independently.

## Workaround

Build each GPU's engine in a separate process.

## Regression range

**10.13.3.9 → 10.14.1.48.post1**

Fixed in **11.2.1.2**.

Navigation Menu

Uh oh!

TensorRT 10.14+ multi-GPU build regression with Cask-backed ops #4845

Summary

Environment

Minimal reproducer

Reproduction matrix

Isolation

Expected behavior

Suspected cause

Workaround

Regression range

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

TensorRT	Result
10.12.0.36 (`cu12`)	OK
10.13.2.6	OK
10.13.3.9	OK
10.14.1.48.post1	FAIL
10.15.1.29	FAIL
10.16.0.72	FAIL
10.16.1.11	FAIL
11.2.1.2	OK

Uh oh!

TensorRT 10.14+ multi-GPU build regression with Cask-backed ops #4845

Description

Summary

Environment

Minimal reproducer

Reproduction matrix

Isolation

Expected behavior

Suspected cause

Workaround

Regression range

Activity

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions