[TRTLLM-14475][chore] Self-sufficient transfer-agent dlopen and drop onnx/modelopt deps#16829
[TRTLLM-14475][chore] Self-sufficient transfer-agent dlopen and drop onnx/modelopt deps#16829Wanli-Jiang wants to merge 1 commit into
Conversation
…onnx/modelopt deps
Trailing cleanup after the TensorRT backend removal, in two parts.
1) Make the KV-cache transfer-agent wrapper dlopen self-sufficient.
The wrapper libraries (libtensorrt_llm_{nixl,ucx,mooncake}_wrapper.so)
carry no DT_NEEDED on libtensorrt_llm.so (circular) and resolve its
symbols from the process global scope. In the wheel they only worked
because tensorrt_llm/_common.py preloaded libtensorrt_llm.so with ctypes
RTLD_GLOBAL - a stopgap added when the removed TensorRT plugin's
RTLD_GLOBAL load stopped providing this as a side effect. New
executor::kv_cache::promoteHostLibraryToGlobalScope() (dladdr-self +
RTLD_NOW|RTLD_GLOBAL|RTLD_NOLOAD, call_once, no-op when statically
linked) is called at both wrapper-load sites (DynLibLoader::getHandle and
the UCX dllOpen), and the _common.py preload is removed.
2) Drop the onnx and nvidia-modelopt dependencies.
- onnx / onnx_graphsurgeon: TensorRT ONNX-engine-build graph tools with
zero consumers repo-wide. Legacy relics of the removed engine path.
- nvidia-modelopt[torch]: removed as a dependency and its AutoDeploy
package import sites stripped (all four were try/except-guarded). The
main PyTorch backend never imported the package. PRESERVED (package
free): loading pre-quantized FP8/NVFP4 checkpoints via
models/quant_config_reader.py + _compat.py + the local
modelopt_fp4_scale_to_cutlass_fp4_scale, which read the modelopt
*checkpoint format* and never imported the package. DISABLED: loading
graphs with live modelopt fake-quant ops (is_quantized_graph=True) and
the modelopt export context. Guarded test helpers degrade gracefully,
so test collection is unaffected; examples/auto_deploy/
build_and_run_flux.py hard-imports the package and will not run without
it (out of scope here).
Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
|
/bot run --disable-fail-fast |
|
PR_Github #61511 [ run ] triggered by Bot. Commit: |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (9)
💤 Files with no reviewable changes (4)
WalkthroughThe change moves host-library symbol promotion from Python initialization into C++, invokes it before dynamic wrapper loading, removes ModelOpt auto-deploy integration, and removes ONNX and ModelOpt requirements. ChangesGlobal symbol promotion
ModelOpt integration removal
ONNX dependency removal
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant CacheTransceiver
participant DynLibLoader
participant HostLibrary
participant UCXWrapper
CacheTransceiver->>HostLibrary: promoteHostLibraryToGlobalScope()
CacheTransceiver->>UCXWrapper: open UCX wrapper
DynLibLoader->>HostLibrary: promote before handle lookup
DynLibLoader->>UCXWrapper: resolve wrapper symbols
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
| modelopt_quantize_op = None | ||
| modelopt_dynamic_block_quantize_op = None | ||
| # modelopt quantization support has been removed; these ops are no longer available. | ||
| modelopt_quantize_op = None |
There was a problem hiding this comment.
@nv-guomingz can you help review this part? Are we still using modelopt.torch.quantization.tensor_quant in torch path?
| # table, while Python extension modules and their dependencies load with | ||
| # RTLD_LOCAL. This promotion was previously a side effect of loading the | ||
| # TensorRT plugin library with RTLD_GLOBAL. | ||
| if platform.system() != "Windows": |
There was a problem hiding this comment.
@Wanli-Jiang can you explain why we switch to implement this logic in C++? Is it that the python version no longer works?
There was a problem hiding this comment.
The Python preload wasn't broken — but it was a hack: on every import tensorrt_llm it force-loads the whole libtensorrt_llm.so with RTLD_GLOBAL via ctypes, purely to satisfy a C++ dlopen requirement (the nixl/ucx/mooncake wrappers carry no DT_NEEDED on it and need its symbols in the global scope). It also only kicks in when you enter through the Python import path.
The C++ version is simpler and cleaner: the library promotes its own symbols right at the wrapper-load site — promoteHostLibraryToGlobalScope() (RTLD_NOLOAD | RTLD_GLOBAL, call_once, no-op when statically linked). No Python involvement, works for any caller (C++ embeddings, executor, gtests), and the requirement lives where it belongs.
the Python preload was a hack (a whole-library RTLD_GLOBAL force-load on every import, just to satisfy a C++ dlopen need), and the C++ version is the simpler, cleaner home for it.
|
PR_Github #61511 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #61573 [ run ] triggered by Bot. Commit: |
|
PR_Github #61573 [ run ] completed with state
|
6 participants
Trailing cleanup after the TensorRT backend removal, in two parts.
Make the KV-cache transfer-agent wrapper dlopen self-sufficient.
The wrapper libraries (libtensorrt_llm_{nixl,ucx,mooncake}_wrapper.so)
carry no DT_NEEDED on libtensorrt_llm.so (circular) and resolve its
symbols from the process global scope. In the wheel they only worked
because tensorrt_llm/_common.py preloaded libtensorrt_llm.so with ctypes
RTLD_GLOBAL - a stopgap added when the removed TensorRT plugin's
RTLD_GLOBAL load stopped providing this as a side effect. New
executor::kv_cache::promoteHostLibraryToGlobalScope() (dladdr-self +
RTLD_NOW|RTLD_GLOBAL|RTLD_NOLOAD, call_once, no-op when statically
linked) is called at both wrapper-load sites (DynLibLoader::getHandle and
the UCX dllOpen), and the _common.py preload is removed.
Drop the onnx and nvidia-modelopt dependencies.
Dev Engineer Review
libtensorrt_llm.soglobally at KV-cache wrapper load sites, removing the_common.pypreload.QA Engineer Review
No test changes.
Description
Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.