Repository navigation
Hi all, torch_musa 2.11.0.post1 is now available. This is the first release based on PyTorch 2.11, following torch_musa 2.9.1.post1. It is compatible with MUSA SDK 5.1 and 5.2, provides full LibTorch Stable ABI coverage for PyTorch 2.9 through 2.11, upgrades the distributed infrastructure, and includes several bug fixes and performance optimizations.
The details are as follows:
- Upgraded
pyahocorasickto 2.3.0 to resolve CUDA-porting issues caused by earlier versions. - Added full support for the LibTorch Stable ABI, including common and CUDA APIs for PyTorch 2.9 through 2.11.
- Updated ProcessGroupMCCL to v2.11.0.
- Added
noinlineto device functions to reduce build times. - Added support for scalar-tensor inputs to
index_select. - Added support for complex dtypes in cumulative operations.
- Removed
libmusa.sofromDT_NEEDEDentries to support lazy initialization. - Fixed an inter-node
all_to_allhang when one side uses zero-size send/receive buffers.
Known && blocked issues
-
MUPTI upgrade: requires enabling Kineto's finalize API and manually setting
TEARDOWN_MUPTI=1. Testing showed that kernel names may be reported out of order, so this upgrade is not included in the current release. -
GIL in PyTorch 2.11:
intrusive_ptracquires the GIL when references to Python-wrappedTensororStorageobjects transition between reference counts 1 and 2. This preserves object-lifetime correctness but turns inexpensive reference updates into synchronization points, which may delay kernel submission and reduce throughput.When
DataLoaderis configured withnum_workers > 1andpin_memory=True, thept_data_pinthread may intensify GIL contention. Reverting this behavior is unsafe. Possible mitigations include settingnum_workers=0, reducing prefetching, or using main-thread pinning.
Please feel free to contact us with any issues or questions.