Skip to content

torch_musa Release 2.11.0.post1

Latest

Choose a tag to compare

@lijing-mt lijing-mt released this 17 Aug 07:49
· 6 commits to main since this release
246d43e

Hi all, torch_musa 2.11.0.post1 is now available. This is the first release based on PyTorch 2.11, following torch_musa 2.9.1.post1. It is compatible with MUSA SDK 5.1 and 5.2, provides full LibTorch Stable ABI coverage for PyTorch 2.9 through 2.11, upgrades the distributed infrastructure, and includes several bug fixes and performance optimizations.

The details are as follows:

  • Upgraded pyahocorasick to 2.3.0 to resolve CUDA-porting issues caused by earlier versions.
  • Added full support for the LibTorch Stable ABI, including common and CUDA APIs for PyTorch 2.9 through 2.11.
  • Updated ProcessGroupMCCL to v2.11.0.
  • Added noinline to device functions to reduce build times.
  • Added support for scalar-tensor inputs to index_select.
  • Added support for complex dtypes in cumulative operations.
  • Removed libmusa.so from DT_NEEDED entries to support lazy initialization.
  • Fixed an inter-node all_to_all hang when one side uses zero-size send/receive buffers.

Known && blocked issues

  • MUPTI upgrade: requires enabling Kineto's finalize API and manually setting TEARDOWN_MUPTI=1. Testing showed that kernel names may be reported out of order, so this upgrade is not included in the current release.

  • GIL in PyTorch 2.11: intrusive_ptr acquires the GIL when references to Python-wrapped Tensor or Storage objects transition between reference counts 1 and 2. This preserves object-lifetime correctness but turns inexpensive reference updates into synchronization points, which may delay kernel submission and reduce throughput.

    When DataLoader is configured with num_workers > 1 and pin_memory=True, the pt_data_pin thread may intensify GIL contention. Reverting this behavior is unsafe. Possible mitigations include setting num_workers=0, reducing prefetching, or using main-thread pinning.

Please feel free to contact us with any issues or questions.