Asynchronous Pipeline Parallel

This is the official implementation of the ICML 2025 paper Nesterov Method for Asynchronous Pipeline Parallel Optimization.

Instructions

First install requirements.txt and simply run the bash script run.bash. This script assumes an instance with at least 8 GPUs and runs our method with 8 pipeline stages. Tested on PyTorch 2.5.1, CUDA 12.6, and Python 3.12.

Credits

• PipeDream
• PiPPy
• NanoGPT

License

Copyright © Pluralis Research. All rights reserved.

This project is licensed under the MIT License. See the LICENSE file for details.

Citation

@article{ajanthan2025asyncpp,
  title={Nesterov Method for Asynchronous Pipeline Parallel Optimization},
  author={Ajanthan, Thalaiyasingam and Ramasinghe, Sameera and Zuo, Yan and Avraham, Gil and Long, Alexander},
  journal={ICML},
  year={2025}
}