NumPy & SciPy for GPU

An open collection of implementation tips, tricks and resources for training large language models

An open collection of methodologies to help with successful training of large language models.

Distributed and decentralized training framework for PyTorch over graph

Best practices & guides on how to write distributed pytorch training code

Federated Learning Utilities and Tools for Experimentation

Python Distributed Non Negative Matrix Factorization with custom clustering

Experiments with low level communication patterns that are useful for distributed training.

Nvidia NCCL2 Python bindings using ctypes and numba.

Tool to run rccl-tests/nccl-tests based on from an application and gather performance.

Distributed deep learning framework based on pytorch/numba/nccl and zeromq.