Experiments with low level communication patterns that are useful for distributed training.

Tool to run rccl-tests/nccl-tests based on from an application and gather performance.

Nvidia NCCL2 Python bindings using ctypes and numba.

An open collection of implementation tips, tricks and resources for training large language models

Distributed deep learning framework based on pytorch/numba/nccl and zeromq.

Python Distributed Non Negative Matrix Factorization with custom clustering

Federated Learning Utilities and Tools for Experimentation

An open collection of methodologies to help with successful training of large language models.

Distributed and decentralized training framework for PyTorch over graph

Best practices & guides on how to write distributed pytorch training code

NumPy & SciPy for GPU