Pinned Loading

1. zo2 zo2 Public

   ZO2 (Zeroth-Order Offloading): Full Parameter Fine-Tuning 175B LLMs with 18GB GPU Memory

   Python 159 10

2. Tiny-FSDP Tiny-FSDP Public

   Tiny-FSDP, a minimalistic re-implementation of the PyTorch FSDP

   Python 51 4

3. Tiny-DeepSpeed Tiny-DeepSpeed Public

   Tiny-DeepSpeed, a minimalistic re-implementation of the DeepSpeed library

   Python 32 6

4. Tiny-Megatron Tiny-Megatron Public

   Tiny-Megatron, a minimalistic re-implementation of the Megatron library

   Python 12 1

5. MetaProfiler MetaProfiler Public

   MetaProfiler is a lightweight, structure-agnostic operator-level profiler for PyTorch models that leverages MetaTensor execution to simulate and benchmark individual ops without loading the full mo…

   Python 1

6. Streaming-Dataloader Streaming-Dataloader Public

   A memory-efficient streaming data loader designed for LLM pretraining under limited CPU and GPU memory constraints

   Python 2