Some plugins for using torchtitan.
- Load local dataset.
- Support wandb.
- Add gnorm stat.
- Support pp.
- Test dp.
- Test fsdp.
- Test tp.
- Support gpt2 tokenzer, hf tokenzer.
- Integrate xmixers.
- Add llama.
- [ ]
The benchmark results for speed, all tested with two A800 80g gpus.
small (465m, 36, 2k, fsdp+act):
| desc | tgs | memory |
|---|---|---|
| metaseq | 30k | 62.9 |
| torchtitan | 40k | 70 |
| xmixers | 43k | 44.64 |
large(7b, 8, 2k, fsdp+act):
| desc | tgs | memory |
|---|---|---|
| metaseq | 2.5k | 75 |
| torchtitan | 3.6k | 76 |
| xmixers | 3.8k | 71~76 |