Skip to content

An open collection of implementation tips, tricks and resources for training large language models

License

Notifications You must be signed in to change notification settings

huggingface/large_language_model_training_playbook

Repository files navigation

📖 The Large Language Model Training Playbook

This playbook is a companion to the LLM Training Handbook which contains a lot more details and scripts.

An open collection of implementation tips, tricks and resources for training large language models.

The following covers questions related to various topics which are interesting or challenging when training large language models.

Deciding on a model parallelism strategy

Deciding on the model size

Scaling laws

Trade-off of large language model sizes

Issues and questions related to tensor precision

What to chose between fp32, fp16, bf16

Mixed-precisions for optimizers, weights, specifics modules

How to finetune and integrate a model trained in a precision in another precision

Learning rate and learning rate schedules

Questions on batch size

Detecting instabilities early

Training tips to reduce instabilities

Issues with data and data processing

Tips on what metrics to follow during the training

Releases

No releases published

Packages

No packages published

Languages