LargeScale

Set DATA_PATH, MULTITASK_DATA_PATH, CHECKPOINT_PATH in configs/glm-130b/glm-130b.sh and HOST_FILE_PATH in scripts/submit_gpu.sh. Run the following scripts to reproduce GLM-130B's training.

bash scripts/submit_gpu.sh configs/glm-130b/glm-130b.sh

At least 24 DGX-A100 (40G) is needed to lanuch training. A more detailed README will be released soon.

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
.github/workflows		.github/workflows
config_tasks		config_tasks
configs/glm-130b		configs/glm-130b
evaluation		evaluation
examples		examples
glm		glm
images		images
megatron		megatron
patch		patch
scripts		scripts
tasks		tasks
tests		tests
tools		tools
.gitignore		.gitignore
.gitlab-ci.yml		.gitlab-ci.yml
LICENSE		LICENSE
MANIFEST.in		MANIFEST.in
Makefile		Makefile
README.md		README.md
pretrain_bert.py		pretrain_bert.py
pretrain_glm.py		pretrain_glm.py
pretrain_gpt.py		pretrain_gpt.py
pretrain_ict.py		pretrain_ict.py
pretrain_prefix_lm.py		pretrain_prefix_lm.py
pretrain_t5.py		pretrain_t5.py
pretrain_vit.py		pretrain_vit.py
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt
run.sh		run.sh
run_bf16.sh		run_bf16.sh
run_fp16.sh		run_fp16.sh
setup.cfg		setup.cfg
setup.py		setup.py

License

BBuf/LargeScale

Folders and files

Latest commit

History

Repository files navigation

LargeScale

About

Resources

License

Stars

Watchers

Forks

Languages