New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Training instruction followers as composable layers and expert layers #41

Open

huu4ontocord opened this issue May 4, 2023 · 1 comment

Assignees

Labels

Owner

huu4ontocord commented May 4, 2023

check out: /content/drive/Shareddrives/MDEL/dataset/minipile_instruct.jsonl

Train a 1b on the 300K minipile instructions? Train 5 variants:

Layers 14,15,16
Layers 4,5,6,7,8
Layers 4,5,6,7,8,14,15,16 (edited)
all layers
layers 9,10,11,12,13

Collaborator

mrcabbage972 commented May 5, 2023

@ontocord Can you please provide some context on what is the goal here?

jordiclive self-assigned this

P1ayer-1 self-assigned this

mrcabbage972 added Evaluation Training Runs and removed Evaluation labels

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment