Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models on bacalhau

[js-ts]

ETA: 2023Q2

source https://twitter.com/jay_wooow/status/1556589324879085568?s=21&t=0zZfExIRj-ddccFHQ07yvA

Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models
abs: https://arxiv.org/abs/2208.03306

BTM training, a new algorithm to train an ELMFOREST, which contains many EXPERT LMs that can be added and removed, ensembled, or parameter averaged at any time

code https://github.com/hadasah/btm (model released!)

Stages

• Check for updates to the repo https://github.com/hadasah/btm
• get to run on google colab
• convert the notebook into a python script and run it on bacalhau GPU

[aronchick]

@js-ts did you work on this?