Update multi-node.qmd #1688
Update multi-node.qmd #1688
Conversation
Added Distributed Finetuning For Multi-Node with Axolotl and Deepspeed
|
@muellerzr seem right? |
|
This seems to assume that you have access to each node before your training starts. However, a lot of cloud systems like AzureML, SLURM, SageMaker does not let you follow guides like this because the assumptions of the guide is that you can modify these variables. @shahdivax @winglian I would suggest a bit more of an automatic setup if you want this to work well for users. |
This assumes that user are using EC2 instances from AWS. ( I forgot to add that 😓) Edit: Added in the heading |
| On Node 1 (server), run the finetuning process using Accelerate: | ||
|
|
||
| ```bash | ||
| accelerate launch -m axolotl.cli.train examples/llama-2/qlora.yml | ||
| ``` | ||
|
|
||
| This will start the finetuning process across all nodes. You can check the different IP addresses before each step to verify that the training is running on every node. |
There was a problem hiding this comment.
To my knowledge this is not the case. You need to do accelerate launch -m on every server else it will sit there and never actually start
There was a problem hiding this comment.
When we tasted, we were only starting on single node (server) and it was able to use the resources from other nodes,
As a proof, we were able to see the ip of both the machines on the left, and in the total GPU it were showing all the GPU form all the nodes.
There was a problem hiding this comment.
@muellerzr My guess is this is probably specific to deepspeed since the IP addresses are set in a hostfile. We should probably disambiguate this that it only needs to be run on the first node when this is the case. Most other cases like FSDP or plain multinode DDP will likely still need accelerate launch to be run on each node.
There was a problem hiding this comment.
@winglian, @muellerzr That might be the case , because for us deepspeed was a good options where we were using multi node for finetuning via EC2, as it provides the public ip , and we used hostfile, it was really easy to connect both machines and run the finetuning on root only, this indeed connected all the other instances. (using all the resources from all the nodes via single node)
Co-authored-by: Wing Lian <wing.lian@gmail.com>
Title: Distributed Finetuning For Multi-Node with Axolotl and Deepspeed
Description:
This PR introduces a comprehensive guide for setting up a distributed finetuning environment using Axolotl and Accelerate. The guide covers the following steps: