Why use FSDP instead of Deepspeed?

[nrailg]

We found that the overhead of FSDP is very high compared to deepspeed. Especially if you upgrade transformer to >= 4.29 (huggingface/transformers#24724).

In Phase 3 PPO, seems I can only use FSDP?

Can anyone pls explain why FSDP is adopted? Thank U~

[lxuechen]

This can be transformers-version-dependent, but we generally find the same. See this note I wrote a long time ago. https://github.com/tatsu-lab/stanford_alpaca#addressing-oom

DeepSpeed model saving can be a little annoying at times.

[nrailg]

This can be transformers-version-dependent, but we generally find the same. See this note I wrote a long time ago. https://github.com/tatsu-lab/stanford_alpaca#addressing-oom

DeepSpeed model saving can be a little annoying at times.

Thanks for Ur reply.