Multi GPU with PEFT on LLM

[avishniakov]

This PR brings the multi-GPU DDP showcase to the PEFT training.
It is using the function helper from the core and the other changes are mostly related to Accelerate specifically.

ZenML core companion PR: zenml-io/zenml#2746 [ blocking merging of this one ]

[dagshub]

Join the discussion on DagsHub!

[avishniakov]

@schustmi @htahir1 you are optional reviewers, just in case you have interest 🙂

[htahir1]

I havnt given this a fair shake but in general what is the better way of doing this vs using subprocess :-D Any ideas?

[avishniakov]

I havnt given this a fair shake but in general what is the better way of doing this vs using subprocess :-D Any ideas?

Not sure, if this answers your question, but I plan to extend the ZenML core to serve automated capabilities for the creation of wrappers + making calls. https://zenml.atlassian.net/jira/software/c/projects/OSSK/boards/13?selectedIssue=OSSK-535
What's, in general, off with subprocessing in your opinion?

[htahir1]

@avishniakov Personally I feel they are quite unstable and unreliable... a better way would to use the internal library and do this in code right?

[avishniakov]

@avishniakov Personally I feel they are quite unstable and unreliable... a better way would to use the internal library and do this in code right?

In theory, we can hack around the accelerate.command.launch module, but in this module they still will call subprocess.Popen for you, so you cannot get away from subprocessing anyway. I will explore how we can use the module directly.

[avishniakov]

I somewhat heavily reworked how the preparation of the functions was done in this project. This is tightly coupled with the changes on the ZenML side.

Looking forward to some conceptual feedback. There are definitely a few weak points:

• Cache invalidation mechanism due to the use of the external function is far from perfect.
• The calls are handled via the function from inside the step. It is not straightforward to make the step "script-function" by itself. This is doable but would need more effort and shaking of how we work with steps in the core.

[avishniakov]

To be merged after 0.58.3/0.59.0 is released