Initial add of torch.distributed.pipelining

[kwen2501]

Stack from ghstack (oldest at bottom):

• -> Initial add of torch.distributed.pipelining #124776

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @awgu @penguinwu @fegin @XilunWu @wanchaol @fduwjj @wz337 @tianyu-l @wconstab @yf225 @chauhang @d4l3k

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/124776

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit 866111e with merge base c82fcb7 ():

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• pull / linux-focal-py3.12-clang10 / test (dynamo, 3, 3, linux.2xlarge) (gh)
  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[awgu]

Random thought: Any chance that torch.distributed.pipeline_parallel is an option? :)

[kwen2501]

Thanks for the suggestion @awgu !
We did consider torch.distributed.pipeline_parallel but unfortunately it seemed a bit long :)
Also, some people may argue that pipelining is not a kind of parallelism in a strict sense.

[wanchaol]

Thanks for the suggestion @awgu ! We did consider torch.distributed.pipeline_parallel but unfortunately it seemed a bit long :) Also, some people may argue that pipelining is not a kind of parallelism in a strict sense.

I was about to say I have a similar suggestion with @awgu, I feel pipeline_parallel aligns well with other parts of our offerings: fully_sharded_data_parallel, tensor_parallel.

pipelining is not a kind of parallelism in a strict sense.

Curious why it's not a kind of parallelism?

[kwen2501]

fully_sharded_data_parallel, tensor_parallel

Where are these names in our package offering?

Curious why it's not a kind of parallelism?

It is really a subtle difference, here is an answer from Quora's bot:

Pipelining and parallelism are both techniques used in computer architecture to improve performance, but they operate in different ways.
Pipelining involves breaking down the execution of instructions into a series of stages, where each stage performs a different part of the instruction. This allows multiple instructions to be processed simultaneously, with each stage working on a different instruction. As a result, the overall throughput of the processor is increased.
Parallelism, on the other hand, involves executing multiple instructions simultaneously by using multiple processing units. This can be achieved through techniques such as multi-core processors or multi-processor systems. Parallelism allows for true simultaneous execution of instructions, which can significantly improve overall system performance.
In summary, pipelining focuses on breaking down the execution of individual instructions into smaller stages to improve throughput, while parallelism involves executing multiple instructions at the same time using multiple processing units to improve overall system performance.

In short, pipelining focuses on breaking down a job, while parallelism focuses on having multiple workers do the same job.

[wconstab]

I think the name 'pipeline parallel' is sufficiently established that it doesn't matter what quora says, people know what it means and understand it. This is the obvious safe option to me.

Pipelining is arguably a correct name but also not one the ML community uses as often. It is also shorter and clean. I would have been more convinced by this if I wasn't already so used to saying pp, but I am.

[kwen2501]

Thanks @wconstab. I agree that "pipeline parallel" is a well-known concept -- it is also what we use in our README. But, as a package name, I think it is too long. "PP" is short, but no descriptive enough.

Ideally, I prefer a package name that's one word, such as "distributed", "compiler", "profiler", "export". It can also be two shortened meaningful words concatenated, such as "autograd". But I think "pipepara" looks weird.

[wconstab]

Are the links to pippy repo going to migrate too eventually, or do we leave the examples there?

[kwen2501]

The temporary decision is leaving the examples there (not migrated). Eventually they would be hosted in some tutorial repo I hope.

[kwen2501]

@pytorchbot merge

[pytorchmergebot]

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here