Use ATen CUDA event/stream wrappers in c10d

[pietern]

In c10d we still use legacy CUDA event and stream wrappers. These are not aware of the CUDA stream pool that was added by @mruberry in #9938.

One issue we need to solve beforehand is that currently Gloo algorithms take a stream at construction time and we cache them in the c10d process group. This needs to be redone to support passing a stream when running the algorithm instead of when it is constructed.

@mruberry I owed you this issue. We can use this issue to discuss how to proceed.

[mruberry]

"I owed you this issue" sounds so ominous!

The legacy wrappers do actually use the pool, but I agree it's likely better to not cache the stream upfront and acquire it later.

If the stream pool has caused a perf hit, then we have some options to consider addressing it:

• Use the high priority option when creating a stream. The high priority stream pool is separate from the low/default priority pool.
• Extend the stream API to allow the creation of a private pool.
• Extend the stream API to allow the creation of streams outside any pool, which are then managed directly by c10d.

To recap, moving from the legacy to the new wrappers will have no performance impact (aside from avoiding the forwarding from c10d->THC->ATen). If pooling has caused a perf hit (unclear?) then we have some options to address it, but these options also require changes in Gloo, and it may be easier to simply make the fix you mention in Gloo, anyway.

[pietern]

This is now almost done. Between your work in #13464 and the new style collectives I have added in ProcessGroupGloo this past week, we're no longer holding on to CUDA streams there. The code that did that isn't removed yet, pending plumbing through the timeout support in Gloo. Assigning to myself so that we can close this when the old code is really removed.

[teng-li]

Isn't this done @pietern ?

[pietern]

With #14295 merging shortly nothing is holding on to CUDA streams any more (beyond using them).