[question] Need clarification on the purpose and performance benefits of GarbageCollection class

[qsh-zh]

For the impl

I have several questions about the motivation and use cases for this class:

Could you provide examples of scenarios where this class can improves performance? compare against default Python GC?

To my understanding, during backward, activation cuda memory should be released timely when we run backward in computational graph, will the GarbageCollection affect how we release cuda memory?

What are the tradeoffs of disabling automatic GC (gc.disable())?

[qsh-zh]

Are there cases where using this class might lead to worse performance or memory usage?

[tianyu-l]

This was added in #148. You can check the discussions there for more context.

cc: @lessw2020 @yifuwang -- The GC change we made seems worth better documentation.

[fegin]

In distributed training, Python's garbage collection (GC) can kick in at different times for different ranks, leading to slow communication and stragglers. To avoid this issue, we need to perform manual GC to ensure all ranks collect garbage simultaneously. Consider a scenario where we're using FSDP and Python triggers garbage collection for rank0 during the first allgather, rank1 during the second allgather, and so on. In this case, communication will be severely slowed down due to the straggling effect caused by asynchronous garbage collection.

While disabling Python GC might seem like an easy solution, it's not feasible due to potential circular dependencies in the code. We still rely on GC to recycle some Python objects. Disabling GC altogether could lead to CPU or GPU memory exhaustion.

So, performing manual GC is the best approach to resolve this issue.

[qsh-zh]

hi @fegin thank for your response. If we disable GC and run collection manually, will it fail to release CUDA memory in a timely manner? For example, during the backward pass, activations from later layers should ideally be released after backpropagation uses them for gradient computation in the previous layer as GC not collect zero reference count variables?

[yifuwang]

@qsh-zh disabling gc doesn't affect acyclic refs, which destruct the underlying object when the ref count reaches 0, even when gc is disabled. This trick only affects objects with cyclic references which are gc'd periodically anyway.

[qsh-zh]

@yifuwang @fegin thank you for explanation! looks good to me