[Core] remove cupy dependency #3625
Conversation
|
Support evidence from #3617 : this PR solved cupy initialization issue 👍 |
youkaichao
changed the title
|
Can we get confirmation from AMD people that we can do the same thing to replace PyTorch RCCL? While I believe we don't have to do this in the current PR, I'd like to get confirmation and find someone who takes that. |
|
@WoosukKwon @simon-mo thanks for the review! I talked to amd folks to ask for review. Let's see if we can get feedback today. If not, we can merge first, and let them send fix PRs later. |
simon-mo
added
the
release-blocker
vllm/worker/model_runner.py
Outdated
| # Delete the CUDA graphs before deleting the CuPy NCCL communicator. | ||
| # Delete the CUDA graphs before deleting the pynccl communicator. | ||
| # NOTE(woosuk): This is necessary because otherwise deadlocks can | ||
| # happen. | ||
| # FIXME(woosuk): This is a bit hacky. Find a more robust solution. | ||
| self.graph_runners.clear() | ||
| self.cupy_nccl_backend = None |
There was a problem hiding this comment.
I think this part can be deleted as we remove CuPy?
There was a problem hiding this comment.
I don't know why this code was needed before. So I don't want to change it 🤣 If you have more context, we can discuss if we can delete this.
There was a problem hiding this comment.
When using CuPy + CUDA graphs, deadlocks happen when the CuPy backend is deleted before the CUDA graphs using it are deleted. I actually don't know the reason for this, but this doesn't happen when using NCCL through PyTorch probably because the NCCL communicator managed by PyTorch is deleted at the very end.
There was a problem hiding this comment.
@youkaichao could you check whether we can delete this? You can simply run python examples/llm_engine_example.py -tp 2 and see if the process hangs.
There was a problem hiding this comment.
Okay, will have a try.
There was a problem hiding this comment.
BTW, I do agree with you that we can remove this code some time later. If you'd like to do so, could you please add TODO in the code?
There was a problem hiding this comment.
I tested the code (removing the above two lines) for 100 times for i in {1..100}; do python3 examples/llm_engine_example.py -tp=2 && echo "Test passed $i times" || break; done , and I don't see any deadlocks. This gives us confidence in removing the code later. I left a comment there to remove it in v0.4.1 .
|
Overall I think we should take a small step for every release. Distributed related bugs like hang/deadlock are highly unstable and difficult to test. My plan is to use cc @WoosukKwon |
Separate code from #3442 , only remove cupy dependency.