how to inference by multi-node and multi-card?

[vicwer]

Thank you very much for your work. May I ask how to use multi-node and multi-card to do bloom model inference in deepspeed? I have 4nodes-4v100gpus.

[mayank31398]

Hi @vicwer
Currently, this repo doesn't support multi-node inference.
But you can look at run 176b in int8 on 4x V100 GPUs. I am not sure if it will work but its worth a try.

[vicwer]

@mayank31398 OK, thanks!

[mayank31398]

closing this :)

[TingchenFu]

Hi @vicwer do you find any possible way to run the 176b model with V100? How much V100s do you use?