[Feature]: automatically select distributed inference backend

[youkaichao]

🚀 The feature, motivation and pitch

ray is kind of an overkill for single gpu case, but is currently the only choice for multi-node inference.

we can add an auto backend, that checks the world size and the number of gpus available in the node, if this fits within this node, we can use multiprocessing, otherwise we can use ray.

this will help performance a lot.

@njhill do you have any bandwidth for this?

Alternatives

No response

Additional context

No response

[Yard1]

this will help performance a lot.

Given that we have almost completely replaced Ray communication layer for data (and soon for control), I doubt there will be a huge performance difference between multiprocessing and Ray at this point. But it would be good to benchmark.

[njhill]

@youkaichao yep I had been intending to do this as soon as I had a chance, should be able to tomorrow.

[njhill]

@youkaichao I have opened #5230, PTAL.