Multi-gpu inference, the query gets stuck when using my own provider

[Aaron-mindverse]

I modified this file with my own defined provider. No problems with single cards, but query gets stuck with multiple cards. Here is the file I mainly modified.
"DeepSpeed-MII/mii/models/providers/huggingface.py"

Now it's stuck

Here is my deepspeed version

[mrwyattii]

Hi @Aaron-mindverse it's not clear why your provider is hanging with multi-GPU. If you copy and paste the code (or provide a link to a branch with the modifications) I can test it on my side. But I would first recommend taking a look at our BloomPipeline and load_hf_llm:

DeepSpeed-MII/mii/models/providers/llm.py

Line 25 in 747072b

class BloomPipeline(object):

Use this as a template for creating your OPTProvider and it should work with multi-GPU. Thanks!

[Aaron-mindverse]

Hi @Aaron-mindverse it's not clear why your provider is hanging with multi-GPU. If you copy and paste the code (or provide a link to a branch with the modifications) I can test it on my side. But I would first recommend taking a look at our BloomPipeline and load_hf_llm:

DeepSpeed-MII/mii/models/providers/llm.py

Line 25 in 747072b

class BloomPipeline(object):

Use this as a template for creating your OPTProvider and it should work with multi-GPU. Thanks!

Thank you so much, this worked for me, although I had to modify a lot of DeepSpeed code specifically tailored for the Bloom model, such as 'get_transformer_name'

https://github.com/microsoft/DeepSpeed/blob/21c28029648df4c98acd6331a8fdc0d29f297fa7/deepspeed/module_inject/replace_module.py#L124

[mrwyattii]

@Aaron-mindverse I don't think you should need to modify DS to get the OPT models running. In general, the OPT models will work with the HuggingFace provider. Could you share the modified code and I can help debug why you're running into Bloom-specific code paths on the DeepSpeed side?