No text is shown when using MII in fp32 and greedy search

[marshmellow77]

When using greedy search (do_sample=False) and dtype=fp32 the generated tokens are not shown in the output of the query. I believe the text generation is happening, because different values for max_new_tokens lead to different runtimes for the query. See this notebook as a minimal example.

• possibly related to Error when trying MII with dtype=fp32 with sampling #101
• 1 T4, GPU memory 16GB
• deepspeed-mii version 0.0.3
• transformers version 4.24.0
• Amazon Linux 2

[mrwyattii]

We don't currently support fp32 for the Bloom models in MII & DeepSpeed-Inference. I believe this is because the checkpoints are all in half precision. We correctly check the configs with Bloom-176B model, but fail to do so for the smaller variants. I added a fix for this in #107

I just ran your example using fp16 and I see output.

[marshmellow77]

Thansk for adding a warning.