Some generate parameters do not work for query

[mallorbc]

When using DeepSpeed MII, there are some parameters that do not work when querying the model that otherwise work when using model.generate or when using huggingface pipelines. I have also tried these parameters using DeepSpeed inference on its own and found them to work

The parameters that cause issues for me are num_beams and bad_words_ids but there may be more.

I have found do_sample, max_length, min_length, top_k, top_p, temperature, repetition_penalty, and early_stopping to not cause issues but there may be more.

[mrwyattii]

At this point we support kwargs passed with query that are int, float, bool, or str types (see

DeepSpeed-MII/mii/grpc_related/proto/modelresponse.proto

Line 34 in 4040dae

message Value {

)

This means that's num_beams should work, but we do not support bad_words_ids as the type is Iterable[int].

Could you please share the error you are seeing with num_beams?

Thanks

[mallorbc]

@mrwyattii
I believe the num_beams was an error on my part.

Are there plans to support lists? If lists are added, I believe all the arguments would become supported

[heya5]

num_beams doesn't work for me. I got the error:

details = "Exception calling application: DeepSpeed does not support `num_beams` > 1, if this is important to you please add your request to: https://github.com/microsoft/DeepSpeed/issues/2506"