Example/Support of converting Fairseq Model to run in LightSeq

[pttzty]

I am curious of trying LightSeq to speed up my inference for a vanilla Transformer Encoder-Decoder (Vasawani 17) model. My original model was trained with FairSeq (or OpenNMT-py). Is there any example or places that you can refer to help me convert my transformer model to the format compatible of running LightSeq?

[Taka152]

@pttzty you could modify this file to export weight from FairSeq or OpenNMT, and pass the weight file to python API easily. Also, we will consider updating more examples for different training toolkit

[pttzty]

@Taka152 Thanks, I will see if there is a straight forward mapping that I can convert

[q2044757581]

@Taka152 i can't find an example with topk sampling, and don't know how to set the parameters, can you help?

[Taka152]

@q2044757581 check here to use top-k sampling, generation_method="topk" and topk=4 means top-4 sampling on huggingface bart model. In detail, this parameter is stored here that will be passed to LightSeq c++ backend

[q2044757581]

@Taka152 does lightseq support T5 model?

[Taka152]

@Taka152 does lightseq support T5 model?

#42