Top-k sampling and top-p sampling for generating phrases on batches with GPT-2?

[Barbara931120]

How can I generate on batches with GPT-2 if I want to make use of these awesome sampling techniques: top-k sampling and top-p sampling?.

There is this implementation for generating phrases on batches already in issue #3021.

Any advice? thanks!

@patrickvonplaten

[patrickvonplaten]

It's on our ToDo-List :-) Currently batch generation with GPT2 is not possible, so you will have to rely on the code in #3021

[Barbara931120]

Can top-k and top-p sampling be implemented in batches?

[patrickvonplaten]

Sure, the provided top-k-top-p sampling function provided that :-)