sequence length independent generation

[tmphex]

Currently generation require passing sequence length to generate sequences of given length but say in tasks such as summary or translation, one doesn't know about the final sequence length. Currently I am trying to generate candidates with passing various lengths as work around. Also is it possible to add support for beam search method for generation in addition to current top_p/top_k methods.

[lucidrains]

@tmphex i believe there's already support for detecting end of string tokens https://github.com/lucidrains/x-transformers/blob/main/x_transformers/autoregressive_wrapper.py#L45

[lucidrains]

as for beam search, let me think about it - i saw some paper out there with some fast optimized beam search, and maybe its worth creating a separate repo for that

[tmphex]

Thanks @lucidrains for pointing out the eos_token I have created a PR to be able to pass arguments directly to model.generate function rather than need to call model.decoder.generate.

I look forward to try out the optimized beam search once available.

[lucidrains]

@tmphex ohh yes, you indeed found a bug, thank you! one other thing is that the eos_token may not work when the batch size is greater than 1, but i'll get that fixed this week (we can keep this issue open)

[tmphex]

@lucidrains just pinging if you gotten around fixing the eos_token and adding beam search support 🙏

[tmphex]

Newly released Fastseq (https://github.com/microsoft/fastseq) might be interesting to integrate with when you plan to work on optimizing generation and beam search support

[tmphex]

@lucidrains seems like eos_token issue has been fixed. If you plan to implement the beam-search then we can keep this issue open otherwise feel free to close it