How to use this for speech/audio generation?

[jinglescode]

Great work Phil! In their paper, the authors applied this model to speech modeling, how would you advise on what should I change to use for speech. Because in speech, the data are signals, we do not have num_tokens, nor do we have emb_dim. Our data input is simply, [batch, channel, time]. Any advice?

[lucidrains]

@jinglescode oh hey Hong! sorry I missed this issue

you should use https://github.com/lucidrains/vector-quantize-pytorch to first quantize your speech signals into discrete dictionary, and then use that to train your auto-regressive model!

[lucidrains]

https://openreview.net/forum?id=3InxcRQsYLf

[jinglescode]

Very helpful. Thanks!