Reformer for Squad and other QA

[hugokitano]

Hi, thanks for this codebase. I'm looking to adapt Reformer for Squad, and I'm not sure exactly how. Do you think it will look something like this, from the pytorch-transformers github?https://github.com/huggingface/transformers/blob/master/examples/run_squad.py

I'm having trouble seeing how I am going to pass two parameters (context and query) into the model. forward() function instead of one. Thanks!

[lucidrains]

@hugokitano Hi! I am working with non-language use cases, but for Squad, I think you should be able to encode the entire context and query, separated with some delimiter token, and pass it all into the Reformer?

[hugokitano]

Interesting, kind've like the "full Reformer sequence → sequence" example in the ReadMe? And that means I will have to pass the outputs of the Reformer through a decoder to get the final answer?

[lucidrains]

@hugokitano I think you can frame it many ways, and with enough parameters, it probably won't matter. T5 treated almost all language tasks as a sequence to sequence and got SOTA just by sheer scale. Otherwise, in the past, people have gotten it to work on one stack of attention like BERT

[lucidrains]

closing due to inactivity