Finetune with transformer-xl pretrained models

[mudong0419]

Hi, thanks for your excellent work. Transformer-xl is the most elegant model for long sequences by now. Do you plan to finetune pretrained models for document classification just like Bert?

[kimiyoung]

Yes I believe it's a good direction. Transformer-XL is presumably good for document-level representations due to the ability of handling long context. On short text, Transformer-XL might also have an edge (see results on One Billion Word).

[BoPengGit]

Hi!

Thank you so much for your time! I'm trying to learn more about transformers-xl. Could you please help answer this beginner question?

Is it possible to change the loss function and fine tune the transformer-xl model to do classification, similar to BERT? So for the transformer-xl model, there isn't one general pre-trained model like in BERT, but instead there are many pre-trained models based on different tasks?

Thank you so much for your time and God bless!

[Nfekih]

@BoPengGit did you manage to create a transfoXL based classifier like Bert ?

[BoPengGit]

I don't remember, this was a long time ago.

[chunduri11]

Is it possible to classify document of length 30k tokens/words, using transformer-XL?