Request for pre-fine tuning self-art weights

[carlos-gemmell]

Hi

I would like to obtain the self attention model weights before any fine tuning was done to it.
Does this link from the leaderboard contain such a weights file. If it is the fine tuned one, could you make the base self-att model for python available.

My intention is to pass these weights to a model benefitting from contextual word embeddings in PyTorch. Any information pertaining to the structure of the weights file would be beneficial.

Thank you

Carlos

[hamelsmu]

Hello Carlos, we did not use any pre-training / fine tuning approaches. This is one of the areas of opportunity we expressly identified for future researchers in our paper.

We would love to see what kind of results you get by applying this approach! Please let us know how it goes!