Learn words representation with CBOW plus position-weights

[loretoparisi]

As for the recent paper "Learning Word Vectors for 157 Languages", the model CBOW is used with position dependent weights. Using that the new pre-trained model were produced.
Is it possible to train a unsupervised model with CBOW in this version of fastText using the same approach with position weights?

[luthfianto]

Also, which command in fasttext corresponds to the positional-weighting?

[matanox]

I don't think this is implemented in the publicly available codebase.

[loretoparisi]

@matanster you are right, it is not, at least not yet, while the most two recent papers show models that has been trained FastText with position dependent weights.

[ghost]

Do you know if there are any updates of positional-weighting in fasttext base code?

[loretoparisi]

@omriFdna so far I'm not aware of a version that implements the positional weights bow, but I will have a look maybe someone did...

[ghost]

@loretoparisi any luck?

[loretoparisi]

@omriFdna not yet so far :

[loretoparisi]

A dataset with position weights trained with CBOW has been recently released for Wikipedia and Common Crawl - https://fasttext.cc/docs/en/crawl-vectors.html

The model parameters were:

DIM	NGRAM	WS	NEG
300	5	5	10

It would be great to have this as training option as well.

[ghost]

I agree, they also report in their papers that position weights improves the performance, I wish it was part of the training options.

[adam2326]

+1

…

On Thu, Oct 11, 2018 at 2:58 AM Omri ***@***.***> wrote: I agree, they also report in their papers that position weights improves the performance, I wish it was part of the training options. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#445 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHUWqNswFJkmbxOlBP3L8vJjmMogNTYlks5ujuv9gaJpZM4STDKV> .

[Witiko]

I am attempting to add this feature to Gensim, see piskvorky/gensim#2905.

[Witiko]

The work from piskvorky/gensim#2905 has been accepted as a journal paper that is to appear in J.UCS 28:2. To conduct experiments for the paper, I have produced the high-level PInE library that uses a fork of Gensim 3.8.3 for model training. Perhaps the PInE library can serve as an inspiration for an implementation to facebook's fastText and to current Gensim. 💡