Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initialize with GloVe and train on the target corpus #296

Closed
jgc128 opened this issue Aug 15, 2017 · 2 comments

Comments

@jgc128
Copy link

@jgc128 jgc128 commented Aug 15, 2017

Hi,

Could you please explain what is the proper way of using the parameter -pretrainedVectors for a initialization of the embeddings before training them on the target corpus?

I would like to initialize the embeddings with GloVe vectors and then tune it on the target corpus. Is using the parameter -pretrainedVectors the proper way of doing this? The description of this parameter says pretrained word vectors for supervised learning [] - aren't word vectors trained in an unsupervised fashion?

More over, I could not find any documentation how to load a pre-trained binary model and continue the training.

Thanks!

@EdouardGrave

This comment has been minimized.

Copy link
Member

@EdouardGrave EdouardGrave commented Aug 17, 2017

Hi @jgc128,

The -pretrainedVectors option was added mostly to initialize the word embeddings of supervised models with vectors which were trained in an unsupervised fashion. Do you want to initialize an unsupervised model (cbow or skipgram) with pre-trained vectors?

If so, I believe that using the -pretrainedVectors option is the proper way of doing this (I have never tried this though). It will initialize the embeddings and then fine-tune these on the target corpus.

By the way, you can also try to use our recently released word vectors, trained on Wikipedia and common crawl, which are available on the fastText website: https://fasttext.cc/docs/en/english-vectors.html.

As of now, it is not possible to load pre-trained binary models.

Please feel free to re-open this task if this does not answer your questions!

Best,
Edouard.

@jgc128

This comment has been minimized.

Copy link
Author

@jgc128 jgc128 commented Aug 17, 2017

Hi,

Do you want to initialize an unsupervised model (cbow or skipgram) with pre-trained vectors?

Yes, that's exactly what I would like to do.

By the way, you can also try to use our recently released word vectors, trained on Wikipedia and common crawl

Great! I did not know about this! It's not on this page:
https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md

Thank you for your responses, it's very helpful!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.