-
-
Notifications
You must be signed in to change notification settings - Fork 4.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ipython notebook for fastText comparison #815
Conversation
Great stuff! I would make all parameters explicit, to avoid seeing differences that are only because of different defaults (and to also better ensure consistent operation if the various packages' defaults are changed in the future). In particular, I've noticed that the fastText word2vec defaults differ from gensim Word2Vec (and word2vec.c before it) in three ways: skip-gram (instead of CBOW), initial learning rate of 0.05 (no matter the mode, instead of 0.025 for skip-gram), and downsampling threshold of 0.0001 (instead of 0.001). It'd be best to compare against the gensim Word2Vec results with those options. Also, except for the char-ngrams, the fastText For all the major steps, using notebook timing (eg FYI: I needed to patch the logging-setup with |
Great, thanks! Like @gojomo says, let's compare apples to apples, then draw conclusions. |
Thanks for the detailed response! Yes, I'd completely missed word2vec was using cbow, not skipgram as default. That could be a major difference. I'll recheck the other hyperparameters once too, and push the updated notebook. |
b64812a
to
6bbef25
Compare
6bbef25
to
2481dbe
Compare
So I've pushed a bunch of changes -
As for the logging to notebook vs console, I wasn't sure if logging to the notebook was a great idea since Gensim prints a lot of logs during training word2vec. The current notebook has these - they look a little ugly, so I'm thinking I should remove them |
Comments/suggestions on current notebook:
|
Thanks @jayantj @gojomo for the analysis. Super useful. Related to the "syntactic ngram mode": morphology for word embeddings (potentially combining the best of both worlds). Re. @gojomo 's last bullet point: perhaps different alpha decay across epochs? |
@piskvorky Yes, me and @tmylk have discussed that paper, the authors haven't published any related code though. I'll mail the authors asking them about it, we were thinking of reproducing their paper, and if it goes well maybe integrating it into gensim? @gojomo Really appreciate all the help and feedback you've given so far.
3, 4, 5. Yes, sounds good. Will do
|
2b65bc6
to
0395f4a
Compare
0395f4a
to
ecfc4e4
Compare
@gojomo I've made some changes, do you think this looks better? |
It looks pretty good! It's already very useful, but if you want to keep refining, other comments:
Your various conclusions about the reasons for observed results seem exactly right. I would add an extra caveat on the Brown numbers that the corpus is so small the results (and thus relative accuracies) could vary a lot from run to run. (For example, only more runs or more iterations would give me confidence that the fastText semantic-accuracy increase from 16.5% to 18.1% isn't just jitter.) (Also, it may be notable that the Finally, I think a summary graph of all time/accuracy results would help hammer-home the results... and might significantly drive interest in the reporting/discussing the notebook, if it's easy to pull out for display in other tweets/blogposts. Specifically: I'm thinking a bar graph, left axis train-time, right-axis analogy-accuracy-percent. Three bottom-axis clusters of bars: brown, text8, text9. Within each cluster, runtime(gensim, ft-ng, ft); semantic-acc (gensim, ft-ng, ft); syntactic-acc (gensim, ft-ng, ft). It's a lot to squeeze into one graph (27 bars!)... but it'd still be interpretable and likely be re-used a lot! |
Very nice @jayantj ! In addition to what @gojomo wrote (all great points), it looks like NLTK is doing something clever in What BLAS does your gensim installation use @jayantj ? |
"cell_type": "markdown", | ||
"metadata": {}, | ||
"source": [ | ||
"For training the models yourself, you'll need to have both [Gensim](github.com/RaRe-Technologies/gensim) and [FastText](https://github.com/facebookresearch/fastText) set up on your machine." |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The gensim link gives 404.
4effd58
to
5a4d607
Compare
Thanks for the great ideas. I've added the check-if-existing conditions, updated the links, and removed the @piskvorky Haven't manually installed any BLAS packages, and can't seem to find any with
|
b97ea8f
to
d3cf3d7
Compare
d3cf3d7
to
5d438a8
Compare
Worth installing OpenBlas then (and then re-install numpy+scipy). Btw easiest way to check BLAS linkage is with |
Yeah, ran that already, that gives me
This means it's linked, right? |
So I've added the text9 comparison, and a nice looking graph - thanks for the advice @gojomo One tiny nitpick - I''ve had to hardcode the training time values, there doesn't seem an easy way of retrieving the output of |
Hi there guys, I tried to run the notebook FastText_Tutorial.ipynb bu to no avail. I have installed fastText already. The error I get is when I try to import FastText from the gensim package, i.e.,
|
Hi. This is not yet released. You need to clone the develop branch from github |
Thanks Lev. But now another error is showing up when I try to import.
|
A brief comparison of word representations from fastText and word2vec. I got some interesting looking results while trying out fastText, so I thought I'd share them.
Jupyter notebook.
Let me know if this is worth merging, and if you think any changes/additions could be made.
@tmylk