[WIP] Implement Word Mover's Distance (WMD) in Gensim

[olavurmortensen]

In response to issue #482.

We want to compute WMD between documents, based on "From Word Embeddings To Documents Distances" by Matt Kusner et al. (link to paper). Using word2vec embeddings, WMD finds the minimum "traveling distance" (in a matter of speaking) of two documents. The method shows great promise in kNN classification, as shown by Vlad Niculae and Matt Kusner in a blog post using Scikit-Learn.

Our implementation will be based on pyemd, which is a Python wrapper of a C implementation of EMD (Earth Mover's Distance) (link to GitHub repo) by Will Mayner, which will be directly included in Gensim.

After the simple implementation of a method that computes the WMD, a kNN classifier will be implemented, inspired by the aforementioned blog post of Vlad Niculae and Matt Kusner. Gensim's docsim module will be used for this purpose (link to docsim).

At the moment, only WMD between two documents is implemented in this PR, and it's done by importing pyemd. This Python wrapper needs to be directly included in Gensim, with no need for this external module.

@tmylk @piskvorky

[olavurmortensen]

@tmylk @piskvorky I'm done implementing the wmdistance method, using pyemd natively in Gensim.

[tmylk]

Great. Could you please add unit tests for this new functionality?

[piskvorky]

Good job! @tmylk please review.

@olavurmortensen how do I use it? Please provide a concrete example.

[tmylk]

@mkusner Should we expect RWMD to be much faster than WMD?
It gives very little speed up in these tests, maybe because it is numpy vs C implementation of EMD. If we can't improve RWMD runtime then suggest not using it at all.

RWMD code here is almost the same as in matlab RWMD code, just without a square root.

[tmylk]

How much time does this take? Where are the NaNs coming from?

[mkusner]

Hi all! Thanks a lot for spending so much careful time on this, I just
looked through the thread!

It depends a bit on the dataset how much RWMD speeds up WMD; we saw
speedups between 1.1x and 16x (in Figure 7 of the paper:
http://mkusner.github.io/publications/WMD.pdf) if EMD is in C and RWMD is
in numpy this could certainly shrink the speedup. RWMD is even more
effective at speeding up document distances when used to prune documents
for approximate WMD computation (in the prefetch and prune algorithm at the
end of Section 4 in the paper). I'd be happy to give more details about
this if this is something that people would want included.

In any case, thanks again!

On Sat, Jan 23, 2016 at 3:36 PM, Lev Konstantinovskiy <
notifications@github.com> wrote:

@mkusner https://github.com/mkusner Should we expect RWMD to be much
faster than WMD?
It gives very little speed up in these tests, maybe because it is numpy vs
C implementation of EMD. If we can't improve RWMD runtime then suggest not
using it at all.

RWMD code here is almost the same as in matlab RWMD code
https://github.com/mkusner/wmd/blob/master/compute_rwmd.m, just without
a square root.

—
Reply to this email directly or view it on GitHub
#521 (comment).

[tmylk]

Where is this variable used later?

[olavurmortensen]

In gensim.interfaces.SimilarityABC, which WmdSimilarity extends. The Similarity base class uses index to sort the documents.

[menshikh-iv]

Ping @olavurmortensen, what status of this PR? Will you finish it soon?

[ghost]

I believe part of this PR was merged in #659 (just WMD), and that that optimizations (RWMD and WCD) were taken up by @RishabGoel in #800.

I'm no longer working on this PR and haven't been for over a year. So the PR should be closed.

[olavurmortensen]

Just want to note that I used the wrong account in my previous comment (olavurfargen), but it was me.