Implement Soft Cosine Measure

[Witiko]

Introduction

I implemented the Soft Cosine Measure (SCM) [wiki, 1, 2] as a part of research for my thesis [3]. Although the original algorithm [1] has a time complexity that is quadratic in the document length, I implemented a linear-time approximative algorithm that I sketch in [3, sec. 4.4]. Since Gensim was such an indispensable asset in my work, I thought I would give back and contribute code. The implementation is showcased in a jupyter notebook on corpora from the SemEval 2016 and 2017 competitions.

Description

My original implementation closely followed the Gensim implementation of the Word Mover's Distance (WMD), which is split into a gensim.models.keyedvectors.EuclideanKeyedVectors.wmdistance method that takes two token lists and computes the WMD for them, and into the gensim.similarity.WdmSimilarity class that provides batch similarity queries. However, I was not quite happy with this for the following reasons:

1. Not all useful term similarity matrices are constructed using word embeddings. Therefore, putting the entire logic into gensim.models.keyedvectors.EuclideanKeyedVectors immediately seemed like a bad idea that would hinder further extensions.
2. By automatically converting token lists into bag-of-words representation behind the curtain, the user is unable to apply document length normalization methods such as tf-idf.

For the above reasons, I ultimately decided to split the implementation into a function, a method, and a class as follows:

1. The gensim.matutils.softcossim function takes two documents in the bag-of-words representation, a sparse term similarity matrix in the scipy CSC format, and computes SCM.
2. The gensim.models.keyedvectors.EuclideanKeyedVectors.similarity_matrix method takes a corpus of bag-of-words vectors, a dictionary, and produces a sparse term similarity matrix Mrel described by Charlet and Damnati, 2017 [1].
3. The gensim.similarities.SoftCosineSimilarity class takes a corpus of bag-of-words vectors, a sparse term similarity matrix in the scipy CSC format, and provides batch SCM queries against the corpus.

The above design achieves a much looser coupling between the individual components and eliminates the original concerns. I demonstrate the implementation in a jupyter notebook on the corpus of Yelp reviews. The approximative linear-time approximative algorithm for SCM achieves about the same speed as the linear-time approximative algorithm for WMD (see the corresponding jupyter notebook).

Future work

The gensim.similarities.SoftCosineSimilarity class goes over the entire corpus and computes the SCM between the query and each document separately by calling gensim.matutils.softcossim. If performance is a concern, SCM can be computed in a single step by computing q^T * S * C, where q is the normalized query vector, S is the term similarity matrix, C is the normalized term-document matrix of the corpus, and “normalized” in this context stands for a vector v being divided by sqrt(v^T * S * v). This is similar to what e.g. the gensim.similarity.MatrixSimilarity.get_similarities method does, only with the basic cosine similarity rather than SCM.

References

1. Grigori Sidorov et al. Soft Similarity and Soft Cosine Measure: Similarity of Features in Vector Space Model, 2014. (link to PDF)
2. Delphine Charlet and Geraldine Damnati, SimBow at SemEval-2017 Task 3: Soft-Cosine Semantic Similarity between Questions for Community Question Answering, 2017. (link to PDF)
3. Vít Novotný, Vector Space Representations in Information Retrieval (preprint), 2017. (link to PDF)

[Witiko]

I added numpy-style documentation and unit tests. Hopefully, the code should be good to go now.

[menshikh-iv]

Great work @Witiko, in general, looks nice!

[menshikh-iv]

unused var

[Witiko]

Fixed in 08dea4e.

[menshikh-iv]

sentence_obama = 'Obama speaks to the media in Illinois'
sentence_obama = sentence_obama.lower().split()

to

sentence_obama = 'Obama speaks to the media in Illinois'.lower().split()

same for other sentences.

[Witiko]

Fixed in 621ed0d.

[menshikh-iv]

To track time of cell, better to use "magic" %%time ,

%%time

.. <SOME CODE> ..

instead of

start = time()

.. <SOME CODE> ..

print('Cell took %.2f seconds to run.' % (time() - start))

here and everywhere.

[Witiko]

Fixed in 8af5f67.

[menshikh-iv]

Better to use gensim-data functionality instead of this part

import gensim.downloader as api
model = api.load("word2vec-google-news-300")

[menshikh-iv]

It will be great if we add this dataset to gensim-data and use it here. Can you investigate @Witiko, is this possible (Yelp license allows us to share it or not)?

[Witiko]

I don't think we can, the license seems to explicitly forbid any distribution of the dataset (see section C). The dataset is also used in the Word Mover's Distance notebook, but from the description, it contained different data back when the notebook was created. This would indicate that the dataset keeps changing over time.

[Witiko]

Should I update both the WMD and SCS notebooks to use some open dataset available in gensim-data?

[menshikh-iv]

For this case, I see two possible solutions:

• Leave as is (because Yelp dataset is really good for this task, but we can't store it in gensim-data), probably add link, when user can download this dataset
• Choice other datasets (that will be really nice for demonstration + possible to add it to gensim-data by license). @Witiko have you good candidates for it?

[Witiko]

I use the SemEval 2015–2017 Task 3 Subtask B question answering datasets in my thesis. These are also a good fit for the semantic similarity task and are permissively licensed, so it should be possible to add them to gensim-data.

[menshikh-iv]

Nice, please replace Yelp dataset with SemEval + create an issue in gensim-data with needed info (guide: https://github.com/RaRe-Technologies/gensim-data#want-to-add-a-new-corpus-or-model), probably add other "tasks" from semeval will be really good too.

[Witiko]

I am running the evaluations. If all looks fine, I will submit an issue to gensim-data. Until then, please consider the current state of the Jupyter Notebook temporary.

[menshikh-iv]

Please use hanging indents (instead of vertical).

[Witiko]

Fixed in e1eb7cd and c7f0ce1.

[menshikh-iv]

similarity_matrix is symmetrical, maybe better to store only "half" of this matrix -> reduce memory usage twice?

[Witiko]

I agree that such a saving would be nice, but there seems to be no support in SciPy for a dot product of a symmetrical matrix that only stores the upper / lower triangle ((vec1.T).dot(similarity_matrix).dot(vec2)[0, 0]). Even beyond SciPy, I don't know of a sparse matrix format that would allow both row-wise and column-wise efficient access.

[menshikh-iv]

Sad but true, thanks for the clarification.

[Witiko]

Note that this knowledge can still be useful when storing and transmitting the similarity matrix.

[menshikh-iv]

if num_rows instead of len(dictionary) ?

[Witiko]

Fixed in effef71.

[menshikh-iv]

corpus isn't used

[Witiko]

Fixed in 08dea4e.

[menshikh-iv]

are you sure that this an equivalent?

[Witiko]

Yes, this exact if-then-else statement currently appears at six places in test_similarities.py at gensim/develop. Rather than add two new lines for SoftCosineSimilarity at each of the six locations, I decided to refactor the if-then-else statement into factory methods.

[Witiko]

Should I turn this refactoring step into a separate pull request?

[menshikh-iv]

No (because this is really small change).

[Witiko]

Ok, I was more concerned with the fact that this pull request may take a little longer to finish up, whereas this single refactoring is independent and could be incorporated fast.

[menshikh-iv]

@Witiko also please merge develop from upstream to your branch.

[Witiko]

I will look into these, thank you for taking the time to do the review.

[menshikh-iv]

ping @Witiko, how is going?

[Witiko]

I am planning to push new commits this weekend.

[Witiko]

@evanmiltenburg That should be interesting; I will see if I can run a comparison by the end of the week.

[thomasniebler]

@evanmiltenburg @Witiko I already worked on that :) Long story short: The soft cosine with respective weights can improve semantic word similarity (and I'm sure, also semantic textual similarity).
I learned the respective weights as a posdef matrix using a metric learning approach (seeing as this the soft cosine is actually the same) from evaluation datasets like WordSimilarity353 or MEN. If you are interested in the code, see here: https://thomasniebler.github.io/semantics-metriclearning/
The corresponding paper can be read at https://arxiv.org/abs/1705.07425 and has been published as a poster at the ISWC2017: https://www.thomas-niebler.de/pub/niebler2017rrl.pdf
If you have any questions regarding that, let me know.

[evanmiltenburg]

@thomasniebler: Note that MEN has been criticized for not being a good measure of similarity. Rather, it measures relatedness between terms. See the SimLex-999 dataset for an alternative.

[thomasniebler]

@evanmiltenburg You're correct. Indeed, improving relatedness was my first goal with this measure. However, since we're talking about skewing the vector space, learning on SimLex data should allow us to learn similarity. Maybe not as strict similarity scores as with other metric learning based algorithms (Retrofitting, Counterfitting, Paragram Embeddings), as those do not consider intensity scores, but still. Overall, you have a good point. I will consider this in future work.
Just as a side note: If needed, just try an algorithm from the https://github.com/metric-learn/ repositories. There should be plenty of metric learning algorithms that give you a matrix for the soft cosine that is learned on similarity/dissimilarity constraints. Technically, even the retrofitting/counterfitting/paragram algorithms are nothing different than metric learning algorithms applied to semantic tasks, as they learn a (nonlinear) transformation of the embedding vector space.

[Naruto-Sasuke]

I wonder it can be used in image patch similarity. For example, replacing L2 loss with this.

[evanmiltenburg]

Well, the issue for me is not how to learn a model that can predict good similarity measures. The issue is that when people try soft cosine for themselves, without any transformations of the vector space, what will the model be good at? It's good to know this in advance, and it's important to be precise in your claims about this. (Don't want to give users the wrong impression and then disappoint them.)

The STS task is a decent intrinsic test to see whether the model can predict whether two sentences are similar. It's a nice addition to the (slightly more extrinsic) Community Question Answering task that the author of this issue already tested Soft Cosine Similarity on.

[thomasniebler]

Well, technically, the soft cosine measure is a vector transformation, as it makes use of the dot product which can be easily parameterized by a quadratic matrix as e.g. done in the Mahalanobis distance. By using a parameterized similarity or distance measure, you already transforming the vector space, as metric spaces are defined by their metrics.
So basically, you rewrite the soft cosine as <a, Sb> / (sqrt(<a, Sa>) * sqrt(<b, Sb>)) and thus you transformed the vector space. If you don't want to transform your vector space, well, then you shouldn't use the soft cosine. But I admit that there is a difference in learning S and using a simple heuristic.

[menshikh-iv]

@Witiko great work 👍