-
-
Notifications
You must be signed in to change notification settings - Fork 4.4k
/
keyedvectors.py
2514 lines (2081 loc) · 98.8 KB
/
keyedvectors.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
# Author: Shiva Manne <manneshiva@gmail.com>
# Copyright (C) 2018 RaRe Technologies s.r.o.
# Licensed under the GNU LGPL v2.1 - http://www.gnu.org/licenses/lgpl.html
"""This module implements word vectors and their similarity look-ups.
Since trained word vectors are independent from the way they were trained (:class:`~gensim.models.word2vec.Word2Vec`,
:class:`~gensim.models.fasttext.FastText`, :class:`~gensim.models.wrappers.wordrank.WordRank`,
:class:`~gensim.models.wrappers.varembed.VarEmbed` etc), they can be represented by a standalone structure,
as implemented in this module.
The structure is called "KeyedVectors" and is essentially a mapping between *entities*
and *vectors*. Each entity is identified by its string id, so this is a mapping between {str => 1D numpy array}.
The entity typically corresponds to a word (so the mapping maps words to 1D vectors),
but for some models, the key can also correspond to a document, a graph node etc. To generalize
over different use-cases, this module calls the keys **entities**. Each entity is
always represented by its string id, no matter whether the entity is a word, a document or a graph node.
Why use KeyedVectors instead of a full model?
=============================================
+---------------------------+--------------+------------+-------------------------------------------------------------+
| capability | KeyedVectors | full model | note |
+---------------------------+--------------+------------+-------------------------------------------------------------+
| continue training vectors | ❌ | ✅ | You need the full model to train or update vectors. |
+---------------------------+--------------+------------+-------------------------------------------------------------+
| smaller objects | ✅ | ❌ | KeyedVectors are smaller and need less RAM, because they |
| | | | don't need to store the model state that enables training. |
+---------------------------+--------------+------------+-------------------------------------------------------------+
| save/load from native | | | Vectors exported by the Facebook and Google tools |
| fasttext/word2vec format | ✅ | ❌ | do not support further training, but you can still load |
| | | | them into KeyedVectors. |
+---------------------------+--------------+------------+-------------------------------------------------------------+
| append new vectors | ✅ | ✅ | Add new entity-vector entries to the mapping dynamically. |
+---------------------------+--------------+------------+-------------------------------------------------------------+
| concurrency | ✅ | ✅ | Thread-safe, allows concurrent vector queries. |
+---------------------------+--------------+------------+-------------------------------------------------------------+
| shared RAM | ✅ | ✅ | Multiple processes can re-use the same data, keeping only |
| | | | a single copy in RAM using |
| | | | `mmap <https://en.wikipedia.org/wiki/Mmap>`_. |
+---------------------------+--------------+------------+-------------------------------------------------------------+
| fast load | ✅ | ✅ | Supports `mmap <https://en.wikipedia.org/wiki/Mmap>`_ |
| | | | to load data from disk instantaneously. |
+---------------------------+--------------+------------+-------------------------------------------------------------+
TL;DR: the main difference is that KeyedVectors do not support further training.
On the other hand, by shedding the internal data structures necessary for training, KeyedVectors offer a smaller RAM
footprint and a simpler interface.
How to obtain word vectors?
===========================
Train a full model, then access its `model.wv` property, which holds the standalone keyed vectors.
For example, using the Word2Vec algorithm to train the vectors
.. sourcecode:: pycon
>>> from gensim.test.utils import common_texts
>>> from gensim.models import Word2Vec
>>>
>>> model = Word2Vec(common_texts, size=100, window=5, min_count=1, workers=4)
>>> word_vectors = model.wv
Persist the word vectors to disk with
.. sourcecode:: pycon
>>> from gensim.test.utils import get_tmpfile
>>> from gensim.models import KeyedVectors
>>>
>>> fname = get_tmpfile("vectors.kv")
>>> word_vectors.save(fname)
>>> word_vectors = KeyedVectors.load(fname, mmap='r')
The vectors can also be instantiated from an existing file on disk
in the original Google's word2vec C format as a KeyedVectors instance
.. sourcecode:: pycon
>>> from gensim.test.utils import datapath
>>>
>>> wv_from_text = KeyedVectors.load_word2vec_format(datapath('word2vec_pre_kv_c'), binary=False) # C text format
>>> wv_from_bin = KeyedVectors.load_word2vec_format(datapath("euclidean_vectors.bin"), binary=True) # C bin format
What can I do with word vectors?
================================
You can perform various syntactic/semantic NLP word tasks with the trained vectors.
Some of them are already built-in
.. sourcecode:: pycon
>>> import gensim.downloader as api
>>>
>>> word_vectors = api.load("glove-wiki-gigaword-100") # load pre-trained word-vectors from gensim-data
>>>
>>> result = word_vectors.most_similar(positive=['woman', 'king'], negative=['man'])
>>> print("{}: {:.4f}".format(*result[0]))
queen: 0.7699
>>>
>>> result = word_vectors.most_similar_cosmul(positive=['woman', 'king'], negative=['man'])
>>> print("{}: {:.4f}".format(*result[0]))
queen: 0.8965
>>>
>>> print(word_vectors.doesnt_match("breakfast cereal dinner lunch".split()))
cereal
>>>
>>> similarity = word_vectors.similarity('woman', 'man')
>>> similarity > 0.8
True
>>>
>>> result = word_vectors.similar_by_word("cat")
>>> print("{}: {:.4f}".format(*result[0]))
dog: 0.8798
>>>
>>> sentence_obama = 'Obama speaks to the media in Illinois'.lower().split()
>>> sentence_president = 'The president greets the press in Chicago'.lower().split()
>>>
>>> similarity = word_vectors.wmdistance(sentence_obama, sentence_president)
>>> print("{:.4f}".format(similarity))
3.4893
>>>
>>> distance = word_vectors.distance("media", "media")
>>> print("{:.1f}".format(distance))
0.0
>>>
>>> sim = word_vectors.n_similarity(['sushi', 'shop'], ['japanese', 'restaurant'])
>>> print("{:.4f}".format(sim))
0.7067
>>>
>>> vector = word_vectors['computer'] # numpy vector of a word
>>> vector.shape
(100,)
>>>
>>> vector = word_vectors.wv.word_vec('office', use_norm=True)
>>> vector.shape
(100,)
Correlation with human opinion on word similarity
.. sourcecode:: pycon
>>> from gensim.test.utils import datapath
>>>
>>> similarities = model.wv.evaluate_word_pairs(datapath('wordsim353.tsv'))
And on word analogies
.. sourcecode:: pycon
>>> analogy_scores = model.wv.evaluate_word_analogies(datapath('questions-words.txt'))
and so on.
"""
from __future__ import division # py3 "true division"
from itertools import chain
import logging
from numbers import Integral
try:
from queue import Queue, Empty
except ImportError:
from Queue import Queue, Empty # noqa:F401
from numpy import dot, float32 as REAL, memmap as np_memmap, \
double, array, zeros, vstack, sqrt, newaxis, integer, \
ndarray, sum as np_sum, prod, argmax
import numpy as np
from gensim import utils, matutils # utility fnc for pickling, common scipy operations etc
from gensim.corpora.dictionary import Dictionary
from six import string_types, integer_types
from six.moves import zip, range
from scipy import stats
from gensim.utils import deprecated
from gensim.models.utils_any2vec import (
_save_word2vec_format,
_load_word2vec_format,
ft_ngram_hashes,
)
from gensim.similarities.termsim import TermSimilarityIndex, SparseTermSimilarityMatrix
#
# For backwards compatibility, see https://github.com/RaRe-Technologies/gensim/issues/2201
#
from gensim.models.deprecated.keyedvectors import EuclideanKeyedVectors # noqa
logger = logging.getLogger(__name__)
class Vocab(object):
"""A single vocabulary item, used internally for collecting per-word frequency/sampling info,
and for constructing binary trees (incl. both word leaves and inner nodes).
"""
def __init__(self, **kwargs):
self.count = 0
self.__dict__.update(kwargs)
def __lt__(self, other): # used for sorting in a priority queue
return self.count < other.count
def __str__(self):
vals = ['%s:%r' % (key, self.__dict__[key]) for key in sorted(self.__dict__) if not key.startswith('_')]
return "%s(%s)" % (self.__class__.__name__, ', '.join(vals))
class BaseKeyedVectors(utils.SaveLoad):
"""Abstract base class / interface for various types of word vectors."""
def __init__(self, vector_size):
self.vectors = zeros((0, vector_size))
self.vocab = {}
self.vector_size = vector_size
self.index2entity = []
def save(self, fname_or_handle, **kwargs):
super(BaseKeyedVectors, self).save(fname_or_handle, **kwargs)
@classmethod
def load(cls, fname_or_handle, **kwargs):
return super(BaseKeyedVectors, cls).load(fname_or_handle, **kwargs)
def similarity(self, entity1, entity2):
"""Compute cosine similarity between two entities, specified by their string id."""
raise NotImplementedError()
def most_similar(self, **kwargs):
"""Find the top-N most similar entities.
Possibly have `positive` and `negative` list of entities in `**kwargs`.
"""
return NotImplementedError()
def distance(self, entity1, entity2):
"""Compute distance between vectors of two input entities, specified by their string id."""
raise NotImplementedError()
def distances(self, entity1, other_entities=()):
"""Compute distances from a given entity (its string id) to all entities in `other_entity`.
If `other_entities` is empty, return the distance between `entity1` and all entities in vocab.
"""
raise NotImplementedError()
def get_vector(self, entity):
"""Get the entity's representations in vector space, as a 1D numpy array.
Parameters
----------
entity : str
Identifier of the entity to return the vector for.
Returns
-------
numpy.ndarray
Vector for the specified entity.
Raises
------
KeyError
If the given entity identifier doesn't exist.
"""
if entity in self.vocab:
result = self.vectors[self.vocab[entity].index]
result.setflags(write=False)
return result
else:
raise KeyError("'%s' not in vocabulary" % entity)
def add(self, entities, weights, replace=False):
"""Append entities and theirs vectors in a manual way.
If some entity is already in the vocabulary, the old vector is kept unless `replace` flag is True.
Parameters
----------
entities : list of str
Entities specified by string ids.
weights: list of numpy.ndarray or numpy.ndarray
List of 1D np.array vectors or a 2D np.array of vectors.
replace: bool, optional
Flag indicating whether to replace vectors for entities which already exist in the vocabulary,
if True - replace vectors, otherwise - keep old vectors.
"""
if isinstance(entities, string_types):
entities = [entities]
weights = np.array(weights).reshape(1, -1)
elif isinstance(weights, list):
weights = np.array(weights)
in_vocab_mask = np.zeros(len(entities), dtype=np.bool)
for idx, entity in enumerate(entities):
if entity in self.vocab:
in_vocab_mask[idx] = True
# add new entities to the vocab
for idx in np.nonzero(~in_vocab_mask)[0]:
entity = entities[idx]
self.vocab[entity] = Vocab(index=len(self.vocab), count=1)
self.index2entity.append(entity)
# add vectors for new entities
self.vectors = vstack((self.vectors, weights[~in_vocab_mask]))
# change vectors for in_vocab entities if `replace` flag is specified
if replace:
in_vocab_idxs = [self.vocab[entities[idx]].index for idx in np.nonzero(in_vocab_mask)[0]]
self.vectors[in_vocab_idxs] = weights[in_vocab_mask]
def __setitem__(self, entities, weights):
"""Add entities and theirs vectors in a manual way.
If some entity is already in the vocabulary, old vector is replaced with the new one.
This method is alias for :meth:`~gensim.models.keyedvectors.BaseKeyedVectors.add` with `replace=True`.
Parameters
----------
entities : {str, list of str}
Entities specified by their string ids.
weights: list of numpy.ndarray or numpy.ndarray
List of 1D np.array vectors or 2D np.array of vectors.
"""
if not isinstance(entities, list):
entities = [entities]
weights = weights.reshape(1, -1)
self.add(entities, weights, replace=True)
def __getitem__(self, entities):
"""Get vector representation of `entities`.
Parameters
----------
entities : {str, list of str}
Input entity/entities.
Returns
-------
numpy.ndarray
Vector representation for `entities` (1D if `entities` is string, otherwise - 2D).
"""
if isinstance(entities, string_types):
# allow calls like trained_model['office'], as a shorthand for trained_model[['office']]
return self.get_vector(entities)
return vstack([self.get_vector(entity) for entity in entities])
def __contains__(self, entity):
return entity in self.vocab
def most_similar_to_given(self, entity1, entities_list):
"""Get the `entity` from `entities_list` most similar to `entity1`."""
return entities_list[argmax([self.similarity(entity1, entity) for entity in entities_list])]
def closer_than(self, entity1, entity2):
"""Get all entities that are closer to `entity1` than `entity2` is to `entity1`."""
all_distances = self.distances(entity1)
e1_index = self.vocab[entity1].index
e2_index = self.vocab[entity2].index
closer_node_indices = np.where(all_distances < all_distances[e2_index])[0]
return [self.index2entity[index] for index in closer_node_indices if index != e1_index]
def rank(self, entity1, entity2):
"""Rank of the distance of `entity2` from `entity1`, in relation to distances of all entities from `entity1`."""
return len(self.closer_than(entity1, entity2)) + 1
class WordEmbeddingsKeyedVectors(BaseKeyedVectors):
"""Class containing common methods for operations over word vectors."""
def __init__(self, vector_size):
super(WordEmbeddingsKeyedVectors, self).__init__(vector_size=vector_size)
self.vectors_norm = None
self.index2word = []
@property
@deprecated("Attribute will be removed in 4.0.0, use self instead")
def wv(self):
return self
@property
def index2entity(self):
return self.index2word
@index2entity.setter
def index2entity(self, value):
self.index2word = value
@property
@deprecated("Attribute will be removed in 4.0.0, use self.vectors instead")
def syn0(self):
return self.vectors
@syn0.setter
@deprecated("Attribute will be removed in 4.0.0, use self.vectors instead")
def syn0(self, value):
self.vectors = value
@property
@deprecated("Attribute will be removed in 4.0.0, use self.vectors_norm instead")
def syn0norm(self):
return self.vectors_norm
@syn0norm.setter
@deprecated("Attribute will be removed in 4.0.0, use self.vectors_norm instead")
def syn0norm(self, value):
self.vectors_norm = value
def __contains__(self, word):
return word in self.vocab
def save(self, *args, **kwargs):
"""Save KeyedVectors.
Parameters
----------
fname : str
Path to the output file.
See Also
--------
:meth:`~gensim.models.keyedvectors.WordEmbeddingsKeyedVectors.load`
Load saved model.
"""
# don't bother storing the cached normalized vectors
kwargs['ignore'] = kwargs.get('ignore', ['vectors_norm'])
super(WordEmbeddingsKeyedVectors, self).save(*args, **kwargs)
def word_vec(self, word, use_norm=False):
"""Get `word` representations in vector space, as a 1D numpy array.
Parameters
----------
word : str
Input word
use_norm : bool, optional
If True - resulting vector will be L2-normalized (unit euclidean length).
Returns
-------
numpy.ndarray
Vector representation of `word`.
Raises
------
KeyError
If word not in vocabulary.
"""
if word in self.vocab:
if use_norm:
result = self.vectors_norm[self.vocab[word].index]
else:
result = self.vectors[self.vocab[word].index]
result.setflags(write=False)
return result
else:
raise KeyError("word '%s' not in vocabulary" % word)
def get_vector(self, word):
return self.word_vec(word)
def words_closer_than(self, w1, w2):
"""Get all words that are closer to `w1` than `w2` is to `w1`.
Parameters
----------
w1 : str
Input word.
w2 : str
Input word.
Returns
-------
list (str)
List of words that are closer to `w1` than `w2` is to `w1`.
"""
return super(WordEmbeddingsKeyedVectors, self).closer_than(w1, w2)
def most_similar(self, positive=None, negative=None, topn=10, restrict_vocab=None, indexer=None):
"""Find the top-N most similar words.
Positive words contribute positively towards the similarity, negative words negatively.
This method computes cosine similarity between a simple mean of the projection
weight vectors of the given words and the vectors for each word in the model.
The method corresponds to the `word-analogy` and `distance` scripts in the original
word2vec implementation.
Parameters
----------
positive : list of str, optional
List of words that contribute positively.
negative : list of str, optional
List of words that contribute negatively.
topn : int or None, optional
Number of top-N similar words to return, when `topn` is int. When `topn` is None,
then similarities for all words are returned.
restrict_vocab : int, optional
Optional integer which limits the range of vectors which
are searched for most-similar values. For example, restrict_vocab=10000 would
only check the first 10000 word vectors in the vocabulary order. (This may be
meaningful if you've sorted the vocabulary by descending frequency.)
Returns
-------
list of (str, float) or numpy.array
When `topn` is int, a sequence of (word, similarity) is returned.
When `topn` is None, then similarities for all words are returned as a
one-dimensional numpy array with the size of the vocabulary.
"""
if isinstance(topn, Integral) and topn < 1:
return []
if positive is None:
positive = []
if negative is None:
negative = []
self.init_sims()
if isinstance(positive, string_types) and not negative:
# allow calls like most_similar('dog'), as a shorthand for most_similar(['dog'])
positive = [positive]
# add weights for each word, if not already present; default to 1.0 for positive and -1.0 for negative words
positive = [
(word, 1.0) if isinstance(word, string_types + (ndarray,)) else word
for word in positive
]
negative = [
(word, -1.0) if isinstance(word, string_types + (ndarray,)) else word
for word in negative
]
# compute the weighted average of all words
all_words, mean = set(), []
for word, weight in positive + negative:
if isinstance(word, ndarray):
mean.append(weight * word)
else:
mean.append(weight * self.word_vec(word, use_norm=True))
if word in self.vocab:
all_words.add(self.vocab[word].index)
if not mean:
raise ValueError("cannot compute similarity with no input")
mean = matutils.unitvec(array(mean).mean(axis=0)).astype(REAL)
if indexer is not None and isinstance(topn, int):
return indexer.most_similar(mean, topn)
limited = self.vectors_norm if restrict_vocab is None else self.vectors_norm[:restrict_vocab]
dists = dot(limited, mean)
if not topn:
return dists
best = matutils.argsort(dists, topn=topn + len(all_words), reverse=True)
# ignore (don't return) words from the input
result = [(self.index2word[sim], float(dists[sim])) for sim in best if sim not in all_words]
return result[:topn]
def similar_by_word(self, word, topn=10, restrict_vocab=None):
"""Find the top-N most similar words.
Parameters
----------
word : str
Word
topn : int or None, optional
Number of top-N similar words to return. If topn is None, similar_by_word returns
the vector of similarity scores.
restrict_vocab : int, optional
Optional integer which limits the range of vectors which
are searched for most-similar values. For example, restrict_vocab=10000 would
only check the first 10000 word vectors in the vocabulary order. (This may be
meaningful if you've sorted the vocabulary by descending frequency.)
Returns
-------
list of (str, float) or numpy.array
When `topn` is int, a sequence of (word, similarity) is returned.
When `topn` is None, then similarities for all words are returned as a
one-dimensional numpy array with the size of the vocabulary.
"""
return self.most_similar(positive=[word], topn=topn, restrict_vocab=restrict_vocab)
def similar_by_vector(self, vector, topn=10, restrict_vocab=None):
"""Find the top-N most similar words by vector.
Parameters
----------
vector : numpy.array
Vector from which similarities are to be computed.
topn : int or None, optional
Number of top-N similar words to return, when `topn` is int. When `topn` is None,
then similarities for all words are returned.
restrict_vocab : int, optional
Optional integer which limits the range of vectors which
are searched for most-similar values. For example, restrict_vocab=10000 would
only check the first 10000 word vectors in the vocabulary order. (This may be
meaningful if you've sorted the vocabulary by descending frequency.)
Returns
-------
list of (str, float) or numpy.array
When `topn` is int, a sequence of (word, similarity) is returned.
When `topn` is None, then similarities for all words are returned as a
one-dimensional numpy array with the size of the vocabulary.
"""
return self.most_similar(positive=[vector], topn=topn, restrict_vocab=restrict_vocab)
@deprecated(
"Method will be removed in 4.0.0, use "
"gensim.models.keyedvectors.WordEmbeddingSimilarityIndex instead")
def similarity_matrix(self, dictionary, tfidf=None, threshold=0.0, exponent=2.0, nonzero_limit=100, dtype=REAL):
"""Construct a term similarity matrix for computing Soft Cosine Measure.
This creates a sparse term similarity matrix in the :class:`scipy.sparse.csc_matrix` format for computing
Soft Cosine Measure between documents.
Parameters
----------
dictionary : :class:`~gensim.corpora.dictionary.Dictionary`
A dictionary that specifies the considered terms.
tfidf : :class:`gensim.models.tfidfmodel.TfidfModel` or None, optional
A model that specifies the relative importance of the terms in the dictionary. The
columns of the term similarity matrix will be build in a decreasing order of importance
of terms, or in the order of term identifiers if None.
threshold : float, optional
Only embeddings more similar than `threshold` are considered when retrieving word
embeddings closest to a given word embedding.
exponent : float, optional
Take the word embedding similarities larger than `threshold` to the power of `exponent`.
nonzero_limit : int, optional
The maximum number of non-zero elements outside the diagonal in a single column of the
sparse term similarity matrix.
dtype : numpy.dtype, optional
Data-type of the sparse term similarity matrix.
Returns
-------
:class:`scipy.sparse.csc_matrix`
Term similarity matrix.
See Also
--------
:func:`gensim.matutils.softcossim`
The Soft Cosine Measure.
:class:`~gensim.similarities.docsim.SoftCosineSimilarity`
A class for performing corpus-based similarity queries with Soft Cosine Measure.
Notes
-----
The constructed matrix corresponds to the matrix Mrel defined in section 2.1 of
`Delphine Charlet and Geraldine Damnati, "SimBow at SemEval-2017 Task 3: Soft-Cosine Semantic Similarity
between Questions for Community Question Answering", 2017
<http://www.aclweb.org/anthology/S/S17/S17-2051.pdf>`_.
"""
index = WordEmbeddingSimilarityIndex(self, threshold=threshold, exponent=exponent)
similarity_matrix = SparseTermSimilarityMatrix(
index, dictionary, tfidf=tfidf, nonzero_limit=nonzero_limit, dtype=dtype)
return similarity_matrix.matrix
def wmdistance(self, document1, document2):
"""Compute the Word Mover's Distance between two documents.
When using this code, please consider citing the following papers:
* `Ofir Pele and Michael Werman "A linear time histogram metric for improved SIFT matching"
<http://www.cs.huji.ac.il/~werman/Papers/ECCV2008.pdf>`_
* `Ofir Pele and Michael Werman "Fast and robust earth mover's distances"
<https://ieeexplore.ieee.org/document/5459199/>`_
* `Matt Kusner et al. "From Word Embeddings To Document Distances"
<http://proceedings.mlr.press/v37/kusnerb15.pdf>`_.
Parameters
----------
document1 : list of str
Input document.
document2 : list of str
Input document.
Returns
-------
float
Word Mover's distance between `document1` and `document2`.
Warnings
--------
This method only works if `pyemd <https://pypi.org/project/pyemd/>`_ is installed.
If one of the documents have no words that exist in the vocab, `float('inf')` (i.e. infinity)
will be returned.
Raises
------
ImportError
If `pyemd <https://pypi.org/project/pyemd/>`_ isn't installed.
"""
# If pyemd C extension is available, import it.
# If pyemd is attempted to be used, but isn't installed, ImportError will be raised in wmdistance
from pyemd import emd
# Remove out-of-vocabulary words.
len_pre_oov1 = len(document1)
len_pre_oov2 = len(document2)
document1 = [token for token in document1 if token in self]
document2 = [token for token in document2 if token in self]
diff1 = len_pre_oov1 - len(document1)
diff2 = len_pre_oov2 - len(document2)
if diff1 > 0 or diff2 > 0:
logger.info('Removed %d and %d OOV words from document 1 and 2 (respectively).', diff1, diff2)
if not document1 or not document2:
logger.info(
"At least one of the documents had no words that were in the vocabulary. "
"Aborting (returning inf)."
)
return float('inf')
dictionary = Dictionary(documents=[document1, document2])
vocab_len = len(dictionary)
if vocab_len == 1:
# Both documents are composed by a single unique token
return 0.0
# Sets for faster look-up.
docset1 = set(document1)
docset2 = set(document2)
# Compute distance matrix.
distance_matrix = zeros((vocab_len, vocab_len), dtype=double)
for i, t1 in dictionary.items():
if t1 not in docset1:
continue
for j, t2 in dictionary.items():
if t2 not in docset2 or distance_matrix[i, j] != 0.0:
continue
# Compute Euclidean distance between word vectors.
distance_matrix[i, j] = distance_matrix[j, i] = sqrt(np_sum((self[t1] - self[t2])**2))
if np_sum(distance_matrix) == 0.0:
# `emd` gets stuck if the distance matrix contains only zeros.
logger.info('The distance matrix is all zeros. Aborting (returning inf).')
return float('inf')
def nbow(document):
d = zeros(vocab_len, dtype=double)
nbow = dictionary.doc2bow(document) # Word frequencies.
doc_len = len(document)
for idx, freq in nbow:
d[idx] = freq / float(doc_len) # Normalized word frequencies.
return d
# Compute nBOW representation of documents.
d1 = nbow(document1)
d2 = nbow(document2)
# Compute WMD.
return emd(d1, d2, distance_matrix)
def most_similar_cosmul(self, positive=None, negative=None, topn=10):
"""Find the top-N most similar words, using the multiplicative combination objective,
proposed by `Omer Levy and Yoav Goldberg "Linguistic Regularities in Sparse and Explicit Word Representations"
<http://www.aclweb.org/anthology/W14-1618>`_. Positive words still contribute positively towards the similarity,
negative words negatively, but with less susceptibility to one large distance dominating the calculation.
In the common analogy-solving case, of two positive and one negative examples,
this method is equivalent to the "3CosMul" objective (equation (4)) of Levy and Goldberg.
Additional positive or negative examples contribute to the numerator or denominator,
respectively - a potentially sensible but untested extension of the method.
With a single positive example, rankings will be the same as in the default
:meth:`~gensim.models.keyedvectors.WordEmbeddingsKeyedVectors.most_similar`.
Parameters
----------
positive : list of str, optional
List of words that contribute positively.
negative : list of str, optional
List of words that contribute negatively.
topn : int or None, optional
Number of top-N similar words to return, when `topn` is int. When `topn` is None,
then similarities for all words are returned.
Returns
-------
list of (str, float) or numpy.array
When `topn` is int, a sequence of (word, similarity) is returned.
When `topn` is None, then similarities for all words are returned as a
one-dimensional numpy array with the size of the vocabulary.
"""
if isinstance(topn, Integral) and topn < 1:
return []
if positive is None:
positive = []
if negative is None:
negative = []
self.init_sims()
if isinstance(positive, string_types) and not negative:
# allow calls like most_similar_cosmul('dog'), as a shorthand for most_similar_cosmul(['dog'])
positive = [positive]
all_words = {
self.vocab[word].index for word in positive + negative
if not isinstance(word, ndarray) and word in self.vocab
}
positive = [
self.word_vec(word, use_norm=True) if isinstance(word, string_types) else word
for word in positive
]
negative = [
self.word_vec(word, use_norm=True) if isinstance(word, string_types) else word
for word in negative
]
if not positive:
raise ValueError("cannot compute similarity with no input")
# equation (4) of Levy & Goldberg "Linguistic Regularities...",
# with distances shifted to [0,1] per footnote (7)
pos_dists = [((1 + dot(self.vectors_norm, term)) / 2) for term in positive]
neg_dists = [((1 + dot(self.vectors_norm, term)) / 2) for term in negative]
dists = prod(pos_dists, axis=0) / (prod(neg_dists, axis=0) + 0.000001)
if not topn:
return dists
best = matutils.argsort(dists, topn=topn + len(all_words), reverse=True)
# ignore (don't return) words from the input
result = [(self.index2word[sim], float(dists[sim])) for sim in best if sim not in all_words]
return result[:topn]
def doesnt_match(self, words):
"""Which word from the given list doesn't go with the others?
Parameters
----------
words : list of str
List of words.
Returns
-------
str
The word further away from the mean of all words.
"""
self.init_sims()
used_words = [word for word in words if word in self]
if len(used_words) != len(words):
ignored_words = set(words) - set(used_words)
logger.warning("vectors for words %s are not present in the model, ignoring these words", ignored_words)
if not used_words:
raise ValueError("cannot select a word from an empty list")
vectors = vstack(self.word_vec(word, use_norm=True) for word in used_words).astype(REAL)
mean = matutils.unitvec(vectors.mean(axis=0)).astype(REAL)
dists = dot(vectors, mean)
return sorted(zip(dists, used_words))[0][1]
@staticmethod
def cosine_similarities(vector_1, vectors_all):
"""Compute cosine similarities between one vector and a set of other vectors.
Parameters
----------
vector_1 : numpy.ndarray
Vector from which similarities are to be computed, expected shape (dim,).
vectors_all : numpy.ndarray
For each row in vectors_all, distance from vector_1 is computed, expected shape (num_vectors, dim).
Returns
-------
numpy.ndarray
Contains cosine distance between `vector_1` and each row in `vectors_all`, shape (num_vectors,).
"""
norm = np.linalg.norm(vector_1)
all_norms = np.linalg.norm(vectors_all, axis=1)
dot_products = dot(vectors_all, vector_1)
similarities = dot_products / (norm * all_norms)
return similarities
def distances(self, word_or_vector, other_words=()):
"""Compute cosine distances from given word or vector to all words in `other_words`.
If `other_words` is empty, return distance between `word_or_vectors` and all words in vocab.
Parameters
----------
word_or_vector : {str, numpy.ndarray}
Word or vector from which distances are to be computed.
other_words : iterable of str
For each word in `other_words` distance from `word_or_vector` is computed.
If None or empty, distance of `word_or_vector` from all words in vocab is computed (including itself).
Returns
-------
numpy.array
Array containing distances to all words in `other_words` from input `word_or_vector`.
Raises
-----
KeyError
If either `word_or_vector` or any word in `other_words` is absent from vocab.
"""
if isinstance(word_or_vector, string_types):
input_vector = self.word_vec(word_or_vector)
else:
input_vector = word_or_vector
if not other_words:
other_vectors = self.vectors
else:
other_indices = [self.vocab[word].index for word in other_words]
other_vectors = self.vectors[other_indices]
return 1 - self.cosine_similarities(input_vector, other_vectors)
def distance(self, w1, w2):
"""Compute cosine distance between two words.
Calculate 1 - :meth:`~gensim.models.keyedvectors.WordEmbeddingsKeyedVectors.similarity`.
Parameters
----------
w1 : str
Input word.
w2 : str
Input word.
Returns
-------
float
Distance between `w1` and `w2`.
"""
return 1 - self.similarity(w1, w2)
def similarity(self, w1, w2):
"""Compute cosine similarity between two words.
Parameters
----------
w1 : str
Input word.
w2 : str
Input word.
Returns
-------
float
Cosine similarity between `w1` and `w2`.
"""
return dot(matutils.unitvec(self[w1]), matutils.unitvec(self[w2]))
def n_similarity(self, ws1, ws2):
"""Compute cosine similarity between two sets of words.
Parameters
----------
ws1 : list of str
Sequence of words.
ws2: list of str
Sequence of words.
Returns
-------
numpy.ndarray
Similarities between `ws1` and `ws2`.
"""
if not(len(ws1) and len(ws2)):
raise ZeroDivisionError('At least one of the passed list is empty.')
v1 = [self[word] for word in ws1]
v2 = [self[word] for word in ws2]
return dot(matutils.unitvec(array(v1).mean(axis=0)), matutils.unitvec(array(v2).mean(axis=0)))
@staticmethod
def _log_evaluate_word_analogies(section):
"""Calculate score by section, helper for