Wikipedia Entity Vectors
Wikipedia Entity Vectors  is a distributed representation of words and named entities (NEs). The words and NEs are mapped to the same vector space. The vectors are trained with skip-gram algorithm using preprocessed Wikipedia text as the corpus.
Pre-trained vectors are downloadable from the Releases page.
Several old versions are available at this site.
Each version of the pre-trained vectors contains three files:
These files are text files in word2vec output format, in which the first line declares the vocabulary size and the vector size (number of dimensions), followed by lines of tokens and their corresponding vectors.
entity_vectors.txt contains vectors for words and NEs, respectively.
entity_vectors.txt, white spaces within names of NEs are replaced with underscores like
all_vectors.txt contains vectors of both words and embeddings in one file, where each NE token is formatted with square brackets like
Pre-trained vectors are trained under the configurations below (see Manual training for details):
We used mecab-ipadic-NEologd (v0.0.6) for tokenizing Japanese texts.
||path to the output file|
||path to a corpus file made with
||path to the output directory|
You can manually process Wikipedia dump file and train a skip-gram model on the preprocessed file.
- Python 3.6
- MeCab and its Python binding (mecab-python3) (optional: required for tokenizing Japanese texts)
Download Wikipedia Cirrussearch dump file from here.
- Make sure to choose a file named like
- Make sure to choose a file named like
Clone this repository.
Preprocess the downloaded dump file.
$ python make_corpus.py --cirrus_file <dump file> --output_file <corpus file>
If you're processing Japanese version of Wikipedia, make sure to use MeCab tokenizer by setting
--tokenizer mecaboption. Otherwise, the text will be tokenized by a simple rule based on regular expression.
Train the model
$ python train.py --corpus_file <corpus file> --output_dir <output directory>
You can configure options below for training a model.
usage: train.py [-h] --corpus_file CORPUS_FILE --output_dir OUTPUT_DIR [--embed_size EMBED_SIZE] [--window_size WINDOW_SIZE] [--sample_size SAMPLE_SIZE] [--min_count MIN_COUNT] [--epoch EPOCH] [--workers WORKERS] optional arguments: -h, --help show this help message and exit --corpus_file CORPUS_FILE Corpus file (.txt) --output_dir OUTPUT_DIR Output directory to save embedding files --embed_size EMBED_SIZE Dimensionality of the word/entity vectors  --window_size WINDOW_SIZE Maximum distance between the current and predicted word within a sentence  --sample_size SAMPLE_SIZE Number of negative samples  --min_count MIN_COUNT Ignores all words/entities with total frequency lower than this  --epoch EPOCH number of training epochs  --workers WORKERS Use these many worker threads to train the model 
There are several methods to learn distributed representations (or embeddings) of words, such as CBOW and skip-gram . These methods train a neural network to predict contextual words given a word in a sentence from large corpora in an unsupervised way.
However, there are a couple of problems when applying these methods to learning distributed representations of NEs. One problem is that many NEs consist of multiple words (such as "New York" and "George Washington"), which makes a simple tokenization of text undesirable.
Other problems are the diversity and ambiguity of NE mentions. For each NE, several expression can be used to mention the NE. For example, "USA", "US", "United States", and "America" can all express the same country. On the other hand, the same words and phrases can refer to different entities. For example, the word "Mercury" may represent a planet or an element or even a person (such as "Freddie Mercury", the vocalist for the rock group Queen). Therefore, in order to learn distributed representations of NEs, one must identify the spans of NEs in the text and recognize the mentioned NEs so that they are not treated just as a sequence of words.
To address these problems, we used Wikipedia as the corpus and utilized its internal hyperlinks for identifying mentions of NEs in article text.
For each article in Wikipedia, we performed the following preprocessing.
First, we extracted all hyperlinks (pairs of anchor text and the linked article) from the source text (a.k.a wikitext) of an article.
Next, for each hyperlink, we replaced the appearances of anchor text in the article body with special tokens representing the linked articles.
For instance, if an article has a hyperlink to "Mercury (planet)" assigned to the anchor text "Mercury", we replace all the other appearances of "Mercury" in the same article with the special token
Note that the diversity of NE mentions is resolved by replacing possibly diverse anchor texts with special tokens which are unique to NEs.
Moreover, the ambiguity of NE mentions is also addressed by making "one sense per discourse" assumption; we assume that NEs mentioned by possibly ambiguous phrases can be determined by the context or the document.
With this assumption, the phrases "Mercury" in the above example are neither replaced with
##Freddie_Mercury##, since the article does not have such mentions as hyperlinks.
We used the preprocessed Wikipedia articles as the corpus and applied skip-gram algorithm to learn distributed representations of words and NEs. This means that words and NEs are mapped to the same vector space.
The pre-trained vectors are distributed under the terms of the Creative Commons Attribution-ShareAlike 3.0.
The codes in this repository are distributed under the MIT License.
 Masatoshi Suzuki, Koji Matsuda, Satoshi Sekine, Naoaki Okazaki and Kentaro Inui. A Joint Neural Model for Fine-Grained Named Entity Classification of Wikipedia Articles. IEICE Transactions on Information and Systems, Special Section on Semantic Web and Linked Data, Vol. E101-D, No.1, pp.73-81, 2018.
 Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space. ICLR, 2013.
 Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, Jeff Dean. Distributed Representations of Words and Phrases and their Compositionality. NIPS, 2013.
This work was partially supported by Research and Development on Real World Big Data Integration and Analysis.