Skip to content
ETNLP: A toolkit to evaluate, extract, and visualize multiple embeddings
Python HTML Shell CSS
Branch: master
Clone or download
Latest commit be769b1 Oct 15, 2019
Type Name Latest commit message Commit time
Failed to load latest commit information.

ETNLP: A Toolkit for Extraction, Evaluation and Visualization of Pre-trained Word Embeddings

Table of contents

  1. Introduction
  2. More about ETNLP
  3. Installation and How to Use
  4. Download Resources

I. Overview

A glimpse of ETNLP:

II. How do I cite ETNLP?

Please CITE paper the Arxiv paper whenever ETNLP (or the pre-trained embeddings) is used to produce published results or incorporated into other software:

  title={ETNLP: A Visual-Aided Systematic Approach to Select Pre-Trained Embeddings for a Downstream Task},
  author={Vu, Xuan-Son and Vu, Thanh and Tran, Son N and Jiang, Lili},
  booktitle={Proceedings of the International Conference Recent Advances in Natural Language Processing (RANLP)},

III. More about ETNLP :

1. Embedding Evaluator:

To compare quality of embedding models on the word analogy task.

  • Input: a pre-trained embedding vector file (word2vec format), and word analogy file.
  • Output: (1) evaluate quality of the embedding model based on the MAP/P@10 score, (2) Paired t-tests to show significant level between different word embeddings.

1.1. Note: The word analogy list is created by:

  • Adopt from the English list by selecting suitable categories and translating to the target language (i.e., Vietnamese).
  • Removing inappropriate categories (i.e., category 6, 10, 11, 14) in the target language (i.e., Vietnamese).
  • Adding custom category that is suitable for the target language (e.g., cities and their zones in Vietnam for Vietnamese). Since most of this process is automatically done, it can be applied in other languages as well.

1.2. Selected categories for Vietnamese:

  1. capital-common-countries
  2. capital-world
  3. currency: E.g., Algeria | dinar | Angola | kwanza
  4. city-in-zone (Vietnam's cities and its zone)
  5. family (boy|girl | brother | sister)
  6. gram1-adjective-to-adverb (NOT USED)
  7. gram2-opposite (e.g., acceptable | unacceptable | aware | unaware)
  8. gram3-comparative (e.g., bad | worse | big | bigger)
  9. gram4-superlative (e.g., bad | worst | big | biggest)
  10. gram5-present-participle (NOT USED)
  11. gram6-nationality-adjective-nguoi-tieng (e.g., Albania | Albanian | Argentina | Argentinean)
  12. gram7-past-tense (NOT USED)
  13. gram8-plural-cac-nhung (e.g., banana | bananas | bird | birds) (NOT USED)
  14. gram9-plural-verbs (NOT USED)

1.3 Evaluation results (in details)

  • Analogy: Word Analogy Task

  • NER (w): NER task with hyper-parameters selected from the best F1 on validation set.

  • NER (w.o): NER task without selecting hyper-parameters from the validation set.

 Model NER.w NER.w.o Analogy
BiLC3 + w2v 89.01 89.41 0.4796
BiLC3 + Bert_Base 88.26 89.91 0.4609
BiLC3 + w2v_c2v 89.46 89.46 0.4796
BiLC3 + fastText 89.65 89.84 0.4970
BiLC3 + Elmo 89.67 90.84 0.4999
BiLC3 + MULTI_WC_F_E_B 91.09 91.75 0.4906

2. Embedding Extractor: To extract embedding vectors for other tasks.

  • Input: (1) list of input embeddings, (2) a vocabulary file.
  • Output: embedding vectors of the given vocab file in .txt, i.e., each line conains the embedding for a word. The file then be compressed in .gz format. This format is widely used in existing NLP Toolkits (e.g., Reimers et al. [1]).

Extra options:

  • -input-c2v: character embedding file
  • solveoov:1: to solve OOV words of the 1st embedding. Similarly for more than one embedding: e.g., solveoov:1:2.

[1] Nils Reimers and Iryna Gurevych, Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging, 2017,, arXiv.

3. Visualizer: to explore the embedding space and compare between different embeddings.

Screenshot of viewing multiple-embeddings side-by-side (Vietnamese):

Alt text

Screenshot of viewing each embedding interactively (Vietnamese):

Alt text

Screenshot of viewing each embedding side-by-side (English):

Alt text

IV. Installation and How to use ETNLP

1. Installation:

From source codes (Python 3.6.x):

  1. cd src/codes/
  2. pip install -r requirements.txt
  3. python install

From pip (python 3.6.x)

  1. sudo apt-get install python3-dev
  2. pip install cython
  3. pip install git+git://


  1. pip install etnlp

2. Examples

  1. cd src/examples
  2. python
  3. python
  4. python
  5. python

3. Visualization

Side-by-side visualization:

  1. sh src/codes/

Interactive visualization:

  1. sh src/codes/

V. Available Lexical Resources

1. Word Analogy List for Vietnamese

 Word Analogy List Download Link (NER Task) Download Link (General)
Vietnamese (This work) Link1 Link1
English (Mirkolov et al. [2]) [Link2] Link2
Portuguese (Hartmann et al. [3]) [Link3] Link3

2. Multiple pre-trained embedding models for Vietnamese

  • Training data: Wiki in Vietnamese:
 # of sentences # of tokenized words
 6,685,621 114,997,587
  • Download Pre-trained Embeddings:
    (Note: The MULTI_WC_F_E_B is the concatenation of four embeddings: W2V_C2V, fastText, ELMO, and Bert_Base.)
 Embedding Model Download Link (NER Task) Download Link (AIVIVN SentiTask) Download Link (General)
w2v Link1 (dim=300) [Link1] [Link1]
w2v_c2v Link2 (dim=300) [Link2] [Link2]
fastText Link3 (dim=300) [Link3] [Link3]
Elmo Link4 (dim=1024) Link4 (dim=1024) Link4 (dim=1024, 731MB and 1.9GB after extraction.)
Bert_base Link5 (dim=768) [Link5] [Link5]
MULTI_WC_F_E_B Link6 (dim=2392) [Link6] [Link6]

VI. Versioning

For transparency and insight into our release cycle, and for striving to maintain backward compatibility, ETNLP will be maintained under the Semantic Versioning guidelines as much as possible.

Releases will be numbered with the following format:


And constructed with the following guidelines:

  • Breaking backward compatibility bumps the major (and resets the minor and patch)
  • New additions without breaking backward compatibility bumps the minor (and resets the patch)
  • Bug fixes and misc changes bumps the patch

For more information on SemVer, please visit

You can’t perform that action at this time.