Skip to content

ytsvetko/qvec

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
src
 
 
 
 
 
 
 
 
 
 

QVEC

Yulia Tsvetkov, ytsvetko@cs.cmu.edu

This is an easy-to-use, fast tool to measure the intrinsic quality of word vectors. The evaluation score depends on how well the word vectors correlate with a matrix of features from manually crafted lexical resources. The evaluation score is shown to correlate strongly with performance in downstream tasks (cf. Tsvetkov et al, 2015 for details and results). QVEC is model agnostic and thus can be used for evaluating word vectors produced by any given model.

Evaluation of Word Vector Representations by Subspace Alignment

Usage

Each vector file should have one word vector per line as follows (space delimited):-

the -1.0 2.4 -0.3 ...

Semantic content evaluation:

./qvec_cca.py --in_vectors  ${your_vectors} --in_oracle  oracles/semcor_noun_verb.supersenses.en    

To obtain vector column labels, add the --interpret parameter; to print top K values in each dimension add --top K:

./qvec.py --in_vectors ${your_vectors} --in_oracle oracles/semcor_noun_verb.supersenses.en --interpret --top 10

Multilingual evaluation for English, Danish, and Italian:

./qvec_cca.py --in_vectors  ${your_vectors} --in_oracle   --in_oracle oracles/semcor_noun_verb.supersenses.en,oracles/semcor_noun_verb.supersenses.it,oracles/semcor_noun_verb.supersenses.da 

Syntactic content evaluation:

./qvec_cca.py --in_vectors  ${your_vectors} --in_oracle  oracles/ptb.pos_tags    

Citation:

@InProceedings{qvec:enmlp:15,
author = {Tsvetkov, Yulia and Faruqui, Manaal and Ling, Wang and Lample, Guillaume and Dyer, Chris},
title={Evaluation of Word Vector Representations by Subspace Alignment},
booktitle={Proc. of EMNLP},
year={2015},
}

This repository is made available under the Open Database License: http://opendatacommons.org/licenses/odbl/1.0/. Any rights in individual contents of the database are licensed under the Database Contents License: http://opendatacommons.org/licenses/dbcl/1.0/

About

Intrinsic evaluation of word vectors

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages