GloVe Thesaurus

This is a Julia web app that uses the GloVe word vector data to look up similarities among word meanings.

Installation

• Download the CommonCrawl 300-dimension data set: Common Crawl (840B tokens): 300d
• [OPTIONAL] Separate it into a .vec and a .idx file (this allows us to use a fast readdlm julia call):

wget http://nlp.stanford.edu/data/glove.840B.300d.zip

unzip -c glove.840B.300d.zip \
| awk '{for (i=2; i<NF; i++) printf $i " "; print $NF}' \
| head -100000 >crawl300-100k.vec

unzip -c glove.840B.300d.zip \
| awk '{print $1}' \
| head -100000 >crawl300-100k.idx

The .vec file is the large file--it contains only floating point values, one row per word, with 300 dimensions per row. The .idx file is simply an index of all of the words, each word listed per line, in the same order as the .vec file.

See the REQUIRE file for dependencies.

julia -e 'Pkg.add("Distances")'
julia -e 'Pkg.add("Morsel")'
julia -e 'Pkg.add("JSON")'
...

Running

$ julia 

               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "help()" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.3.3 (2014-11-23 20:19 UTC)
 _/ |\__'_|_|_|\__'_|  |  'Official http://julialang.org/ release'
|__/                   |  x86_64-apple-darwin13.3.0

julia> include("WordMatrix.jl")

julia> m = WordMatrix("crawl300-100k.idx")

Note that if you'd like to load the idx/vec file, just reference the .idx file and the accompanying .vec file will be assumed.

Example

julia> topn(m, m["laser"] + m["science"], 12)
12-element Array{(String,Float64),1}:
 ("laser",6.939878264355364)     
 ("science",7.804343256488264)   
 ("lasers",8.240093239632245)    
 ("scientific",8.946992980586012)
 ("physics",9.008729968872023)   
 ("technology",9.193295956585269)
 ("sciences",9.426696189473054)  
 ("astronomy",9.54593972112149)  
 ("imaging",9.547669144832966)   
 ("research",9.627897577880903)  
 ("optics",9.651007849517955)    
 ("Laser",9.724459820355438)