R C++
Switch branches/tags
Clone or download
Latest commit 718dac7 Jun 12, 2018

README.md

CRAN_Status_Badge Travis-CI Build Status codecov License Downloads Follow

You've just discovered text2vec!

text2vec is an R package which provides an efficient framework with a concise API for text analysis and natural language processing (NLP).

Goals which we aimed to achieve as a result of development of text2vec:

  • Concise - expose as few functions as possible
  • Consistent - expose unified interfaces, no need to explore new interface for each task
  • Flexible - allow to easily solve complex tasks
  • Fast - maximize efficiency per single thread, transparently scale to multiple threads on multicore machines
  • Memory efficient - use streams and iterators, not keep data in RAM if possible

Tutorials

To learn how to use this package, see text2vec.org and the package vignettes. See also the text2vec articles on my blog.

Features

The core functionality at the moment includes

  1. Fast text vectorization on arbitrary n-grams, using vocabulary or feature hashing.
  2. GloVe word embeddings.
  3. Topic modeling with:
  • Latent Dirichlet Allocation
  • Latent Sematic Analysis
  1. Similarities/distances between 2 matrices

Performance

htop Author of the package is a little bit obsessed about efficiency.

This package is efficient because it is carefully written in C++, which also means that text2vec is memory friendly. Some parts (such as GloVe) are fully parallelized using the excellent RcppParallel package. This means that the word embeddings are computed in parallel on OS X, Linux, Windows, and even Solaris (x86) without any additional tuning or tricks.

Other emrassingly parallel tasks (such as vectorization) can use any parallel backend which supports foreach package. They can achieve near-linear scalability with number of available cores.

Finally, a streaming API means that users do not have to load all the data into RAM.

Contributing

The package has issue tracker on GitHub where I'm filing feature requests and notes for future work. Any ideas are appreciated.

Contributors are welcome. You can help by:

License

GPL (>= 2)