Natural Language Processing (NLP) library for Crystal
Switch branches/tags
Nothing to show
Clone or download
Latest commit c118603 Oct 5, 2018
Failed to load latest commit information.
data Added sentiment analysis Mar 15, 2018
spec Fix ameba issues Sep 29, 2018
src Tidy-up the code a lil’ Sep 29, 2018
.ameba.yml [CI] Add ameba linter Sep 29, 2018
.editorconfig Initial commit Mar 11, 2018
.gitignore Initial commit Mar 11, 2018
.travis.yml [CI] Add ameba linter Sep 29, 2018
LICENSE Initial commit Mar 11, 2018 Added sentiment analysis Mar 15, 2018
shard.yml [CI] Add ameba linter Sep 29, 2018


Travis Dependencies license

Cadmium is a Natrual Language Processing (NLP) library for Crystal. Included are classes and modules for tokenizing, inflecting, stemming, and creating n-grams with much more to come.

It's still in early development, but tests are being written as I go so hopefully it will be somewhat stable.

This library is heavily based on the natural library for node.js, and as such you can expect the API's to be very similar. As a point of fact, most of the specs for Cadmium were copied directly from natural and lightly modified.

Any utilities that can be internationalized will be eventually. For now English is the primary concern.

Table of Contents


Add this to your application's shard.yml:

    github: watzon/cadmium
    branch: master


Require the cadmium library in your project

require "cadmium"


Cadmium includes several different tokenizers, each of which is useful for different applications.

Aggressive Tokenizer

The aggressive tokenizer currently has localization available for English (:en / nil), Spanish (:es), Persian (:fa), French (:fr), Indonesian (:id), Dutch (:nl), Norwegian (:no), Polish (:pl), Portuguese (:pt), Russian (:ru), and Swedish (:sv). If no language is included it will

Use it like so:

tokenizer = :es)
tokenizer.tokenize("hola yo me llamo eduardo y esudié ingeniería")
# => ["hola", "yo", "me", "llamo", "eduardo", "y", "esudié", "ingeniería"]

Case Tokenizer

The case tokenizer doesn't rely on Regex and as such should be pretty fast. It should also work on an international basis fairly easily.

tokenizer =
tokenizer.tokenize("these are strings")
# => ["these", "are", "strings"]

tokenizer = true)
tokenizer.tokenize("Affectueusement surnommé « Gabo » dans toute l'Amérique latine")
# => ["Affectueusement", "surnommé", "Gabo", "dans", "toute", "l", "Amérique", "latine"]

Regex Tokenizer

The whitespace tokenizer, word punctuation tokenizer, and word tokenizer all extend the regex tokenizer. It uses Regex to match on the correct values.

tokenizer =
tokenizer.tokenize("my dog hasn't any fleas.")
# => ["my", "dog", "hasn", "'", "t", "any", "fleas", "."]

Treebank Word Tokenizer

The treebank tokenizer uses regular expressions to tokenize text as in Penn Treebank. This implementation is a port of the tokenizer sed script written by Robert McIntyre. To read about treebanks you can visit wikipedia.

tokenizer =
tokenizer.tokenize("If we 'all' can't go. I'll stay home.")
# => ["If", "we", "'all", "'", "ca", "n't", "go.", "I", "'ll", "stay", "home", "."]

String Distance

Corundum provides an implimentation of two different string distance algorithms, the Jaro-Winkler Distance Algorithm and the Levenshtein Distance Algorithm.


The Jaro-Winkler algorithm returns a number between 0 and 1 which tells how closely two strings match (1 being perfect and 0 being not at all).

# => 0.8133333333333332

# => 1

# => 0.0


The Levenshtein distance algorithm returns the number of edits (insertions, modifications, or deletions) required to transform one string into another.

Cadmium.levenshtein_distance("doctor", "doktor")
# => 1

Cadmium.levenshtein_distance("doctor", "doctor")
# => 0

Cadmium.levenshtein_distance("flad", "flaten")
# => 3


Currently Cadmium only comes with a Porter Stemmer, but Lancaster will be added soon. Stemmer methods stem and tokenize_and_stem have also been added to the String class to simplify use.

# => word

"i am waking up to the sounds of chainsaws".tokenize_and_stem
# => ["wake", "sound", "chainsaw"]


Phonetic matching (sounds-like) matching can be done with the SoundEx or Metaphone algorithms

soundex = Cadmium::Phonetics::SoundEx
metaphone = Cadmium::Phonetics::Metaphone

# => "P532"

soundex.tokenize_and_phoneticize("Ruby aint got nothing on Crystal")
# => ["R100", "A530", "G300", "C234"]

# Keep word stops
soundex.tokenize_and_phoneticize("Ruby aint got nothing on Crystal", true)
# => ["R100", "A530", "G300", "N352", "O000", "C234"]"phonetix", "phonetics")
# => true

# => "FNTKS"

metaphone.tokenize_and_phoneticize("Ruby aint got nothing on Crystal")
# => ["RB", "ANT", "KT", "KRSTL"]

# Keep word stops
metaphone.tokenize_and_phoneticize("Ruby aint got nothing on Crystal", true)
# => ["RB", "ANT", "KT", "N0NK", "ON", "KRSTL"]"phonetix", "phonetics")
# => true

Both classes can also be used with attached String methods. The default class for String methods is Metaphone. The attached methods are phonetics, sounds_like, and tokenize_and_phoneticize.

# => "KRSTL"

# => true

"Crystal".phonetics(nil, Cadmium::Phonetics::SoundEx)
# => "C234"

# Using a max length
"Constitution".phonetics(6, Cadmium::Phonetics::SoundEx)
# => "C52333"



Nouns can be inflected using the NounInflector which has also been attached to the String class.

inflector =

# => radii

# => radius

# => people

# => person

Present Tense Verbs

Present tense verbs can be inflected with the PresentTenseVerb inflector. This has also been attached to the string class.

inflector =

# => became

# => become

"walk".singularize(false) # noun: false
# => walks

"walks".pluralize(false)  # noun: false
# => walk


Numbers can be inflected with the CountInflector which also adds a method to_nth to the Int class.

# => 1st

# => 111th

# => 153rd


N-Grams can be obtained for Arrays of Strings, or with single Strings (which will first be tokenized).


Cadmium::NGrams.bigrams("these are some words")
# => [["these", "are"], ["are", "some"], ["some", "words"]]


Cadmium::NGrams.trigrams("these are some words")
# => [["these", "are", "some"], ["are", "some", "words"]]

arbitrary n-grams

Cadmium::NGrams.ngrams("some other words here for you", 4)
# => [["some", "other", "words", "here"], ["other", "words", "here", "for"], ["words", "here", "for", "you"]]


n-grams can also be returned with left or right padding by passing a start and/or end symbol to the bigrams, trigrams or ngrams.

Cadmium::NGrams.ngrams("these are some words", 4, "[start]", "[end]")
# => [
      ["[start]", "[start]", "[start]", "these"],
      ["[start]", "[start]", "these", "are"],
      ["[start]", "these", "are", "some"],
      ["these", "are", "some", "words"],
      ["are", "some", "words", "[end]"],
      ["some", "words", "[end]", "[end]"],
      ["words", "[end]", "[end]", "[end]"]


Term Frequency–Inverse Document Frequency (tf-idf) is implemented to determine how important a word (or words) is to a document relative to a corpus. The following example will add four documents to a corpus and determine the weight of the word "node" and then the weight of the word "ruby" in each document.

tfidf =
tfidf.add_document("this document is about crystal.")
tfidf.add_document("this document is about ruby.")
tfidf.add_document("this document is about ruby and crystal.")
tfidf.add_document("this document is about crystal. it has crystal examples")

puts "crystal --------------------------------"
tfidf.tfidfs("crystal") do |i, measure, key|
  puts "document ##{i} is #{measure}"

puts "ruby --------------------------------"
tfidf.tfidfs("ruby") do |i, measure, key|
  puts "document ##{i} is #{measure}"

# =>  node --------------------------------
      document #0 is 1
      document #1 is 0
      document #2 is 1
      document #3 is 2
      ruby --------------------------------
      document #0 is 0
      document #1 is 1.2876820724517808
      document #2 is 1.2876820724517808
      document #3 is 0


The Transliterator module provides the ability to transliterate UTF-8 strings into pure ASCII so that they can be safely displayed in URL slugs or file names.

# => "Privet"

# => "Ni Hao Peng You"

# With the string extension

"މިއަދަކީ ހދ ރީތި ދވހކވ".transliterate
# => "mi'adhakee hdh reethi dhvhkv"

# => konnichiwa, You Ren

Sentiment Analysis

The Sentiment module uses the AFINN-165 wordlist and Emoji Sentiment Ranking to provide sentiment analysis on arbitrary blocks of text.

sentiment = Cadmium::Sentiment

"Crystal is seriously the best programming language.".sentiment
# or
sentiment.analyze("Crystal is seriously the best programming language.")
# =>  {
        score: 3,
        comparative: 0,
        tokens: ["Crystal", "is", "seriously", "the", "best", "programming", "language"],
        words: ["best"],
        positive: ["best"],
        negative: []

"I really hate Python".is_negative?
# => true

"I really 💗 Crystal. It's my favorite.".is_positive?
# => true


This is all I want to have done before a v1.0 release.

  • Tokenizers
    • AggressiveTokenizer
      • i18n
    • CaseTokenizer
    • RegexTokenizer
    • SentenceTokenizer
    • TreebankWordTokenizer
    • WhitespaceTokenizer
    • WordPunctuationTokenizer
  • String Distance
    • Levenshein
      • Approximate String Matching
    • JaroWinkler
  • Stemmers
    • PorterStemmer
      • i18n
    • LancasterStemmer
      • i18n
  • Classifiers
    • Bayes
    • Logic Regression
  • Phonetics
    • SoundEx
    • Metaphone
    • Double Metaphone
  • Inflectors
    • Count
    • Noun
    • Verb
    • i18n
  • N-Grams
  • TF-IDF
  • Transliterator
  • Sentiment Analysis
  • Tries
  • EdgeWeightedDigraph
  • ShortestPathTree
  • LongestPathTree
  • WordNet
  • Spellcheck
  • POS Tagger


  1. Fork it ( )
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create a new Pull Request


  • watzon Chris Watson - creator, maintainer