Tokenizers and lemmatizers for Go
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
cmd/jargon
contractions
numbers
stackexchange
testdata
web
.gitignore
LICENSE.txt
README.md
dictionary.go
go.mod
go.sum
lemmatizer.go
lemmatizer_test.go
punct.go
punct_test.go
token.go
tokenizers.go
tokenizers_test.go

README.md

Jargon

Jargon is a lemmatizer, useful for recognizing variations on canonical and synonymous terms.

For example, jargon lemmatizes react, React.js, React JS and REACTJS to a canonical reactjs.

Jargon uses Stack Overflow tags & synonyms, and implements “insensitivity” to spaces, dots and dashes.

Online demo

Give it a try

Command line

go install github.com/clipperhouse/jargon/cmd/jargon

(Assumes a Go installation.)

To display usage, simply type:

jargon
jargon accepts piped UTF8 text from Stdin and pipes lemmatized text to Stdout

  Example: echo "I luv Rails" | jargon

Alternatively, use jargon 'standalone' by passing flags for inputs and outputs:

  -f string
    	Input file path
  -o string
    	Output file path
  -s string
    	A (quoted) string to lemmatize
  -u string
    	A URL to fetch and lemmatize

  Example: jargon -f /path/to/original.txt -o /path/to/lemmatized.txt

In your code

See GoDoc.

Dictionaries

Canonical terms (lemmas) are looked up in dictionaries. Three are available:

To implement your own, see the jargon.Dictionary interface

Tokenizer

Jargon includes its own tokenizer, with an emphasis on handling technology terms correctly:

  • C++, ASP.net, and other non-alphanumeric terms are recognized as single tokens
  • #hashtags and @handles
  • Simple URLs and email address are handled pretty well, though can be notoriously hard to get right

The tokenizer preserves all tokens verbatim, including whitespace and punctuation, so the original text can be reconstructed with fidelity (“round tripped”).

(It turns out that the above rules work well in structured text such as CSV and JSON.)

Background

When dealing with technology terms in text – say, a job listing or a resume – it’s easy to use different words for the same thing. This is acute for things like “react” where it’s not obvious what the canonical term is. Is it React or reactjs or react.js?

This presents a problem when searching for such terms. We know the above terms are synonymous but databases don’t.

A further problem is that some n-grams should be understood as a single term. We know that “Objective C” represents one technology, but databases naively see two words.

Prior art

Existing tokenizers (such as Treebank), appear not to be round-trippable, i.e., are destructive. They also take a hard line on punctuation, so “ASP.net” would come out as two tokens instead of one. Of course I’d like to be corrected or pointed to other implementations.

Search-oriented databases like Elastic handle synonyms with analyzers.

In NLP, it’s handled by stemmers or lemmatizers. There, the goal is to replace variations of a term (manager, management, managing) with a single canonical version.

Recognizing mutli-words-as-a-single-term (“Ruby on Rails”) is named-entity recognition.

What’s it for?

  • Recognition of domain terms in text
  • NLP for unstructured data, when we wish to ensure consistency of vocabulary, for statistical analysis.
  • Search applications, where searches for “Ruby on Rails” are understood as an entity, instead of three unrelated words, or to ensure that “React” and “reactjs” and “react.js” and handled synonmously.