A parts-of-speech tagger using Hidden Markov Models and trained using the Brown Corpus (.zip file).
This repository also contains the files containing processed data in
brown.train contains the
whole of the Brown corpus concattenated into a single file(with some
brown.counts contains n-gram and word-tag counts
The brown.counts file is formatted as follows:
2 WORDTAG jj inward 150 3-GRAM cs dt nn
The first line means that the word "inward" is paired with the "jj" tag 2 times in the corpus.
The second line means that the trigram(3-GRAM) ["cs" "dt" "nn"] appears 150 times in the corpus. There are also lines with counts for 1- and 2-grams.
The code used to produce
brown.train files are in
The Brown corpus tagset is described here.
First start a REPL inside the project dir:
(require '[pos-tagger.hmm :refer :all]) (tag-sequence ["the" "man" "saw" "a" "dog" "."]) ;; ("at" "nn" "vbd" "at" "nn" ".")
- Account for rare words, proper nouns, typos.(smoothing)
- Try a trigram language model.
- Evaluate performance.
Copyright © 2013 Samrat Man Singh
Distributed under the Eclipse Public License either version 1.0 or (at your option) any later version.