regexp tokenizer
JavaScript
Switch branches/tags
Nothing to show
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
Readme.md
component.json
index.js

Readme.md

regexp-tokenizer

Aggressive tokenizer (from node-natural)

Installation

$ component install danzajdband/aggressive-tokenizer

Usage

var tokenizer = require('tokenizer')({discard_empty: true, pattern: /W+/});

tokenizer.tokenize('viva la pepa    ');
  //['viva', 'la', 'pepa']

Options

gaps: Boolean

gaps: Boolean

pattern: Regexp

API

RegexpTokenizer#trim

RegexpTokenizer#tokenize

License

MIT