tokenator: a Node.js tokenization library

tokenator is a simple to use library that rework a string splitting it to meaningful tokens.

Table of Contents

• Description
• Installation
• Code Examples
• Changelog
• License

Description

This library transforms a series of characters to tokens identified by input strings.

e.g.
"The itsy bitsy spider crawled up the water spout. | Down came the rain, and washed the spider out. | Out came the sun, and dried up all the rain, | and the itsy bitsy spider went up the spout again"

Splitting strings: spider, rain

Result tokens:
  > "The itsy bitsy "
  > "spider"
  > " crawled up the water spout. | Down came the "
  > "rain"
  > ", and washed the "
  > "spider"
  > " out. | Out came the sun, and dried up all the "
  > "rain"
  > ", | and the itsy bitsy "
  > "spider"
  > " went up the spout again"

Installation

npm install tokenator

Code Examples

The signatures of the available methods are:

function tokenator(source: string, ...separators: string[]): string[]
function tokenatorStream(...separators: string[]): (separatorsEncoding: string = "utf-8") => stream.Transform

Code examples could be consulted within the test folder:

• tokenator examples
• tokenatorStream examples

These functions work over multiple separators, like in the example above. The tokenatorStream() function creates a Transform stream. NodeJS Stream usage specification is available on the NodeJS official documentation.

Changelog

Available here

License

MIT

Available here