Tokenizer

Tokenizer V1.0.0

The first release of the Tokenizer class, written in PHP.

This is a simple class that allows you to extract tokens from a string using regular expressions.
Define your patterns using the add() method, and then extract the tokens with tokenize().
The tokenizer class also generates an array of tokens, identifying the name and value of each token.

This is a simple to use Lexer, one of the building blocks in creating a grammar that can be parsed and interpreted, such as mathematical expressions, or your own programming language.