Tokenizer

Simple string tokenizer

Installation:

Note: python 3 and cmake is required

1. Clone the repo

git clone https://github.com/berazo29/tokenizer.git

2. Go inside the project

cd tokenizer

3. Build the project with the python script

python build.py

Usage

Inside the folder build there is the executable tokenizer, you can pass any string as argument.

Example:

./tokenizer 123+++=+

Output:

Number 1
Number 2
Number 3
increment: "++"
plus equals: "+="
addition: "+"

License

Distributed under the MIT License. See LICENSE.txt for more information.

(back to top)