Name		Name	Last commit message	Last commit date
Latest commit History 9 Commits
Makefile		Makefile
README.md		README.md
setup.py		setup.py
tok.pxd		tok.pxd
tok.pyx		tok.pyx
tok_test.py		tok_test.py
tokenizer.h		tokenizer.h
tokenizer.re		tokenizer.re

Repository files navigation

RE2C tokenizer

The simple re2c based tokenizer.

Recognizes tokens:

words
numbers
URLs
obfuscated words (like F%$k)

How to install

make
python setup.py install

About

Simple tokenizer based on re2c for NLP tasks

2 stars

0 watching

Report repository

Releases

No releases published

Packages

Contributors

Languages