Simple python lib to tokenize texts into sentences and sentences to words. Small, fast and robust. Comes with ukrainian flavour
Python Makefile
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
docs
tests
tokenize_uk
.editorconfig
.gitignore
.travis.yml
AUTHORS.rst
CONTRIBUTING.rst
HISTORY.rst
LICENSE
MANIFEST.in
Makefile
README.rst
requirements_dev.txt
setup.cfg
setup.py
tox.ini
travis_pypi_setup.py

README.rst

Tokenize UK

Documentation Status

Simple python lib to tokenize texts into sentences and sentences to words. Small, fast and robust. Comes with ukrainian flavour

Features

  • Tokenize given text into sentences
  • Tokenize given sentence into words
  • Works well with accented characters (like stresses) and apostrophes
  • Suitable also for other languages