XML Tokenizer is a low-memory high performance library for parsing simple XML 1.0.
-
Updated
Jun 18, 2024 - Go
A grammar describes the syntax of a programming language, and might be defined in Backus-Naur form (BNF). A lexer performs lexical analysis, turning text into tokens. A parser takes tokens and builds a data structure like an abstract syntax tree (AST). The parser is concerned with context: does the sequence of tokens fit the grammar? A compiler is a combined lexer and parser, built for a specific grammar.
XML Tokenizer is a low-memory high performance library for parsing simple XML 1.0.
Self-contained Japanese Morphological Analyzer written in pure Go
NLP tokenizers written in Go language
OpenShield is a firewall designed for AI models.
A Text Tokenizer library for Golang
The Cidoka Programming Language
A multilingual command line sentence tokenizer in Golang
Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript
Text to tokes