Self-contained Japanese Morphological Analyzer written in pure Go
-
Updated
May 8, 2024 - Go
A grammar describes the syntax of a programming language, and might be defined in Backus-Naur form (BNF). A lexer performs lexical analysis, turning text into tokens. A parser takes tokens and builds a data structure like an abstract syntax tree (AST). The parser is concerned with context: does the sequence of tokens fit the grammar? A compiler is a combined lexer and parser, built for a specific grammar.
Self-contained Japanese Morphological Analyzer written in pure Go
Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript
A multilingual command line sentence tokenizer in Golang
Lex machinary for go.
NLP tokenizers written in Go language
Object mapping for golang.
A tokenizer based on the dictionary and Bigram language models for Go. (Now only support chinese segmentation)
Package assocentity returns the mean distance from tokens to an entity and its synonyms
High-performance, real-time, multi-location data obfuscation tool
Natural Language Tokenizer
A Text Tokenizer library for Golang