UAT es un tokenizer aritmético básico con funciones para determinar errores, separar tokens por tipos y preparar strings para conversiones a infix, etc.
-
Updated
Jul 4, 2022 - C++
UAT es un tokenizer aritmético básico con funciones para determinar errores, separar tokens por tipos y preparar strings para conversiones a infix, etc.
A super string class for C++
Implementation of C++ lexical analyzer to demonstrate how it actually works as a part of the compiler.
experimental programming language made with cpp
Lexical Analyzer for C made in C++
R package for Byte Pair Encoding based on YouTokenToMe
Includes coursework and lab materials for students enrolled in the Bachelor of Science in Computer Science degree at UBIT.
This project is a fully functional compiler for the TINY programming language, which is a language that supports basic arithmetic, boolean, and control flow operations. The compiler can scan, parse, and run code written in the TINY language.
🔧 Demonstration of using ANTLR4 (with runtime for C++) in projects for context-free grammar processing. The ANTLR4 (Java) package is included, and the project is configured to compile on Linux.
Developed Console based API takes reviews from various mobile companies and compares the different aspects of the mobile phone using complex data structures concepts.
Originally a semester project for VT3574 Applied Software Design
A MIPS assembler that converts MIPS Assembly Language into MIPS Machine Language. Implemented DFA algorithms, Parsing, and Semantic Analysis.
tf-idf (Term-Frequency Inverse-Document-Frequency) C++ Library
Code example for the Fundamentals of Programming course at the University of Guilan, Fall 2016.
Tokenizers and Machine Learning Models for biological sequence data
Smart Language Model
The Libre Multilingual Analyzer, a Natural Language Processing (NLP) C++ toolkit.
Fast and customizable text tokenization library with BPE and SentencePiece support
Unsupervised text tokenizer focused on computational efficiency
Add a description, image, and links to the tokenization topic page so that developers can more easily learn about it.
To associate your repository with the tokenization topic, visit your repo's landing page and select "manage topics."