Java library implementing Byte-Pair Encoding Tokenization

This project implements a tokenizer based on the Byte Pair Encoding (BPE) algorithm, with additional custom tokenizers, including one similar to the GPT-4 tokenizer.

A lightweight, from-scratch implementation of Byte Pair Encoding (BPE) tokenization in Python.

[Rust] Unofficial implementation of "SuperBPE: Space Travel for Language Models" in Rust

A python package to build a corpus vocabulary using the byte pair methodology and also a tokenizer to tokenize input texts based on the built vocab.

Byte Pair Encoding (BPE)

Strings Tokenization with Byte Pair Encoding (BPE).

Natural Language Processing course assignments @ University of Tehran

Byte Pair Encoding (BPE) in the Zig programming language (0.13.0)

self made byte-pair-encoding tokenizer

This repository is reimplementation of Transformer model which was introduced in 2017 NeurIPS paper "Attention is all you need"

implementation of BPE algorithm and training of the tokens generated

A web app to compare pre-built or self-built tokenizers

tokenizer for large-scale language models (GPT, Claude, Llama, etc.)

TF-IDF Calculation

Byte-pair Algorithm used in GPT Tokenization based on the paper "Language Model are unsupervised multitask learners"

A pure Python implementation of Byte Pair Encoding (BPE) tokenizer. Train on any text, encode/decode with saved models, and explore BPE tokenization fundamentals.