Web tool to count LLM tokens (GPT, Claude, Llama, ...)
-
Updated
Jun 4, 2024 - TypeScript
A grammar describes the syntax of a programming language, and might be defined in Backus-Naur form (BNF). A lexer performs lexical analysis, turning text into tokens. A parser takes tokens and builds a data structure like an abstract syntax tree (AST). The parser is concerned with context: does the sequence of tokens fit the grammar? A compiler is a combined lexer and parser, built for a specific grammar.
Web tool to count LLM tokens (GPT, Claude, Llama, ...)
Parser Building Toolkit for JavaScript
ZephyrAssets is a blockchain-based platform specializing in the tokenization of real-world assets. Leveraging smart contracts and blockchain technology, we provide a secure and transparent way to digitize, manage, and trade physical assets such as real estate, art, and collectibles
Play with OpenAI API's using your own API Key. Your API Key is stored and used only from your browser.
📐 GPT token estimation and context size utilities without a full tokenizer
Parse any kind of broken json for scrapping easily
A JSON parser made in typescript with tokenizer, ast, parser ...
Contains source-code for viewers following along with my Beginners Guide To Building Interpreters series on my Youtube Channel.
VSCode extension to highlight nested code blocks
React component to convert a text string into React Nodes
Javascript port of HappyFunTokenizer.py by Christopher Potts and HappierFunTokenizing.py by H. Andrew Schwartz