Parser Building Toolkit for JavaScript
-
Updated
Jul 5, 2024 - TypeScript
A grammar describes the syntax of a programming language, and might be defined in Backus-Naur form (BNF). A lexer performs lexical analysis, turning text into tokens. A parser takes tokens and builds a data structure like an abstract syntax tree (AST). The parser is concerned with context: does the sequence of tokens fit the grammar? A compiler is a combined lexer and parser, built for a specific grammar.
Parser Building Toolkit for JavaScript
VSCode extension to highlight nested code blocks
Contains source-code for viewers following along with my Beginners Guide To Building Interpreters series on my Youtube Channel.
Isomorphic JavaScript/TypeScript Tokenizer for GPT-3 and Codex Models by OpenAI.
An extension for the Tiptap editor, enabling the annotation of text. Comes with support for overlapping annotations, useful for e.g. NLP tokenization.
🐉 JavaScript expression tokenizer and evaluator
📐 GPT token estimation and context size utilities without a full tokenizer
A simple lexer written entirely in JavaScript, with no dependencies!
A primitive token library used to read from, and to write a node Buffer
n8n node for working with BPE Tokens with GPT in mind.
Text input augmented with Lexer magic✨ to support advanced search features.
Fast & Non-regex based lexical analysis for Deno.