tokenizer
A grammar describes the syntax of a programming language, and might be defined in Backus-Naur form (BNF). A lexer performs lexical analysis, turning text into tokens. A parser takes tokens and builds a data structure like an abstract syntax tree (AST). The parser is concerned with context: does the sequence of tokens fit the grammar? A compiler is a combined lexer and parser, built for a specific grammar.
Here are 46 public repositories matching this topic...
A natural language tokenizer
-
Updated
Apr 11, 2017 - C
CS214 Rutgers Systems Programming Assignment 2: Using C's file system API to produce an inverted index based on how many times words occur in files
-
Updated
Jul 2, 2017 - C
Colossal compiler to T++ language with the better memory manager to the Compiler discipline - Compiler Construction Principles and Practice by Kenneth C. Louden
-
Updated
Dec 24, 2018 - C
A simple to use, leak free tokenization replacement for strtok and strsep
-
Updated
Jan 9, 2019 - C
My first attemp to write a sqlite3 clone. It's just an experimental attempt to understand RDBMS. Never use it in any production.
-
Updated
May 6, 2019 - C
Implementation of an Deterministic Finite Automaton (DFA) in C to tokenize expressions for a simple Calculator (Lexical Analyser).
-
Updated
Feb 13, 2020 - C
CS352 Organization of Programming Languages | Tokenizer for simple math and logical expressions
-
Updated
Mar 23, 2020 - C
A programming language made in pure C
-
Updated
Oct 23, 2020 - C
CS F301 Principles of Programming Languages: Assignment-1
-
Updated
Mar 8, 2021 - C
A set of libraries that allow you to manipulate files, linked lists, sorting algorithms and much more.
-
Updated
Mar 10, 2021 - C
- Followers
- 10.2k followers
- Wikipedia
- Wikipedia