Skip to content
#

tokenizer

Star

A grammar describes the syntax of a programming language, and might be defined in Backus-Naur form (BNF). A lexer performs lexical analysis, turning text into tokens. A parser takes tokens and builds a data structure like an abstract syntax tree (AST). The parser is concerned with context: does the sequence of tokens fit the grammar? A compiler is a combined lexer and parser, built for a specific grammar.

Here are 3 public repositories matching this topic...

AayushSameerShah / Neural-Net-Zero-to-Hero-with-Andrej

Star 82

This repository contains the collection of explorative notebooks pure in python and in the language that we, humans can read. Have tried to compile all lectures from the Andrej Karpathy's 💎 playlist on Neural Networks - which we will end up with building GPT.

deep-learning neural-network tokenizer pytorch transformer gpt nlg pytorch-tutorial educational-project andrej-karpathy pytorch-implementation zero-to-mastery neural-network-from-scratch zero-to-hero nanogpt gpt-from-scratch
  • Updated Apr 30, 2024
  • Jupyter Notebook

Aalaa4444 / Text_Processing-and-Unique_Word_Extraction_fromHTML

Star 0

Extract text content from an HTML page, process it, and extract unique words from the processed text. This notebook utilizes various text processing techniques including cleaning, normalization, tokenization, lemmatization or stemming, and stop words removal.

tokenizer text-extraction requests data-extraction beautifulsoup text-processing tokenization stemming lemmatization stopwords-removal text-cleaning text-normalization extract-html text-tokenization text-lemmatization
  • Updated Apr 5, 2024
  • Jupyter Notebook
Followers
10.1k followers
Wikipedia
Wikipedia

Related Topics

antlr generated-parser language ll lookahead lr-parser semantic-analysis symbol-table yacc