Repository navigation
A Lexer Module for Compiler Construction
This release introduces a Lexical Analyzer (Lexer) component, designed to support projects in compiler construction
and programming language processing.
The lexer module provides functionality to scan input source code and convert it into a stream of tokens, which can then be consumed by a parser or other language processing tools.
New Package
tulip.compiler
The new lexer module allows applications to:
- Read source code from text input
- Identify lexical elements such as:
- identifiers
- keywords
- numbers
- operators
- delimiters
- Produce a structured stream of tokens for use in parsing stages
- Support experimentation with language design and compiler
construction
Example Usage
Lexer lexer = new Lexer(sourceCode);
lexer.run();
Vector<Token> tokens = lexer.getTokens();
for (Token token : tokens) {
System.out.println(token);
}