Skip to content

A Lexer Module for Compiler Construction

Choose a tag to compare

@javiergs javiergs released this 10 Mar 02:42
· 135 commits to main since this release

This release introduces a Lexical Analyzer (Lexer) component, designed to support projects in compiler construction
and programming language processing
.

The lexer module provides functionality to scan input source code and convert it into a stream of tokens, which can then be consumed by a parser or other language processing tools.

New Package

tulip.compiler

The new lexer module allows applications to:

  • Read source code from text input
  • Identify lexical elements such as:
    • identifiers
    • keywords
    • numbers
    • operators
    • delimiters
  • Produce a structured stream of tokens for use in parsing stages
  • Support experimentation with language design and compiler
    construction

Example Usage

    Lexer lexer = new Lexer(sourceCode);
    lexer.run();
    Vector<Token> tokens = lexer.getTokens();
    for (Token token : tokens) {
       System.out.println(token);
    }