Implementation of the singlestate and multistate lexers
Switch branches/tags
Nothing to show
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
src/Lexer Apply fixes from StyleCI Dec 7, 2018
tests Apply fixes from StyleCI Dec 7, 2018
.editorconfig Git Subtree: Apply fixes from StyleCI Nov 17, 2018
.gitattributes Change antlr language to ebnf Dec 8, 2018
.gitignore
.scrutinizer.yml Initial Jun 7, 2018
.travis.yml Add trailing composer json eol Dec 7, 2018
CODE_OF_CONDUCT.md Git Subtree: Apply fixes from StyleCI Nov 17, 2018
CONTRIBUTING.md Merge branch 'master' of github.com:railt/railt Dec 6, 2018
LICENSE.md Initial Jun 7, 2018
README.md
composer.json Add trailing composer json eol Dec 7, 2018
phpunit.xml

README.md

Railt

Travis CI Scrutinizer CI Latest Stable Version Latest Unstable Version License MIT

Lexer

Note: All questions and issues please send to https://github.com/railt/railt/issues

In order to quickly understand how it works - just write ~4 lines of code:

$lexer = Railt\Lexer\Factory::create(['T_WHITESPACE' => '\s+', 'T_DIGIT' => '\d+'], ['T_WHITESPACE']);

foreach ($lexer->lex(Railt\Io\File::fromSources('23 42')) as $token) {
    echo $token . "\n";
}

This example will read the source text and return the set of tokens from which it is composed:

  1. T_DIGIT with value "23"
  2. T_DIGIT with value "42"

The second argument to the Factory class is the list of token names that are ignored in the lex method result. That's why we only got two significant tokens T_DIGIT. Although this is not entirely true, the answer contains a T_EOI (End Of Input) token which can also be removed from the output by adding an array of the second argument of Factory class.

...and now let's try to understand more!

The lexer contains two types of runtime:

  1. Basic - Set of algorithms with one state.
  2. Multistate - Set of algorithms with the possibility of state transition between tokens.

In connection with the fact that there were almost no differences in speed between several implementations (Stateful vs Stateless) of the same algorithm, it was decided to abandon the immutable stateful lexers.

use Railt\Lexer\Factory;

/**
 * List of available tokens in format "name => pcre"
 */
$tokens = ['T_DIGIT' => '\d+', 'T_WHITESPACE' => '\s+'];

/**
 * List of skipped tokens
 */
$skip   = ['T_WHITESPACE'];

/**
 * Options:
 *   0 - Nothing.
 *   2 - With PCRE lookahead support.
 *   4 - With multistate support.
 */
$flags = Factory::LOOKAHEAD | Factory::MULTISTATE;

/**
 * Create lexer and tokenize sources. 
 */
$lexer = Factory::create($tokens, $skip, $flags);

In order to tokenize the source text, you must use the method ->lex(...), which returns iterator of the TokenInterface objects.

foreach ($lexer->lex(File::fromSources('23 42')) as $token) {
    echo $token . "\n";
}

A TokenInterface provides a convenient API to obtain information about a token:

interface TokenInterface
{
    public function getName(): string;
    public function getOffset(): int;
    public function getValue(int $group = 0): ?string;
    public function getGroups(): iterable;
    public function getBytes(): int;
    public function getLength(): int;
}

Drivers

The factory returns one of the available implementations, however you can create it yourself.

Basic

NativeRegex

NativeRegex implementation is based on the built-in php PCRE functions.

use Railt\Lexer\Driver\NativeRegex;
use Railt\Io\File;

$lexer = new NativeRegex(['T_WHITESPACE' => '\s+', 'T_DIGIT' => '\d+'], ['T_WHITESPACE', 'T_EOI']);

foreach ($lexer->lex(File::fromSources('23 42')) as $token) {
    echo $token->getName() . ' -> ' . $token->getValue() . ' at ' . $token->getOffset() . "\n";
}

// Outputs:
// T_DIGIT -> 23 at 0
// T_DIGIT -> 42 at 3

Lexertl

Experimental lexer based on the C++ lexertl library. To use it, you need support for Parle extension.

use Railt\Lexer\Driver\ParleLexer;
use Railt\Io\File;

$lexer = new ParleLexer(['T_WHITESPACE' => '\s+', 'T_DIGIT' => '\d+'], ['T_WHITESPACE', 'T_EOI']);

foreach ($lexer->lex(File::fromSources('23 42')) as $token) {
    echo $token->getName() . ' -> ' . $token->getValue() . ' at ' . $token->getOffset() . "\n";
}

// Outputs:
// T_DIGIT -> 23 at 0
// T_DIGIT -> 42 at 3

Be careful: The library is not fully compatible with the PCRE regex syntax. See the official documentation.

Multistate

This functionality is not yet implemented.