Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Provides a simple lexer based on the StringScanner class
Ruby
Branch: master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
lib
spec
.gitignore
.rspec
.yardopts
CHANGELOG.md
Gemfile
LICENSE
README.md
Rakefile
rlex.gemspec

README.md

Rlex

Provides a simple lexer based on the StringScanner class.

The lexer was written for use with Racc, a Ruby variant of Yacc. But there is no code dependency on that project so the lexer may also be used on its own or with other packages.

  • Ignored input takes precedence over rules and keywords, so if a prefix is matched by an ignore pattern, it's ignored even if it's also a keyword or matched by a rule
  • The lexer is greedy, so if a prefix is matched by multiple rules or keywords, the lexer chooses the option consuming the most input

Documentation

Find out more about this project on GitHub, RubyGems and RubyDoc.

Installation

Install the gem using:

$ gem install rlex

Alternatively, add this line to your application's Gemfile:

gem 'rlex'

Then execute:

$ bundle

Usage

# Define behavior
lexer = Lexer.new
lexer.ignore /\s+/                   # ignore whitespace
lexer.rule :word, /\w+/              # consider any text a 'word'
lexer.keyword :if                    # treat 'if' as a special keyword
lexer.keyword :lparen, "("           # any fixed input such as parentheses
lexer.keyword :rparen, ")"           #   may be defined as keywords

# Initialize with input
lexer.start "if ( foo ) bar"         # initialize the lexer with a string

# Iterate through tokens
lexer.next_token # => Token (type = :if,     value = 'if' )
lexer.next_token # => Token (type = :lparen, value = '('  )
lexer.next_token # => Token (type = :word,   value = 'foo')
lexer.next_token # => Token (type = :rparen, value = ')'  )
lexer.next_token # => Token (type = :word,   value = 'bar')
lexer.next_token # => EOF_TOKEN

Contributing

  1. Fork it on GitHub
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Added some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request

License

Written by Rasmus Borgsmidt <rasmus@borgsmidt.dk>

Released under the MIT license: www.opensource.org/licenses/MIT

Something went wrong with that request. Please try again.