Skip to content


Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
branch: master
Fetching contributors…

Cannot retrieve contributors at this time

75 lines (52 sloc) 2.234 kb


Provides a simple lexer based on the StringScanner class.

The lexer was written for use with Racc, a Ruby variant of Yacc. But there is no code dependency on that project so the lexer may also be used on its own or with other packages.

  • Ignored input takes precedence over rules and keywords, so if a prefix is matched by an ignore pattern, it's ignored even if it's also a keyword or matched by a rule
  • The lexer is greedy, so if a prefix is matched by multiple rules or keywords, the lexer chooses the option consuming the most input


Find out more about this project on GitHub, RubyGems and RubyDoc.


Install the gem using:

$ gem install rlex

Alternatively, add this line to your application's Gemfile:

gem 'rlex'

Then execute:

$ bundle


# Define behavior
lexer =
lexer.ignore /\s+/                   # ignore whitespace
lexer.rule :word, /\w+/              # consider any text a 'word'
lexer.keyword :if                    # treat 'if' as a special keyword
lexer.keyword :lparen, "("           # any fixed input such as parentheses
lexer.keyword :rparen, ")"           #   may be defined as keywords

# Initialize with input
lexer.start "if ( foo ) bar"         # initialize the lexer with a string

# Iterate through tokens
lexer.next_token # => Token (type = :if,     value = 'if' )
lexer.next_token # => Token (type = :lparen, value = '('  )
lexer.next_token # => Token (type = :word,   value = 'foo')
lexer.next_token # => Token (type = :rparen, value = ')'  )
lexer.next_token # => Token (type = :word,   value = 'bar')
lexer.next_token # => EOF_TOKEN


  1. Fork it on GitHub
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Added some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request


Written by Rasmus Borgsmidt <>

Released under the MIT license:

Jump to Line
Something went wrong with that request. Please try again.