SrcLexer

SrcLexer is a simple source file lexer.

Installation

Add this line to your application's Gemfile:

gem 'src_lexer'

And then execute:

$ bundle

Or install it yourself as:

$ gem install src_lexer

Usage

lexer = SrcLexer::Lexer.new(
  ['struct', 'enum', 'true', 'false'], # kyewords
  ['{', '}', '(', ')', ',', '==', '=', ';'], # symbols
  ['"', '"'], # string literal markers
  '//', # line comment marker
  ['/*', '*/'] # multi line comment markers
)

lexer.analyze(<<-'EOS')
  // comment
  enum ID {
    First = 1,
    Second = 1.5
  }
  /* comment
     againe */
  struct Data {
    string name = "This is a name.";
    ID id;
  }
  bool b = (true==false);
EOS

lexer.pop_token # => ['enum', SrcLexer::Token.new('enum', 2, 3)]
lexer.pop_token # => [:IDENT, SrcLexer::Token.new('ID', 2, 8)]
lexer.pop_token # => ['{', SrcLexer::Token.new('{', 2, 11)]
lexer.pop_token # => [:IDENT, SrcLexer::Token.new('First', 3, 5)]
lexer.pop_token # => ['=', SrcLexer::Token.new('=', 3, 11)]
lexer.pop_token # => [:NUMBER, SrcLexer::Token.new('1', 3, 13)]
lexer.pop_token # => [',', SrcLexer::Token.new(',', 3, 14)]
lexer.pop_token # => [:IDENT, SrcLexer::Token.new('Second', 4, 5)]
lexer.pop_token # => ['=', SrcLexer::Token.new('=', 4, 12)]
lexer.pop_token # => [:NUMBER, SrcLexer::Token.new('1.5', 4, 14)]
lexer.pop_token # => ['}', SrcLexer::Token.new('}', 5, 3)]
lexer.pop_token # => ['struct', SrcLexer::Token.new('struct', 8, 3)]
lexer.pop_token # => [:IDENT, SrcLexer::Token.new('Data', 8, 10)]
lexer.pop_token # => ['{', SrcLexer::Token.new('{', 8, 15)]
lexer.pop_token # => [:IDENT, SrcLexer::Token.new('string', 9, 5)]
lexer.pop_token # => [:IDENT, SrcLexer::Token.new('name', 9, 12)]
lexer.pop_token # => ['=', SrcLexer::Token.new('=', 9, 17)]
lexer.pop_token # => [:STRING, SrcLexer::Token.new('"This is a name."', 9, 19)]
lexer.pop_token # => [';', SrcLexer::Token.new(';', 9, 36)]
lexer.pop_token # => [:IDENT, SrcLexer::Token.new('ID', 10, 5)]
lexer.pop_token # => [:IDENT, SrcLexer::Token.new('id', 10, 8)]
lexer.pop_token # => [';', SrcLexer::Token.new(';', 10, 10)]
lexer.pop_token # => ['}', SrcLexer::Token.new('}', 11, 3)]
lexer.pop_token # => [:IDENT, SrcLexer::Token.new('bool', 12, 3)]
lexer.pop_token # => [:IDENT, SrcLexer::Token.new('b', 12, 8)]
lexer.pop_token # => [:IDENT, SrcLexer::Token.new('=', 12, 10)]
lexer.pop_token # => ['(', SrcLexer::Token.new('(', 12, 12)]
lexer.pop_token # => ['true', SrcLexer::Token.new('true', 12, 13)]
lexer.pop_token # => ['==', SrcLexer::Token.new('==', 12, 17)]
lexer.pop_token # => ['false', SrcLexer::Token.new('false', 12, 19)]
lexer.pop_token # => [')', SrcLexer::Token.new(')', 12, 24)]
lexer.pop_token # => [';', SrcLexer::Token.new(';', 12, 25)]
lexer.pop_token # => SrcLexer::Lexer::END_TOKEN

Contributing

1. Fork it ( http://github.com//src_lexer/fork )
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request