capture groups in lexers

[jneen]

Hey there.

This is awesome. I'm wondering if there's a way to get access to the groups captured by a particular rule's regex, as in String#scan. For example:

class MyLexer < RLTK::Lexer
  rule(/\.(\w+)\[/) { [:THING, $1] }
end

Currently it looks like the block is being passed only the whole match. I'm happy to patch it to be more like the core ruby matching methods, but I wanted to check first :)

[chriswailes]

Glad to hear that you've found RLTK useful. Do you mind if I ask what kind of project you are using this for?

Currently there is no functionality in the Lexer class for accessing capture groups in a rule match. This is because you would usually emit multiple tokens in the lexer and then have a rule in the parser that would produce the desired AST node.

Continuing your example, you would want to emit a PERIOD, STRING, and LBRACE token in separate rules and then have a production in your parser something like array_access -> PERIOD STRING LBRACE expression RBRACE. The advantage of this method is that it simplifies your lexer and leaves the heavy lifting to the parser. It also means that you give more information to the parser, which it can then use to disambiguate cases when parsing.

Another solution, which I'd be fine with implementing, is to make the match object available in the Lexer::Environment instance in which the productions are evaluated.

Anyway, I hope that helps.

[jneen]

Hm, okay, that kind of makes sense. I'm prototyping a shell language that relies heavily on whitespace-delimited expressions, so for example

.foo [a b c]

is two expressions, while

.foo[a b c]

is just one.

Currently the lexer has a /\s+/ rule that has no action, so if I used three different tokens, the parser would have no way of differentiating between those two. I suppose I could have a :WHITESPACE token or some such, but it's a lot simpler just to grab the whole thing and extract the name.

Exposing the match object in Lexer::Environment seems like a good solution. And if there's any way to make the perly regex pseudoglobals ($1, $@, etc) available in the action's scope, that'd be awesome too.

[chriswailes]

So I've just pushed some changes that give you access to the MatchData object inside the Lexer::Environment object through the match accessor. I'm a firm believer that global variables are evil so I'll leave exposing the pseudoglobals up to you. Remember that you can extend the Lexer::Environment class in your lexer fairly simply.

Since whitespace has semantic meaning in your language I would still recommend emitting the :WHITESPACE token. Lexers are usually used to handle syntax, while semantics are usually handled in the parser. I think if you plan on adding features to this language you'll find it easier to add them in the parser then in the lexer, and you can only do that if you're emitting :WHITESPACE tokens.

P.S. How did you come across RLTK? I'm trying to promote it better and it would be good to know how people have found it so far.

[jneen]

Good points all. I'll start emitting :WHITESPACE tokens, then. I found RLTK through a search on rubygems.org for "lexer" - it was a few items down, but it's the only project I found that seemed to be under active development.

Thanks!