Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

Fixed Issue #14. The problem was that the method that did lookahead s…

…et pruning used underscores to extend the non-terminal names and then splitting on all underscores instead of just the underscores it added.
  • Loading branch information...
commit c56fd7b6fdb6e9ced141de63b148052450ad623c 1 parent 5c5988f
@chriswailes authored
Showing with 23 additions and 2 deletions.
  1. +4 −0 README.md
  2. +1 −1  lib/rltk/parser.rb
  3. +18 −1 test/tc_parser.rb
View
4 README.md
@@ -338,6 +338,10 @@ The example below, based on one of the unit tests, shows a very basic usage of e
finalize
end
+## A Note on Token Naming
+
+In the world of RLTK both terminal and non-terminal symbols may contain only alphanumeric characters and underscores. The differences between terminal and non-terminal symbols is that terminals are **ALL\_UPPER\_CASE** and non-terminals are **all\_lower\_case**.
+
## ASTNode
The {RLTK::ASTNode} base class is meant to be a good starting point for implementing your own abstract syntax tree nodes. By subclassing {RLTK::ASTNode} you automagically get features such as tree comparison, notes, value accessors with type checking, child node accessors and `each` and `map` methods (with type checking), and the ability to retrieve the root of a tree from any member node.
View
2  lib/rltk/parser.rb
@@ -1100,7 +1100,7 @@ def prune(do_lookahead, do_precedence)
# Translate the G' follow symbols into G lookahead
# symbols.
- lookahead = lookahead.map { |sym| sym.to_s.split('_').last.to_sym }.uniq
+ lookahead = lookahead.map { |sym| sym.to_s.split('_', 2).last.to_sym }.uniq
# Here we remove the unnecessary reductions.
# If there are error productions we need to
View
19 test/tc_parser.rb
@@ -39,6 +39,10 @@ class AlphaLexer < RLTK::Lexer
rule(/\s/)
end
+ class UnderscoreLexer < RLTK::Lexer
+ rule(/\w/) { |t| [:A_TOKEN, t] }
+ end
+
class APlusBParser < RLTK::Parser
production(:a, 'A+ B') { |a, _| a.length }
@@ -166,7 +170,13 @@ class ErrorLine < RLTK::Parser
finalize
end
-
+
+ class UnderscoreParser < RLTK::Parser
+ production(:s, 'A_TOKEN+') { |o| o }
+
+ finalize(explain: 'test.table')
+ end
+
class RotatingCalc < RLTK::Parser
production(:e) do
clause('NUM') {|n| n}
@@ -434,6 +444,13 @@ def test_prefix_calc
assert_raise(RLTK::NotInLanguage) { RLTK::Parsers::PrefixCalc.parse(RLTK::Lexers::Calculator.lex('1 + 2 * 3')) }
end
+ def test_underscore_tokens
+ actual = UnderscoreParser.parse(UnderscoreLexer.lex('abc')).join
+ expected = 'abc'
+
+ assert_equal(expected, actual)
+ end
+
def test_use
tmpfile = File.join(Dir.tmpdir, 'usetest')
Please sign in to comment.
Something went wrong with that request. Please try again.