Browse files

Fixed lexing of character syntax like ?z

Character syntax `?z` was lexed as an integer token `"zz"`.  This has
been fixed to lex as a single character string `"z"`.  While this won't
round-trip, it is acceptable for syntax highlighting.

This was discovered by Xavier Noria (@fxn) in rails/rails#7172.
  • Loading branch information...
1 parent fa9d260 commit 0c2a543a96143127e324f592f86d8688414ea81c @drbrain drbrain committed Aug 2, 2012
Showing with 14 additions and 6 deletions.
  1. +2 −0 History.rdoc
  2. +1 −6 lib/rdoc/ruby_lex.rb
  3. +11 −0 test/test_rdoc_ruby_lex.rb
View
2 History.rdoc
@@ -38,6 +38,8 @@
by Zachary Scott.
* Comment-defined methods ahead of define_method are now discovered. Issue
#133 by eclectic923
+ * Fixed lexing of character syntax (<code>?x</code>). Reported by Xavier
+ Noria.
=== 3.12 / 2011-12-15
View
7 lib/rdoc/ruby_lex.rb
@@ -536,13 +536,8 @@ def lex_init()
@lex_state = EXPR_BEG;
Token(TkQUESTION)
else
- str = ch
- if ch == '\\'
- str << read_escape
- end
@lex_state = EXPR_END
- str << (ch.respond_to?(:ord) ? ch.ord : ch[0])
- Token(TkINTEGER, str)
+ Token(TkSTRING, ch)
end
end
end
View
11 test/test_rdoc_ruby_lex.rb
@@ -30,6 +30,17 @@ def test_class_tokenize
assert_equal expected, tokens
end
+ def test_class_tokenize_character_literal
+ tokens = RDoc::RubyLex.tokenize "?\\", nil
+
+ expected = [
+ @TK::TkSTRING.new( 0, 1, 0, "\\"),
+ @TK::TkNL .new( 2, 1, 2, "\n"),
+ ]
+
+ assert_equal expected, tokens
+ end
+
def test_class_tokenize_def_heredoc
tokens = RDoc::RubyLex.tokenize <<-'RUBY', nil
def x

0 comments on commit 0c2a543

Please sign in to comment.