Skip to content

Commit

Permalink
Improve tokenising of numbers in Elixir lexer (#1225)
Browse files Browse the repository at this point in the history
The Elixir lexer does not include sophisticated rules for tokenising
numbers. This commit adds rules for distinguishing between integer,
float, hexadecimal, octal and binary representations.
  • Loading branch information
pyrmont committed Jun 28, 2019
1 parent f1125f3 commit 51dceea
Show file tree
Hide file tree
Showing 2 changed files with 12 additions and 2 deletions.
7 changes: 5 additions & 2 deletions lib/rouge/lexers/elixir.rb
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,11 @@ class Elixir < RegexLexer
rule %r/[a-zA-Z_!]\w*[!\?]?/, Name
rule %r{::|[%(){};,/\|:\\\[\]]}, Punctuation
rule %r/@[a-zA-Z_]\w*|&\d/, Name::Variable
rule %r{\b(0[xX][0-9A-Fa-f]+|\d(_?\d)*(\.(?![^\d\s])
(_?\d)*)?([eE][-+]?\d(_?\d)*)?|0[bB][01]+)\b}x, Num
rule %r{\b\d(_?\d)*(\.(?![^\d\s])(_?\d)+)([eE][-+]?\d(_?\d)*)?\b}, Num::Float
rule %r{\b0x[0-9A-Fa-f](_?[0-9A-Fa-f])*\b}, Num::Hex
rule %r{\b0o[0-7](_?[0-7])*\b}, Num::Oct
rule %r{\b0b[01](_?[01])*\b}, Num::Bin
rule %r{\b\d(_?\d)*\b}, Num::Integer

mixin :strings
mixin :sigil_strings
Expand Down
7 changes: 7 additions & 0 deletions spec/visual/samples/elixir
Original file line number Diff line number Diff line change
@@ -1,3 +1,10 @@
# numbers
291 # integer
0b100100011 # binary
0o443 # octal
0x123 # hexadecimal
291.0 # float

# keywords with no following whitespace
(cond)
(do)
Expand Down

0 comments on commit 51dceea

Please sign in to comment.