Permalink
Browse files

SimpleLexer now stops at Id.Eol_Tok instead of checking the position.

The makes the echo -e/glob lexers more consistent with the OSH lexer.
  • Loading branch information...
Andy Chu
Andy Chu committed Jun 24, 2018
1 parent c6eb308 commit 294f54fb3fad970d3dafa02dc7f6bd808922100b
Showing with 4 additions and 6 deletions.
  1. +4 −6 osh/match.py
View
@@ -81,13 +81,11 @@ def __init__(self, match_func):
def Tokens(self, line):
"""Yields tokens."""
pos = 0
# NOTE: We're not using Eol_Tok like LineLexer. We probably should. And
# then the consumers of the ECHO_E_DEF and GLOB_DEF should use it. Get rid
# of Glob_Eof.
n = len(line)
while pos != n:
# NOTE: Need longest-match semantics to find \377 vs \.
while True:
tok_type, end_pos = self.match_func(line, pos)
# core/lexer_gen.py always inserts this. We're always parsing lines.
if tok_type == Id.Eol_Tok:
break
yield tok_type, line[pos:end_pos]
pos = end_pos

0 comments on commit 294f54f

Please sign in to comment.