Skip to content

Commit

Permalink
fix: don't raise UnexpectedEOF if peek_until without patterns
Browse files Browse the repository at this point in the history
  • Loading branch information
vberlier committed Jun 24, 2021
1 parent 678f597 commit bcee95a
Showing 1 changed file with 20 additions and 1 deletion.
21 changes: 20 additions & 1 deletion tokenstream/stream.py
Original file line number Diff line number Diff line change
Expand Up @@ -578,13 +578,32 @@ def peek_until(self, *patterns: TokenPattern) -> Iterator[Token]:
'world'
';'
' foo'
The method will raise and error if the end of the stream is reached
before encountering any of the given patterns.
>>> stream = TokenStream("hello world foo")
>>> with stream.syntax(word=r"[a-z]+", semi=r";"):
... for token in stream.peek_until("semi"):
... stream.expect("word").value
Traceback (most recent call last):
UnexpectedEOF: Expected semi but reached end of file
If the method is called without any pattern the iterator will
yield tokens until the end of the stream.
>>> stream = TokenStream("hello world")
>>> with stream.syntax(word=r"[a-z]+"):
... print([stream.expect("word").value for _ in stream.peek_until()])
['hello', 'world']
"""
while token := self.peek():
if token.match(*patterns):
next(self)
return
yield token
raise self.emit_error(UnexpectedEOF(patterns))
if patterns:
raise self.emit_error(UnexpectedEOF(patterns))

@overload
def collect(self) -> Iterator[Token]:
Expand Down

0 comments on commit bcee95a

Please sign in to comment.