Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TNTLexer crashes with input '0' #1568

Closed
kurtmckee opened this issue Oct 9, 2020 · 3 comments · Fixed by #1570
Closed

TNTLexer crashes with input '0' #1568

kurtmckee opened this issue Oct 9, 2020 · 3 comments · Fixed by #1570
Milestone

Comments

@kurtmckee
Copy link
Contributor

While working on #1425 I used Hypothesis to help me identify JSON lexing issues.

Hypothesis immediately reported a crash bug in the TNT lexer. It crashes with an IndexError if you pass 0 as the input string:

>>> import pygments.lexers
>>> list(pygments.lexers.TNTLexer().get_tokens('0'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Kurt\Documents\dev\pygments\pygments\lexer.py", line 188, in streamer
    for _, t, v in self.get_tokens_unprocessed(text):
  File "C:\Users\Kurt\Documents\dev\pygments\pygments\lexers\tnt.py", line 215, in get_tokens_unprocessed
    start = end = self.formula(start, text)
  File "C:\Users\Kurt\Documents\dev\pygments\pygments\lexers\tnt.py", line 107, in formula
    if text[start] in '[]':  # fantasy push or pop
IndexError: string index out of range
@kurtmckee
Copy link
Contributor Author

@Kenny2github and @lonetwin, would you review this when you have an opportunity? Thanks!

Related: #1414

@Kenny2github
Copy link
Contributor

0 is not valid as a string in TNT, though it shouldn't crash. Are error tokens acceptable in the output?

@kurtmckee
Copy link
Contributor Author

kurtmckee commented Oct 11, 2020 via email

Kenny2github added a commit to Kenny2github/pygments that referenced this issue Oct 12, 2020
Catch IndexErrors in each line and error the rest of the line,
leaving whatever tokens were found.
Kenny2github added a commit to Kenny2github/pygments that referenced this issue Oct 12, 2020
Catch IndexErrors in each line and error the rest of the line,
leaving whatever tokens were found.
@Anteru Anteru added this to the 2.7.2 milestone Oct 24, 2020
@Anteru Anteru added changelog-update Items which need to get mentioned in the changelog and removed changelog-update Items which need to get mentioned in the changelog labels Oct 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants