-
Notifications
You must be signed in to change notification settings - Fork 627
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
TNTLexer: Don't crash on unexpected EOL. #1570
Conversation
Catch IndexErrors in each line and error the rest of the line, leaving whatever tokens were found.
Force pushed to just the one commit (the others are already in the repo) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add a test case for this (it should crash without the change applied.)
(Seen 4 hours ago, been writing tests for the whole lexer ever since) |
pygments/lexers/tnt.py: * Fix indentation on import * Fix: TNTLexer.cur is class-level reference if not initialized in get_tokens_unprocessed, so init it in __init__ too * Fix: Fantasy markers are not allowed as components of other formulas, so have a dedicated check for them in the body of get_tokens_unprocessed which disables the normal formula handling if present * Clarify TNTLexer.lineno docstring * Attempt to discard tokens before an IndexError +tests/test_tnt.py: * Test every method, and test both +ve and -ve matches for most * Lexer fixture is test-level to reinitialize cur clean each time * Don't test actual get_tokens_unprocessed method (besides for fantasy markers) because the text testing is left to examplefiles AUTHORS: + Add myself to credits :)
That looks fairly comprehensive, thanks! I'll wait for the tests to finish, then merge -- thanks for your contribution. |
Glad to be of service! Sorry for the original issue I caused. At least I got good tests in :) |
Closes #1568 which is blocking #1425