Skip to content

Commit

Permalink
Tokenizer: Fix line continuation after punctuation
Browse files Browse the repository at this point in the history
Prior to this change, any line ending with [punctuation + '...'], for
example `||...`, would cause the tokenizer to fail.
  • Loading branch information
ChristianStadelmann committed Jan 19, 2021
1 parent 5b76d7f commit 18cf851
Showing 1 changed file with 6 additions and 0 deletions.
6 changes: 6 additions & 0 deletions tokenize_code.m
Expand Up @@ -99,6 +99,12 @@
% any other operator:
else
symbol = skip(punctuation);
% ends with '...'
if length(symbol) > 3 && endsWith(symbol, '...')
% unskip '...'
pos = pos - 3;
symbol = symbol(1:end-3);
end
% one operator:
if any(strcmp(symbol, operators))
add_token('punctuation', symbol);
Expand Down

0 comments on commit 18cf851

Please sign in to comment.