Lexer "Groups" state persists between different inputs tokenization. #287

VILOZNY · 2016-10-10T08:11:27Z

Hi,

While using the same lexer object (tokenize API) and using GRPUPs (for spaces tokens), the group array is not cleaned after every lexer execution. Therefore I get wrong tokens in in the group array at the n execution of the lexer (n > 1)

regards,

Efrat

Fixes #287

bd82 changed the title ~~unclean lexer groups state~~ Lexer "Groups" state persists between different inputs tokenization. Oct 10, 2016

bd82 added a commit that referenced this issue Oct 10, 2016

Lexer Token Groups were not reset on each Tokenize call.

beba98b

Fixes #287

bd82 mentioned this issue Oct 10, 2016

Lexer Token Groups were not reset on each Tokenize call. #288

Merged

bd82 added a commit that referenced this issue Oct 10, 2016

Lexer Token Groups were not reset on each Tokenize call.

ea344c8

Fixes #287

bd82 added a commit that referenced this issue Oct 10, 2016

Lexer Token Groups were not reset on each Tokenize call.

8fd0f84

Fixes #287

bd82 added a commit that referenced this issue Oct 10, 2016

Lexer Token Groups were not reset on each Tokenize call.

d77266e

Fixes #287

bd82 closed this as completed in #288 Oct 10, 2016

bd82 added the Bug 🪲 label Oct 12, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lexer "Groups" state persists between different inputs tokenization. #287

Lexer "Groups" state persists between different inputs tokenization. #287

VILOZNY commented Oct 10, 2016

Lexer "Groups" state persists between different inputs tokenization. #287

Lexer "Groups" state persists between different inputs tokenization. #287

Comments

VILOZNY commented Oct 10, 2016