You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
Motivation
----
Some issues arose when the token reader was used to parse text and
delimeter file tokens
Modifications
----
* Updated the GenericTokenStateHandler/FileTokenReader to better handle
state with text file tokens
* Updated DefaultTokenStateHandler to account for
comments/delimeters/etc. as expected
Result
----
Text, delimeter, and other token isses should be fixed with
DefaultTokenStateHandler
Fixes #7