SPR-16032 - Fix tonkenizer

[zzt93]

It uses the ch == 0 to check whether this expression reaches end, makes me impossible to use \0 in Spring EL like following:

#{#str?.split('\0')}

[snicoll]

@zzt93 please create a Jira issue for this change.

[zzt93]

@snicoll Jira issue created.

[jhoeller]

Superseded by a larger revision of the Tokenizer implementation (which was quite overdue already), as mentioned on JIRA. Thanks for the pull request, in any case!