nonsensical code in Whitespace->__TOKENIZER__on_char

[wchristian]

There is a bit there that seems to try to determine whether a character outside of the ASCII range is word or whitespace, however instead of actually looking at the current character it looks at the stringified tokenizer, which is just a perl address. I'm unclear on whether the tokenizer is supposed to stringify, or whether this was just a piece where the meaning of $t changed without the code adapting. Anything but the obvious change to chr($char) you'd like done here?

[adamkennedy]

The tokenizer doesn't stringy, it's presumably a bug.

Adam
On Dec 18, 2013 5:56 AM, "Christian Walde" notifications@github.com wrote:

There is a bit there that seems to try to determine whether a character
outside of the ASCII range is word or whitespacehttps://github.com/adamkennedy/PPI/blob/master/lib/PPI/Token/Whitespace.pm#L407,
however instead of actually looking at the current character it looks at
the stringified tokenizer, which is just a perl address. I'm unclear on
whether the tokenizer is supposed to stringify, or whether this was just a
piece where the meaning of $t changed without the code adapting. Anything
but the obvious change to chr($char) you'd like done here?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/10
.

[wchristian]

Cool, i'll fix that then.

[wchristian]

Quick note: The specific code that's nonsensical here is:

https://metacpan.org/source/ADAMK/PPI-1.215/lib/PPI/Token/Whitespace.pm#L407

That's still not fixed and does need a fix.

[adamkennedy]

That looks like it might have been a very early attempt to do something at
least remotely usful with Unicode or accented Latin1 characters. Probably
the latter...
On Nov 12, 2014 9:39 AM, "Christian Walde" notifications@github.com wrote:

Quick note: The specific code that's nonsensical here is:

https://metacpan.org/source/ADAMK/PPI-1.215/lib/PPI/Token/Whitespace.pm#L407

That's still not fixed and does need a fix.

—
Reply to this email directly or view it on GitHub
#10 (comment).

[wchristian]

Alright, i worked out what's going on there.

The code tried to figure out whether a non-ascii char is whitespace or a word char, to proceed with that then; and otherwise throw an error message explaining that it doesn't know what to do with the char. With the original version of the code it ALWAYS recognized it as a word char, which caused weird crashes in Token::Word.

After fixing it to inspect that usefully, the error from utf8 that was transformed into unhandlable mojibake now complains about an unexpected character instead.

This still needs further adressing by way of better input handling as mentioned in #26, but for now this is an improvement.

[adamkennedy]

Agreed
On Nov 13, 2014 9:28 AM, "Christian Walde" notifications@github.com wrote:

Alright, i worked out what's going on there.

The code tried to figure out whether a non-ascii char is whitespace or a
word char, to proceed with that then; and otherwise throw an error message
explaining that it doesn't know what to do with the char. With the original
version of the code it ALWAYS recognized it as a word char, which caused
weird crashes in Token::Word.

After fixing it to inspect that usefully, the error from utf8 that was
transformed into unhandlable mojibake now complains about an unexpected
character instead.

This still needs further adressing by way of better input handling as
mentioned in #26 #26, but for
now this is an improvement.

—
Reply to this email directly or view it on GitHub
#10 (comment).