store token position + fix tokenizer

[milahu]

this is useful for a "live diff" editor, where i must reset the cursor position

WIP demo: live html diff editor with htmldiff.js

edit: this use case is deprecated, since every diff-algo will produce "false diffs"
and the only way to get a "real live diff" this is to track the input and selectionchange events

WIP demo 2: live html diff editor with inputevent (also on github)

other small edits ...

6ea5b33 word, should be tokenized to |word|,| not to |word,|

3fefdab     "word should be tokenized to |    |"|word| not to |    |"word|
|word|deletion|"| -> |word|"| should produce a delete, no replace

a77f632 wörd should be tokenized to |wörd| not to |wö|rd|
for (const char of str) will loop unicode chars, const char = str[i] will loop bytes (src)
xregexp is a fast unicode matcher (src)

[jleider]

Hi @milahu, apologies for the delayed response. We are taking a look at your PR today. Do you think you could add some tests specifically for the tokenizer changes you made?

[SuttonKyle]

Makes sense, for the most part. Changes are not particularly relevant to our use cases -- @jleider do you think it's worth including this added functionality in the main repo or should this PR really be another fork?

[SuttonKyle]

Suggested change

	for (let charIdx = 0; charIdx < unicodeChars.length; charIdx++) {
	unicodeChars.forEach((char, charIdx) => {

[SuttonKyle]

why are we removing char here?

[SuttonKyle]

Suggested change

const char = unicodeChars[charIdx] as string;

[SuttonKyle]

You'll have to account for the style_tag case as well now.

[milahu]

sorry, i have to abandon this patch

notes:

• for (var i = 0; ...) is fastest loop in all environments, forEach can have function call overhead
• today i would try to avoid converting string to array

i cannot reproduce why i added Array.from(html) to loop unicode chars

for (var c of '\u00e4\u00f6\u00fc') console.log(c)
for (var c of Array.from('\u00e4\u00f6\u00fc')) console.log(c)
// nodejs:
/*
ä
ö
ü
*/

more micro optimization: maybe the string parsing could be optimized by looping bytes (Uint8Array typed array), using a jump table to handle values <= 127, and treating all values >= 128 as word chars (all unicode bytes have value >= 128)

[jleider]

Closing this in favor of #5 which incorporates some of these changes.