Question / feature request

[redaktor]

for normalizing the input:

How about all normalizing all typographic stuff like curly and special quotes
to well the normalized ones ?

maybe useful for e.g.
O’Reilly to O'Reilly etc.

see http://practicaltypography.com/straight-and-curly-quotes.html

and note to me :
Maybe it would be useful to write a "preprocess" test, testing if everything in .js and .min.js ("expanded") is the same.

[spencermountain]

nice idea. will add to tokenize.js

you wanna do it?

[redaktor]

note: I did it, I'll submit it with the fork but it raised two questions about
methods/tokenization/tokenize.js

This is what it looks now (with complete curly quotes replacement):

function normalise (str) {
    if (!str) { return "" }
    str = str.toLowerCase();
    str = str.replace(/[,\.!:;\?\(\)]/, '');
    // single curly quotes
    str = str.replace(/[\u2018\u2019\u201A\u201B\u2032\u2035]+/g, "'");
    // double curly quotes
    str = str.replace(/[\u201C\u201D\u201E\u201F\u2033\u2036]+/g, '"');
    if (!str.match(/[a-z0-9]/i)) { return '' }
    return str
}

• Is the str trimmed here already ?
and
• if the str is not normalized in the end

if (!str.match(/[a-z0-9]/i)) {

we could normalize it further by your ../transliteration/unicode_normalisation.js
before

return ''

[spencermountain]

the fix looks great, i've added a test too.