BUG when autocompleting unicode strings

[mannol]

Following search provides no results:

redisCommand(ctx, "FT.SUGADD userslex %b 1", "\u010Caji\u0107", sizeof("\u010Caji\u0107") - 1);
redisCommand(ctx, "FT.SUGGET userslex %b MAX 2", "\u010Caj", sizeof("\u010Caj") - 1);

Another example using random bytes:

uint32_t a = 1234432413;
redisCommand(ctx, "FT.SUGADD userslex %b 1", &a, 4);
redisCommand(ctx, "FT.SUGGET userslex %b MAX 2", &a, 3); // nothing found

[dvirsky]

Fixed as far as I can test (haven't run your test code though). Feel free to verify.

[mannol]

Fix confirmed! Thanks!

[dvirsky]

Going over this, it looks like working with 32 bit runes will not be too hard to do, and will allow full fuzzy support in unicode. but it will make memory consumption terrible. Is having unicode supported fuzzy matching critical for you?

[mannol]

It's essential, yes. How much would the memory requirements increase?

[dvirsky]

The idea is not to use variable length encoding like UTF-8 and UTF-16, and use a fixed length encoding, probably 32 bit - per letter.

So for pure latin text, which in utf-8 is represented by 1 byte, you would have to use 4 bytes. Some of the memory consumption is of course pointers and metadata, so we are talking about x2-x3 the amount of RAM.

For purely non-latin, which takes up 2-3 bytes per letter in utf-8, it would probably be x1.5-x2.

If I manage to get away with 16 bit per letter, which won't cover all languages but will cover the most popular ones IIRC, it won't be so bad.

[mannol]

Yeah, 16 bits should cover all the languages we are aiming at so that's okay.

[dvirsky]

yeah, range 0x0000-0xFFFF covers everything a sane person would need. https://en.wikipedia.org/wiki/Plane_(Unicode)#Basic_Multilingual_Plane

Although Unicode and sanity don't go hand in hand, in this case it might be a good compromise.