SORT: use strcoll() when appropriate.

[antirez]

If you search for "change SORT order" thread in the Redis mailing list, you'll find that Redis up to 2.6 version uses binary comparison between strings for everything but when you use BY and ALPHA at the same time in SORT. In that case strcoll() is used, but since no locale is set, the effect is that anyway binary comparison is used since the default locale will be "C".

So well, the net result is that Redis always uses lexicographic comparison, that's bad since SORT is designed in order to return results in a way that is ready to display to the user, so proper collation should be used.

The no brainer would be to use strcoll() everywhere in the context of the SORT command, and use:

setlocale(LC_COLLATE,""); // in order to active the environment locale for strcoll().

However SORT + STORE is a write command, and as such should behave in a predictable way so that AOF and the replication stream are consistent. There is no easy way to understand if a master and a slave are using the same collation, nor I want to introduce an handshake thing for this, so the temporary solution for 2.8 could be the following:

1. Use setlocale as specified above.
2. Use strcoll() everywhere in the scope of SORT>
3. However if STORE is used, sort just with strcmp() in a binary way.

This at least means that for many uses (everytime SORT is not involved) you can expect SORT + ALPHA to return results in the expected order.

Comments welcomed!

[dspezia]

If strcoll is used for sorting, what will happen with strings with \0 characters? Currently, most sort comparisons are done using compareStringObjects which results in sdscmp if both parameters are really strings. sdscmp is using memcmp which makes it binary safe.

We loose this property with strcoll/strcmp.

Now, does it make sense to sort strings containing binary data?

[antirez]

Hello Didier,

IMHO the point is your last sentence, it does not make a lot of sense to sort binary data in this context, and this sorting is only done in read only mode, so no effect is created in the database. I expect that when somebody does a SORT ALPHA, the goal is to display the output to some kind of human being :-)

[antirez]

Fixed by

• 81e55ec
• cf1579a
• 123b221

Closing.

[tsee]

I'd like to see this get some further consideration. If I have binary data in a list, I do think it's perfectly legitimate to want to sort it. If I understand things correctly, right now there is no way to get totally predictable sort order out of "SORT mylist ..." if there's binary data. Using the perl Redis client (since I know first-hand it handles NULs correctly):

$ perl -MRedis -le '$r = Redis->new; $r->del("mylist"); $r->lpush("mylist", $_) for ("a", "b", "a\0b", "a\0c", "a\0a", "a\0d", "c"); print for $r->sort("mylist", "alpha");'
ad
aa
ac
ab
a
b
c