why "ZSCAN" 's "COUNT" didn't work? [Redis version 2.8.7]

[Michael2008S]

ZSCAN "feed_uid_32" 0 COUNT 1

didn't work? in [Redis version 2.8.7]. it return me all the "SortedSet" items:

127.0.0.1:6379> ZSCAN "feed_uid_32" 0 count 1
1) "0"
2)  1) "tcid_76_uid_32"
    2) "189574"
    3) "tcid_67_uid_32"
    4) "44000962"
    5) "tcid_72_uid_32"
    6) "182988820"
    7) "tcid_79_uid_32"
    8) "269762745"
    9) "tcid_82_uid_32"
   10) "273389231"
   11) "tcid_68_uid_32"
   12) "301953866"
   13) "tcid_75_uid_32"
   14) "406466377"
   15) "tcid_63_uid_32"
   16) "413702094"
   17) "tcid_81_uid_32"
   18) "413772463"
   19) "tcid_62_uid_32"
   20) "417314423"
   21) "tcid_70_uid_32"
   22) "482110616"
   23) "tcid_65_uid_32"
   24) "484825967"
   25) "tcid_80_uid_32"
   26) "518171269"
   27) "tcid_83_uid_32"
   28) "551417031"
   29) "tcid_71_uid_32"
   30) "664436922"
   31) "tcid_69_uid_32"
   32) "671237575"
   33) "tcid_84_uid_32"
   34) "676393350"
   35) "tcid_66_uid_32"
   36) "709271494"
   37) "tcid_74_uid_32"
   38) "764818601"
   39) "tcid_73_uid_32"
   40) "889262123"
   41) "tcid_77_uid_32"
   42) "911273045"
   43) "tcid_78_uid_32"
   44) "962082648"
   45) "tcid_64_uid_32"
   46) "996581893"

[antirez]

Hello. It is clearly stated in the documentation, please read it, it's a good read :-)

[Michael2008S]

Sorry. I still don't know it can or not use the "COUNT count" in the documentation --> http://redis.io/commands/zscan

 ZSCAN key cursor [MATCH pattern] [COUNT count] 

If it havd been can used. which version can use it?

[mattsta]

SCAN is a general access pattern described at http://redis.io/commands/scan

What you're looking for is:

[COUNT] is just an hint for the implementation

the server will usually return count or a bit more than count elements per call.

When iterating Sets encoded as intsets (small sets composed of just integers), or Hashes and Sorted Sets encoded as ziplists (small hashes and sets composed of small individual values), usually all the elements are returned in the first SCAN call regardless of the COUNT value.

[Michael2008S]

Got it, Thank you. apologize for my mistake.

[offero]

Thanks for pointing this out as I had the same question, so it is not so obvious and I do RTFM.

Why is it the case the "usuall all the elements are returned"?

[mattsta]

Check out the comment at https://github.com/antirez/redis/blob/95b1979c321eb6353f75df892ab8be68cf8f9a77/src/db.c#L473-L479

Short version: for Redis containers using efficient immediate values (ziplist/skiplist), the entire container is returned because it's all one sequential blob in memory.

Also, the efficient immediate value containers are optimized for storing small numbers of things, so returning the entire container is assumed to be okay because they should only ever have a small number of things in them. Though, if you manually change the maximum ziplist entry count from 64 to 64,000, you may get a surprise when using SCAN because your COUNT will always return chunks of 64,000 elements.

[vvhungy]

I store 20,000,000 items in an IntSet and SSCAN with COUNT 5000 still return all items. Could we consider this as a bug, mean SSCAN should respect the COUNT parameter?

[antirez]

@vvhungy you can't store 20M items an in intSet, since after a small amount the intSet is converted into an hash table.

[vvhungy]

@antirez: I did it with Redis 3.0.2 using config setting: set-max-intset-entries 20000000.

So I propose a config, let say: "scan-max-elements-returned" to prevent redis returning too many data that may slow down server performance, and *SCAN must always respect the COUNT parameter as long as COUNT <= scan-max-elements-returned .

[michael-grunder]

As was mentioned earlier in this thread the intset type is one contiguous blob of memory. Internally it's actually a sorted list of integers (with variant encoding).

It's been a while since I looked at the scan code, but I don't think it is trivial to maintain a cursor for blob structures like intset and ziplist.

Also, IMHO unless you know for a fact that you're only inserting to the tail of this list, setting set-max-intset-entries to 20M is insane. When an insertion is done anywhere but the tail, it requires a memmove operation, which would be a rather substantial performance problem if, for example, you added to the head.

[vvhungy]

@michael-grunder: agree with you in the second point, sure I should be aware when doing this.
But the point here is that *SCAN must always respect the COUNT parameter including the IntSet case, not as being ignored as in my test.

Also with "scan-max-elements-returned" config, we can actively prevent the *SCAN command to ruin server performance.