Use 'scan' function, lost some key

[AttentionZ]

import asyncio
import asyncio_redis
from asyncio_redis.encoders import BytesEncoder
from policy import _redis_config

@asyncio.coroutine
def callback(cors,envenloop):
    result = yield from asyncio.wait_for(cors,timeout=None,loop=envenloop)
    yield from asyncio.sleep(1)

@asyncio.coroutine
def enumkey(redis_conn,m):
    redis_cursor = yield from redis_conn.scan(match=m)
    keys = yield from redis_cursor.fetchall()
    i = 0
    for key in keys:
        print(i,":",key)
        i+=1
    return i
@asyncio.coroutine
def task(eventloop):
    redis_conn = None
    try:
        redis_conn = yield from asyncio_redis.Connection.create(auto_reconnect=False,encoder=BytesEncoder(),loop=eventloop,**_redis_config)
        print('-----------------------------------')
        i = yield from enumkey(redis_conn,b'p*')
        print('p* total %d'%i)
        print('-----------------------------------')
        i = yield from enumkey(redis_conn,b'pu:*')
        print('pu:* total %d'%i)
        print('-----------------------------------')
        i = yield from enumkey(redis_conn,b'pn:*')
        print('pn:* total %d'%i)
        print('-----------------------------------')
        i = yield from enumkey(redis_conn,b'k*')
        print('k* total %d'%i)
        print('-----------------------------------')
        i = yield from enumkey(redis_conn,b'sync*')
        print('sync* total %d'%i)
        print('-----------------------------------')
        i = yield from enumkey(redis_conn,b'*')
        print('* total %d'%i)

    except Exception as e:
        print(e)
    finally:
        if redis_conn:
            redis_conn.close()
if __name__ == '__main__':
    eventloop = asyncio.get_event_loop()
    cors = task(eventloop)
    eventloop.run_until_complete(callback(cors,eventloop))

---

There are output:

p* total 70
pu:* total 0
pn:* total 64
k* total 1
k2* total 0
k3* total 0
sync* total 0

• total 74

---

But, when I use redis lib,there are output:
p* total 70
pu:* total 3
pn:* total 64
k* total 3
k2* total 1
k3* total 1
sync* total 1

• total 74

[AttentionZ]

I fix the bug in : asyncio_redis\cursors.py : change the if to while

class Cursor:
    @asyncio.coroutine
    def fetchone(self):
        """
        Coroutines that returns the next item.
        It returns `None` after the last item.
        """
        while not self._queue and not self._done:
            yield from self._fetch_more()

        if self._queue:
            return self._queue.popleft()

[jonathanslenders]

Thanks for reporting this issue and proposing a bug fix!
I'm not entirely sure that this is the right solution, but I'll try to have a look asap and fix it.

[achimnol]

I just faced the same bug and found that the above fix is correct.
Redis SCAN command may return no items even when the cursor is not finished (i.e., not became zero).
In this case, we need to call _fetch_more() multiple times until we get any item, so that we won't return None even when the cursor is not finished!

I inserted some print statements, and here is the trace of one of such cases:

Cursor: fetchone: q? False, done? False / current cur 0
Cursor: _fetch_more: cur 0, new cur 176, fetched # 0
<-- the original code returns None here!
Cursor: _fetch_more: cur 176, new cur 196, fetched # 0
Cursor: _fetch_more: cur 196, new cur 44, fetched # 0
Cursor: _fetch_more: cur 44, new cur 210, fetched # 0
Cursor: _fetch_more: cur 210, new cur 122, fetched # 0
Cursor: _fetch_more: cur 122, new cur 246, fetched # 0
Cursor: _fetch_more: cur 246, new cur 65, fetched # 0
Cursor: _fetch_more: cur 65, new cur 201, fetched # 1
Cursor: fetchone returned
scan: a37273dee34141f18a7c8f58eac1b751
<-- but we should repeat _fetch_more() until we reach this point.
Cursor: fetchone: q? False, done? False / current cur 201
Cursor: _fetch_more: cur 201, new cur 5, fetched # 0
Cursor: _fetch_more: cur 5, new cur 141, fetched # 0
Cursor: _fetch_more: cur 141, new cur 83, fetched # 0
Cursor: _fetch_more: cur 83, new cur 7, fetched # 0
Cursor: _fetch_more: cur 7, new cur 239, fetched # 0
Cursor: _fetch_more: cur 239, new cur 0, fetched # 0
Cursor: done!

[achimnol]

And, I bump this issue to be fixed asap, because the result is undeterministic and unpredictable: this bug may appear or not appear on repetition of the same test codes, and does not raise explicit exceptions to the user.

[jonathanslenders]

Hi @achimnol,

Thanks for explaining. Now it's clear to me. Can you have a look at this pull request: #77

If it's fine for you, I'll merge and release it.

[achimnol]

@jonathanslenders , I think the PR looks perfect. 👍

[jonathanslenders]

Ok :) It's merged.