Implementing "tick" to avoid using sleep

[kaiku]

Great work on this library. Just wondering, have you considered adding a tick method or some sort of frozen time mode to FakeRedis that allows programmatic advancing of time to avoid using an actual sleep in the test code?

[jamesls]

Wouldn't be opposed to a feature like that. Marking as HelpWanted.

[charettes]

freezegun can be used for this purpose and works like a charm with fakeredis because the later uses the stdlib datetime module to determine if keys are expired. Feels like it's not something that should be implemented in this package.

[bmerry]

Thanks, I haven't come across freezegun before. From a quick look at the docs it appears to mock the functions that get time, but not time.sleep - in which case it would presumably cause an infinite loop since the timeout would never be reached?

[charettes]

@bmerry DataBase.time is retrieved from time.time() on command processing

fakeredis/fakeredis/_server.py

Lines 818 to 820 in e04fc6e

	now = time.time()
	for db in self._server.dbs.values():
	db.time = now

This method is mocked by freezegun which means tests such as

fakeredis/test/test_fakeredis.py

Lines 3994 to 4001 in dcf0c89

	@pytest.mark.slow
	def test_expireat_should_expire_key_by_timestamp(r):
	r.set('foo', 'bar')
	assert r.get('foo') == b'bar'
	r.expireat('foo', int(time() + 1))
	sleep(1.5)
	assert r.get('foo') is None
	assert r.expire('bar', 1) is False

Could be rewritten as

@freeze_time(as_kwarg='frozen_time')
def test_expireat_should_expire_key_by_timestamp(r, frozen_time):
    r.set('foo', 'bar')
    assert r.get('foo') == b'bar'
    r.expireat('foo', int(time() + 1))
    frozen_time.tick(1.5)
    assert r.get('foo') is None
    assert r.expire('bar', 1) is False

[charettes]

In the case of internal usages of time.sleep

fakeredis/fakeredis/_server.py

Line 2592 in dcf0c89

time.sleep(0.01)

They can be mocked with mock.patch('time.sleep', frozen_time.tick). There even seems to be a solution for asyncio.sleep spulec/freezegun#290

[bmerry]

I think one problem is that the tests run against both real redis and fake redis to ensure that the tests have correct expectations, and of course freezegun won't work with real redis. But I can probably build a pytest fixture to do the mocking in the appropriate cases, which would at least halve the sleep time in tests. I'll take a look at some point when I have free time - thanks for the suggestion.

[adamchainz]

FYI I built a faster alternative to freezegun called time-machine. See https://adamj.eu/tech/2020/06/03/introducing-time-machine/ .

[slothyrulez]

@adamchainz sadly I'm trying this right now and does not seem to work 😢

def test_cache_expiration(self,...):
        frozen_time = pytz.UTC.localize(datetime.datetime(2023, 11, 23, 14, 52))
        with time_machine.travel(frozen_time, tick=False) as travel_time:
            print(datetime.datetime.now())
            print(cache.has_key('XXX'))
            print(cache.pttl('XXX'))
            ...
            # some caching with django cache and fakeredis
            # some asserts...
            ...
            before_expire_delta = datetime.timedelta(seconds=(CACHE_TIEMOUT - 10))
            travel_time.shift(before_expire_delta)
            print(datetime.datetime.now())
            print(cache.has_key('XXX'))
            print(cache.pttl('XXX'))
            ...
            # some asserts
            ...
            after_expire_delta = datetime.timedelta(seconds=(CACHE_TIEMOUT + 10))
            travel_time.shift(after_expire_delta)
            print(datetime.datetime.now())
            print(cache.has_key('XXX'))
            print(cache.pttl('XXX'))

2023-11-23 14:52:00                                                                                                                                                                                                                                                                                                                                                        
False                                                                                                                                                                                                                                                                                                                                                                      
0 
...   
2023-11-23 14:56:50                                                                                                                                                                                                                                                                                                                                                        
True                                                                                                                                                                                                                                                                                                                                                                       
299999                                                                                                                                                                                                                                                                                                                                                                     
...
2023-11-23 15:02:00                                                                                                                                                                                                                                                                                                                                                        
True                                                                                                                                                                                                                                                                                                                                                                       
299991                                                                                                                                                                                                                                                                                                                                                                     
...

[adamchainz]

@slothyrulez Hmm I am afraid I can't see why exactly. It looks like fakeredis (the new maintained repo) calls time.time() on each command: https://github.com/cunla/fakeredis-py/blob/04d4703d6bf7bc7c9d98d2d128cd206de80787b3/fakeredis/_basefakesocket.py#L271

time-machine definitely mocks time.time() just fine. If you add time.time() calls in your test they should show updates just like datetime.now().

Maybe the above code path isn't running, so the server time variable is not updating correctly. Calling the time command would probably be instructive.

[slothyrulez]

Ummmmm, should I open a discussion on time-machine or in fakeredis ?

Definitively, thanks for your response @adamchainz

[adamchainz]

I think this is an issue with how fakeredis reads the time. I’d only want a time-machine issue if you can show there’s an underlying cause that needs a fix.

[slothyrulez]

@slothyrulez Hmm I am afraid I can't see why exactly. It looks like fakeredis (the new maintained repo) calls time.time() on each command: https://github.com/cunla/fakeredis-py/blob/04d4703d6bf7bc7c9d98d2d128cd206de80787b3/fakeredis/_basefakesocket.py#L271

time-machine definitely mocks time.time() just fine. If you add time.time() calls in your test they should show updates just like datetime.now().

Maybe the above code path isn't running, so the server time variable is not updating correctly. Calling the time command would probably be instructive.

BEFORE time_machine.travel --
datetime.datetime.now()=datetime.datetime(2023, 11, 24, 6, 1, 23, 8122)
time.time()=1700805683.0081432
cache.client.get_client().time()=(1700805683, 8477)
cache.has_key("XXXX")=False
cache.pttl("XXXX")=0

AFTER time_machine.travel --
datetime.datetime.now()=datetime.datetime(2023, 11, 23, 14, 52)
time.time()=1700751120.0
cache.client.get_client().time()=(1700805683, 16603)
cache.has_key("XXXX")=False
cache.pttl("XXXX")=0
--
CACHED DURING 300 secs (5min.)
--
TRAVEL 4m50s to the future
datetime.datetime.now()=datetime.datetime(2023, 11, 23, 14, 56, 50)
time.time()=1700751410.0
cache.client.get_client().time()=(1700805683, 38205)
cache.has_key("XXXX")=True
cache.pttl("XXXX")=299999
--
TRAVEL 15s to the future
datetime.datetime.now()=datetime.datetime(2023, 11, 23, 14, 57, 5)
time.time()=1700751425.0
cache.client.get_client().time()=(1700805683, 47826)
cache.has_key("XXXX")=True
cache.pttl("XXXX")=299989

[slothyrulez]

Cross-posting here for future visibility cunla/fakeredis-py#253
Thanks @jamesls and sorry for the noise