Slow multithreading

[DrJimFan]

From my understanding, Redis-py is mostly doing network IO under the hood. Python's GIL should not kick in for network IO. However, it looks like multithreading is as slow as single-threaded sequential version. Am I missing anything?

r = StrictRedis()

def test():
    for i in range(10000):
        key = 'mykey'+str(i)
        r.set(key, 'ibvjaiuj65s4fa4sd8ear4')
        r.get(key)
        r.delete(key)

def bench1():
    for _ in range(5):
        test() 

def bench2():  # as slow as bench 1
    ts = [threading.Thread(target=test) for i in range(5)]
    [t.start() for t in ts]
    [t.join() for t in ts]

def bench3():  # much faster, which means GIL is in play
    ts = [multiprocessing.Process(target=test) for i in range(5)]
    [t.start() for t in ts]
    [t.join() for t in ts]

[Grokzen]

@linxifan Multithreading and concurency in python is very hard and i would suggest that you dig into the talks from David Beazley has done on this topic to better understand the problem with threads/multiprocessing/async and so on. One of the better presentations he has done is this one https://www.youtube.com/watch?v=MCs5OvhV9S4 where he show code examples live and the performance of the different methods.

[laixintao]

@linxifan I suppose you run redis at a localhost? Then there is no network request happened.

[DrJimFan]

@laixintao yes, I am running it on localhost. But it's still IO operation, and therefore should not invoke the GIL, no?

[laixintao]

@linxifan GIL is always invoked whenever where is thread. GIL grantee there is only one thread running in Python interpreter.

You should noticed that redis in running on memory, so if you test your code in localhost, there is no networking io or file system io.

Compare to single thread, multi thread need switching(which cost), so slower than single thread.

[NoneGG]

@linxifan In addtition to @laixintao 's suggestion, redis trully has network IO(even run locally, it still listens on port given), but so little that it can be ignored, file system IO(if you turned AOF on， which should be checked before test). The relationship between thread number and cpu number is also needed
to be taken into consider if you want to test in multithreading env (although it is not so important for the sake of GIL, maybe multiprocessing is better)
With so condition variable reffered before can we come to the conclusion that if redis-py can run fast in multithreading env(i recommend multiprocessing + pipeline to improve the efficiency indeed)

[coleifer]

Please don't forget the CPU time needed to encode commands and decode responses. I imagine, because the latency is next-to-nothing, that you're bumping up against the limits of your python interpreter to send and parse the responses.

[github-actions]

This issue is marked stale. It will be closed in 30 days if it is not updated.