Skip to content

Conversation

@funny000
Copy link
Owner

To combine.

ask added 30 commits December 10, 2015 13:20
regen does not work with dequeue(regen(it)), as deque seems
to use some C magic instead of __iter__ for copying the list,
so the iterator ends up being consumed.
ask and others added 30 commits March 10, 2016 13:26
Fixed bug where database backend not passing through URL to BaseBackend __init__.
Getting rid of leaking memory + adding minlen size of the set
minlen is minimal residual size of set after operating for long.
Minlen items are kept, even if they should be expired by time, until
we get newer items.

Problems with older and even more old code:

1)
   Heap would tend to grow in some scenarios
   (like adding an item multiple times).

2) Adding many items fast would not clean them soon enough (if ever).

3) When talking to other workers, revoked._data was sent, but
   it was processed on the other side as iterable.
   That means giving those keys new (current)
   timestamp. By doing this workers could recycle
   items forever. Combined with 1) and 2), this means that in
   large set of workers, you are getting out of memory soon.

All those problems should be fixed now,
also some new unittests are added.

This should fix issues #3095, #3086.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.