filequeue is a Python library that provides a thread-safe queue which is a subclass of
Queue.Queue from the stdlib.
filequeue.FileQueue will overflow into a compressed file if the number of items exceeds maxsize, instead of blocking or raising
Full like the regular
There is also
filequeue.LifoFileQueue will only behave the same as
Queue.LifoQueue respectively if they are initialised with
maxsize=0 (the default). See
__init__ docstring for details (
filequeue.PriorityFileQueue won't currently work exactly the same as a straight out replacement for
Queue.PriorityQueue. The interface is very slightly different (extra optional kw argument on
__init__), although it will work it won't behave the same. It might still be useful to people though and hopefully I'll be able to address this in a future version.
- Python 2.5+ or Python 3.x
The motivation came from wanting to queue a lot of work, without consuming lots of memory.
The interface of
filequeue.FileQueue matches that of
queue.Queue in python 3.x). With the idea being that most people will use
Queue.Queue, and can swap in a
filequeue.FileQueue only if the memory usage becomes an issue. (Same applies for
Made available as-is under the BSD Licence.
Any issues please post on the github page.