Skip to content

Commit

Permalink
Add documentation for new FileHuey implementation.
Browse files Browse the repository at this point in the history
Also make small fix to the KyotoTycoonHuey contrib module.
  • Loading branch information
coleifer committed Jan 8, 2020
1 parent d31940b commit 0c3037f
Show file tree
Hide file tree
Showing 2 changed files with 30 additions and 30 deletions.
57 changes: 28 additions & 29 deletions docs/api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,24 @@ Huey types
Huey that uses in-memory storage. Only should be used when testing or when
using ``immediate`` mode. MemoryHuey fully supports task priorities.

.. py:class:: FileHuey
Huey that uses the file-system for storage. Should not be used in
high-throughput, highly-concurrent environments, as the
:py:class:`FileStorage` utilizes exclusive locks around all file-system
operations.

:param str path: base-path for huey data (queue tasks, schedule and results
will be stored in sub-directories of this path).
:param int levels: number of levels in result-file directory structure to
ensure the results directory does not contain an unmanageable number of
files.
:param bool use_thread_lock: use the standard lib ``threading.Lock``
instead of a lockfile for file-system operations. This should only be
enabled when using the greenlet or thread consumer worker models.

FileHuey fully supports task priorities.


Huey object
-----------
Expand Down Expand Up @@ -1377,42 +1395,23 @@ Huey comes with several built-in storage implementations:
connection constructor.


.. py:class:: FileStorageMethods(name, path, levels=2, **storage_kwargs)
.. py:class:: FileStorage(name, path, levels=2, use_thread_lock=False)
:param str name: (unused by the file storage API)
:param str path: directory path used to store task results. Will be created
if it does not exist.
:param int levels: number of levels in cache-file directory structure to
ensure a given directory does not contain an unmanageable number of
files.
:param storage_kwargs: Additional keyword arguments for the parent storage.

Unlike the other storage implementations described, the
:py:class:`FileStorageMethods` class is intended to be used as a mixin
alongside another storage engine. This class implements the result store
APIs (put, peek, pop), which are used for task result storage among other
things.

Example of using Redis for the queue and the file-system for the result
store:

.. code-block:: python
from huey import Huey
from huey.storage import FileStorageMethods, RedisStorage
# Use the file-system for result storage, Redis for everything else.
class RedisFileStorage(FileStorageMethods, RedisStorage):
pass
huey = Huey(
'my-app',
storage_class=RedisFileStorage,
path='/var/lib/my-app/huey-results/', # File storage params.
levels=2,
url='redis://localhost:6379/15', # Redis storage params.
client_name='my-app-huey')
:param bool use_thread_lock: use the standard lib ``threading.Lock``
instead of a lockfile. Note: this should only be enabled when using the
greenlet or thread consumer worker models.

The :py:class:`FileStorage` implements a simple file-system storage layer.
This storage class should not be used in high-throughput, highly-concurrent
environments, as it utilizes exclusive locks around all file-system
operations. This is done to prevent race-conditions when reading from the
file-system.


.. py:class:: MemoryStorage()
Expand Down
3 changes: 2 additions & 1 deletion huey/contrib/kyototycoon.py
Original file line number Diff line number Diff line change
Expand Up @@ -119,4 +119,5 @@ def flush_all(self):
self.flush_results()


KyotoTycoonHuey = partial(Huey, storage_class=KyotoTycoonStorage)
class KyotoTycoonHuey(Huey):
storage_class = KyotoTycoonStorage

0 comments on commit 0c3037f

Please sign in to comment.