Merge master

docs/source/cli.rst

@@ -0,0 +1,3 @@
+.. click:: redis_tasks.cli:main
+   :prog: redis_tasks
+   :show-nested:

docs/source/conf.py

@@ -10,9 +10,10 @@
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 #
-# import os
-# import sys
-# sys.path.insert(0, os.path.abspath('.'))
+import os
+import sys
+
+sys.path.insert(0, os.path.abspath('../../'))
 
 
 # -- Project information -----------------------------------------------------
@@ -28,6 +29,9 @@
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
+    'sphinx.ext.autodoc',
+    'sphinx.ext.viewcode',
+    'sphinx_click.ext',
 ]
 
 # Add any paths that contain templates here, relative to this directory.

docs/source/index.rst

@@ -1,20 +1,174 @@
-.. redis-tasks documentation master file, created by
-   sphinx-quickstart on Mon May 18 11:09:59 2020.
-   You can adapt this file completely to your liking, but it should at least
-   contain the root `toctree` directive.
-
-Welcome to redis-tasks's documentation!
+Welcome to redis-tasks' documentation!
 =======================================
 
+``redis tasks`` is a framework for reliable background tasks execution,
+specifically for environments where workers are expected to restart frequently,
+like Heroku. The task queueing is handled by ``redis``.
+
+Disclaimer
+----------
+``redis-tasks`` does not store any kind of task results.
+If you need to store task results, you have to use
+a persistence layer of your choice in your task code.
+
+Installation
+============
+
+Your can install ``redis-tasks`` using pip like this::
+
+    $ pip install redis-tasks
+
+.. note:: ``redis-tasks`` only runs on python versions 3.6 and above.
+
+| You have to set the :envvar:`RT_SETTINGS_MODULE` environent variable 
+  before you can use redis-tasks in your code.
+|
+| Alternativaly you can call :py:meth:`redis_tasks.conf.settings.configure`
+  passing in a python settings module or call
+  :py:meth:`redis_tasks.conf.settings.configure_from_dict` passing in a dict
+  of settings.
+| See :py:class:`redis_tasks.conf.Settings` for more details.
+
+Quickstart Guide
+================
+
+Prerequisite: You have successfully installed ``redis-tasks`` and have a ``redis`` insteance ready.
+Prerequisite: You have successfully installed ``redis-tasks`` and have a running ``redis`` instance.
+In this Quickstart Guide we'll use ``"redis://localhost:6379"`` as our :any:`REDIS_URL`
+
+1. Create a `tasks.py` module and write a function that you want to execute in a worker using ``redis-tasks``
+
+.. code:: python
+
+    import urllib.request
+    from redis_tasks import redis_task
+
+    @redis_task()
+    def print_len_of_url_content(url):
+        with urllib.request.urlopen(url) as f:
+            print(len(f.read()))
+
+2. Create a `settings.py` module to supply your custom config to ``redis-tasks``
+   In this example we just pass the :any:`REDIS_URL` from the enviroment
+
+.. code:: python
+
+    import os
+    REDIS_URL = os.environ['REDIS_URL']
+
+3. In a separate terminal/shell start a worker::
+
+   $ export REDIS_URL='redis://127.0.0.1:6379'
+   $ export RT_SETTINGS_MODULE=settings 
+   $ redis_tasks worker
+
+You should see a message that the worker started successfully::
+
+    16:15:39 INFO    redis_tasks.worker: Worker iw-T460p.14152 [0e015d7a-79c7-42be-8b02-add7161c5951] started
+
+4. In a separate terminal/shell you can
+   now enqueue this tasks to be executed by the worker at any time like this:
+
+.. note:: Make sure that the python process is also started with the same ENV VARS as the worker process, see export calls in step 3.
+
+.. code:: python
+
+    from redis_tasks import Queue
+    from tasks import print_len_of_url_content
+    Queue().enqueue_call(
+        print_len_of_url_content,
+        kwargs={'url': 'https://www.readthedocs.org/'})
+
+In the worker logs you should see the task being started and finished::
+
+    16:15:53 INFO      redis_tasks.task: Task tasks.print_len_of_url_content(url='https://www.readthedocs.org/') [095c3eb9-7073-4ad6-a7cc-16885ee962a4] started
+    12862
+    16:15:53 INFO      redis_tasks.task: Task tasks.print_len_of_url_content(url='https://www.readthedocs.org/') [095c3eb9-7073-4ad6-a7cc-16885ee962a4] finished
+
+Adding Scheduled Tasks
+----------------------
+So far the configuration allowed you to run the tasks almost immediately. What if you want to specify the exact time?
+
+5. To add regularly executed tasks we extend the `settings.py` module to this:
+
+.. code:: python
+
+    import os
+    from redis_tasks import run_every
+
+    REDIS_URL = os.environ['REDIS_URL']
+
+    TIMEZONE = "UTC"  # timezone to be used by scheduler
+    SCHEDULE = {
+        'print_len_of_readthedocs_org': {
+            'task': 'tasks.print_len_of_url_content',
+            'schedule': run_every(minutes=20),
+            'args': [],
+            'kwargs': {'url': 'https://readthedocs.org/'},
+        },
+    }
+
+For more details on SCHEDULE configuration see :doc:`scheduler`
+
+
+6. In a separate shell/terminal start a scheduler process::
+
+   $ export REDIS_URL='redis://127.0.0.1:6379'
+   $ export RT_SETTINGS_MODULE=rt_settings
+   $ redis_tasks scheduler
+
+Verify that you see a successful startup message like::
+
+   16:50:33 INFO  redis_tasks.scheduler: redis_tasks scheduler started
+
+Every time the scheduler enqueues something, you should see a message like this in it's logs::
+
+   16:53:00 INFO      redis_tasks.task: Task tasks.print_len_of_url_content(url='https://readthedocs.org/') [f7d4f223-929c-4eb6-b3a7-66f61dd027b7] enqueued
+
+And the worker will log the following messages when it starts/finishes the task processing::
+
+   16:55:53 INFO      redis_tasks.task: Task tasks.print_len_of_url_content(url='https://readthedocs.org/') [095c3eb9-7073-4ad6-a7cc-16885ee962a4] started
+   12862
+   16:55:53 INFO      redis_tasks.task: Task tasks.print_len_of_url_content(url='https://readthedocs.org/') [095c3eb9-7073-4ad6-a7cc-16885ee962a4] finished
+
+
 .. toctree::
    :maxdepth: 2
-   :caption: Contents:
+   :caption: Usage:
 
+   Command Line Interface <cli>
+   settings
+   scheduler
+   middleware
+   taskgraph
 
 
+.. toctree::
+   :maxdepth: 2
+   :caption: API Docs:
+
+   redis_tasks
+
 Indices and tables
 ==================
 
 * :ref:`genindex`
 * :ref:`modindex`
 * :ref:`search`
+
+
+How it works
+==================
+
+Signal handling
+---------------
+
+In order to assure reliable task processing, ``redis-tasks`` ships with the
+``SIGTERM`` and ``SIGINT`` signals handling, making it compatible with
+platforms like Heroku. When one of the signals is sent, the ``WorkerProcess``
+requests shutdown from the ``WorkHorse`` (a process taking care of a task). The
+``WorkHorse`` ensures that the task was handled properly and interrupts the task
+processing only during the actual task run. The outcome of the task execution is
+determined based on whether the task is reentrant or not. Reentrant tasks are
+re-enqueued to be the first to run after the worker restart, while not reentrant
+tasks are considered failed. Only after that the ``WorkerProcess`` is shut down.

docs/source/middleware.rst

@@ -0,0 +1,58 @@
+The Middleware
+==============
+
+A middleware is a piece of code which is sitting between some kind of request 
+and the actual code that is going to be executed and ``redis-tasks`` is no 
+exception. Just before the actual running, the framework passes the task as first
+class function to the middleware. That means the worker had processed the
+information from the ``Queue`` already, gotten the task and its arguments and created a 
+middleware instance. Then it calls a method of this instance with appropriate 
+information. You are free to decide if, when and how to run the actual task.
+
+In the following example we've created a middleware for delaying the task 
+execution:
+
+.. code:: python
+
+   import time
+
+   class SleepMiddleware:
+      def run_task(self, task, run, args, kwargs):
+         time.sleep(1)
+         run(*args, *kwargs)
+
+The middleware definition is a simple Python class, which must have a method called run_task. The function takes two obligatory parameters - task and run which define respectively the name of the task and the reference to the function which should be executed. As can be seen, the code before the execution of the run call is what is of our interest.
+
+In order to use the previously defined Middleware additional entries to the `settings.py` file are necessary. It is a good practice to put the middleware definitions in a separate directory, e.g. middleware, Then the definition would be:
+
+.. code:: python
+
+   MIDDLEWARE = [
+      "middleware.SleepMiddleware",
+   ]
+
+But wait, what if you want to do stuff right after the task is done? You use the
+same approach as above but instead of declaring ``run_task`` method you name it 
+``process_outcome``. In the example below we are sending a report via an email:
+``process_outcome`` is always called, regardless of the outcome (also if some task raised exceptions for example)
+The ``task`` that produced the outcome is passed in as the first argument to the method.
+
+.. code:: python
+
+   import smtplib
+   from email.message import EmailMessage
+
+   class ReportMiddleware:
+      def process_outcome(self, task, *exc_info):
+         msg = EmailMessage()
+         msg['Subject'] = 'Daily Report'
+         msg['From'] = 'development@test.com'
+         msg['To'] = 'client@test.com'
+
+         s = smtplib.SMTP('localhost')
+         s.send_message(msg)
+         s.quit()
+
+And again you don't have the result of the task here. You should either use
+database, filesystem or even remote resource. By adding it to
+``rt_settings`` you are making it active, just like the first example.

docs/source/modules.rst

@@ -0,0 +1,7 @@
+redis_tasks
+===========
+
+.. toctree::
+   :maxdepth: 4
+
+   redis_tasks

docs/source/redis_tasks.rst

@@ -0,0 +1,110 @@
+redis\_tasks package
+====================
+
+Submodules
+----------
+
+redis\_tasks.cli module
+-----------------------
+
+.. automodule:: redis_tasks.cli
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+redis\_tasks.conf module
+------------------------
+
+.. automodule:: redis_tasks.conf
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+redis\_tasks.defaults module
+----------------------------
+
+.. automodule:: redis_tasks.defaults
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+redis\_tasks.exceptions module
+------------------------------
+
+.. automodule:: redis_tasks.exceptions
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+redis\_tasks.queue module
+-------------------------
+
+.. automodule:: redis_tasks.queue
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+redis\_tasks.registries module
+------------------------------
+
+.. automodule:: redis_tasks.registries
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+redis\_tasks.scheduler module
+-----------------------------
+
+.. automodule:: redis_tasks.scheduler
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+redis\_tasks.smear\_dst module
+------------------------------
+
+.. automodule:: redis_tasks.smear_dst
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+redis\_tasks.task module
+------------------------
+
+.. automodule:: redis_tasks.task
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+redis\_tasks.utils module
+-------------------------
+
+.. automodule:: redis_tasks.utils
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+redis\_tasks.worker module
+--------------------------
+
+.. automodule:: redis_tasks.worker
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+redis\_tasks.worker\_process module
+-----------------------------------
+
+.. automodule:: redis_tasks.worker_process
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+Module contents
+---------------
+
+.. automodule:: redis_tasks
+   :members:
+   :undoc-members:
+   :show-inheritance: