Python worker for the Faktory project
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
examples cleanup Feb 21, 2018
faktory bumped version to 0.4.0 Feb 21, 2018
.gitignore version 0.2 Nov 12, 2017
CHANGELOG.md bumped version to 0.4.0 Feb 21, 2018
LICENSE Initial commit Oct 26, 2017
README.md Update README.md Jul 30, 2018
docker-compose.yml Send the 'priority' key to Faktory + Updated docker-compose.yml Jan 2, 2018
setup.cfg version 0.2 Nov 12, 2017
setup.py bumped version to 0.4.0 Feb 21, 2018

README.md

Python worker for Faktory

Overview

This project is a complete worker and client implementation for the Faktory job server. You can use it to either consume jobs from Faktory or push jobs to the Faktory server to be processed.

Requires Python 3.5+.

Supported Faktory Versions

0.5.0
0.6.0+
0.7.0+
0.8.0

Features

  • Creating a worker to run jobs from Faktory
  • Concurrency (with multiple processes or threads with the use_threads=True option)
  • Pushing work to Faktory from Python (with retries, custom metadata and scheduled support)
  • Pushing exception / errors from Python back up to Faktory
  • Sends worker status back to Faktory
  • Supports quiet and teminate from the Faktory web UI
  • Password authentication
  • TLS support
  • Graceful worker shutdown (ctrl-c will allow 15s for pending jobs to finish)

Todo

  • Documentation (in progress, help would be appreciated)
  • Tests (in progress, help would be appreciated)
  • Django integration (./manage.py runworker and app/tasks.py support)

Installation

pip install faktory

Pushing Work to Faktory

There is a client context manager that you can use like this:

import faktory

with faktory.connection() as client:
    client.queue('test', args=(1, 2))
    client.queue('test', args=(4, 5), queue='other')

test doesn't need to be implemented by the Python worker, it can be any of the available worker implementations.

Worker Example

Sample worker:

from faktory import Worker

def your_function(x, y):
    return x + y

w = Worker(queues=['default'], concurrency=1)
w.register('test', your_function)

w.run()  # runs until control-c or worker shutdown from Faktory web UI

Connection to Faktory

faktory_worker_python uses this format for the Faktory URL:

tcp://:password@localhost:7419

or with TLS:

tcp+tls://:password@localhost:7419

If the environment variable FAKTORY_URL is set, that is used. Otherwise you can pass the server URL in to the Worker or Client constructor, like this:

w = Worker(faktory="tcp://localhost:7419")

Logging

The worker users Python's built in logging module, which you can enable like this before calling .run():

import logging
logging.basicConfig(level=logging.DEBUG)