Like celeryq but tastier.
Warning: This software is in a state of being useful but unstable. You should acquire a good understanding of the codebase before using this.
The goal is to be simple, small, and self-contained.
- Built on SQLAlchemy. Everything is contained in one table named
turnip_task. This means you can mess around with an SQLite storage during development, and push it out with PostgreSQL in production. Less moving pieces. - Supports recurring tasks with a cron-like scheduling syntax and and followup tasks.
- Multiple workers should work in theory (untested, may need some transaction tweaking before it'll work reliably).
Usage: turnip --engine=ENGINE COMMAND [ARGS ...]
Turnip task management. Tastier than celery.
Commands:
run
Process tasks forever.
prune
Delete old tasks with states 'complete' or 'error'.
revive
Set 'started' tasks to 'pending'.
rebuild [FN]
Delete 'pending' tasks (and run FN if given).
list
List tasks.
Options:
-h, --help show this help message and exit
-v, --verbose Enable verbose output. Use twice to enable debug
output.
-e ENGINE, --engine=ENGINE
Database engine to use where tasks are stored.
This will get you an idea of what defines a task.
class Task(BaseModel):
__tablename__ = 'turnip_task'
id = Column(types.Integer, primary_key=True)
time_created = Column(types.DateTime, default=datetime.now, nullable=False)
time_updated = Column(types.DateTime, onupdate=datetime.now)
time_wait_until = Column(types.DateTime, default=datetime.now, nullable=False)
seconds_elapsed = Column(types.Float)
# Tag tasks that depend on a specific resource so they can be queried
# Example: An error from a 'twitter-api' tag would trigger a 1hr delay on all pending tasks of that group.
resource_group = Column(types.String(32))
# A ``method`` is a Python object path to a callable that takes ``params``.
method = Column(types.Text, nullable=False)
params = Column(types.PickleType, default=dict, nullable=False)
# Random string (like a uuid)
lock_key = Column(types.LargeBinary(16))
recurring_cron = Column(types.String(128)) # Cron-like string for recurring. If not set, wont recur.
# Set when the current task is related to another task.
parent_task_id = Column(types.Integer, ForeignKey('task.id'))
parent_task = orm.relationship('Task', remote_side=id)
parent_type = Column(mytypes.Enum(['master', 'retry', 'recurring']), default='master', nullable=False)
state = Column(mytypes.Enum(['pending', 'started', 'completed', 'failed']), default='pending', nullable=False, index=True)
Head over to the examples directory.
- Environment setup hook.
- Paster plugin for loading Pylons environment natively.
MIT