Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Turnip Task Scheduler: Like celery but tastier.
Python
branch: master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
bin
examples
turnip
.gitignore
README.md
setup.cfg
setup.py

README.md

Turnip

Like celeryq but tastier.

Warning: This software is in a state of being useful but unstable. You should acquire a good understanding of the codebase before using this.

Tasty

The goal is to be simple, small, and self-contained.

  • Built on SQLAlchemy. Everything is contained in one table named turnip_task. This means you can mess around with an SQLite storage during development, and push it out with PostgreSQL in production. Less moving pieces.
  • Supports recurring tasks with a cron-like scheduling syntax and and followup tasks.
  • Multiple workers should work in theory (untested, may need some transaction tweaking before it'll work reliably).

Usage

Usage: turnip --engine=ENGINE COMMAND [ARGS ...]

Turnip task management. Tastier than celery.

Commands:

    run
        Process tasks forever.

    prune
        Delete old tasks with states 'complete' or 'error'.

    revive
        Set 'started' tasks to 'pending'.

    rebuild [FN]
        Delete 'pending' tasks (and run FN if given).

    list
        List tasks.


Options:
  -h, --help            show this help message and exit
  -v, --verbose         Enable verbose output. Use twice to enable debug
                        output.
  -e ENGINE, --engine=ENGINE
                        Database engine to use where tasks are stored.

Schema

This will get you an idea of what defines a task.

class Task(BaseModel):
    __tablename__ = 'turnip_task'

    id = Column(types.Integer, primary_key=True)
    time_created = Column(types.DateTime, default=datetime.now, nullable=False)
    time_updated = Column(types.DateTime, onupdate=datetime.now)

    time_wait_until = Column(types.DateTime, default=datetime.now, nullable=False)
    seconds_elapsed = Column(types.Float)

    # Tag tasks that depend on a specific resource so they can be queried 
    # Example: An error from a 'twitter-api' tag would trigger a 1hr delay on all pending tasks of that group.
    resource_group = Column(types.String(32))

    # A ``method`` is a Python object path to a callable that takes ``params``.
    method = Column(types.Text, nullable=False)
    params = Column(types.PickleType, default=dict, nullable=False)

    # Random string (like a uuid)
    lock_key = Column(types.LargeBinary(16))

    recurring_cron = Column(types.String(128)) # Cron-like string for recurring. If not set, wont recur.

    # Set when the current task is related to another task.
    parent_task_id = Column(types.Integer, ForeignKey('task.id'))
    parent_task = orm.relationship('Task', remote_side=id)
    parent_type = Column(mytypes.Enum(['master', 'retry', 'recurring']), default='master', nullable=False)

    state = Column(mytypes.Enum(['pending', 'started', 'completed', 'failed']), default='pending', nullable=False, index=True)

Get started

Head over to the examples directory.

TODO

  • Environment setup hook.
  • Paster plugin for loading Pylons environment natively.

License

MIT

Something went wrong with that request. Please try again.