Haskell-bee

A lightweight, type-safe library for implementing asynchronous job workers in Haskell.

This is a library for implementing asynchronous workers which can fetch jobs from a (configurable) broker.

You can think of it as a simple Haskell rewrite of Celery.

Why Haskell? Haskell's type system makes async job processing much safer and easier to reason about compared to Python's Celery. Combined with Haskell's excellent concurrency story, you get robust job processing with compile-time guarantees.

Key features:

• Multiple broker backends (PostgreSQL/PGMQ, Redis, STM)
• Configurable timeout and retry strategies
• Exception-safe job processing
• Support for delayed/periodic tasks

Getting started

The quickest way to understand how to use haskell-bee is to check out the ./demo app, which shows practical examples of job processing.

If you're interested in reading about various design aspects, see design notes.

Testing

Tests are generic, they are bundled as a library in haskell-bee-tests.

Each broker implements these generic tests in its own test suite.

PGMQ

Start postgresql:

podman run --rm -p 5432:5432 -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=postgres postgres

Then run tests:

cabal v2-test haskell-bee-pgmq --test-show-details=streaming

Redis

Start redis:

podman run --rm -it -p 6379:6379 redis:latest

Then run tests:

cabal v2-test haskell-bee-redis --test-show-details=streaming

STM

No need to start the broker, just run the tests:

cabal v2-test haskell-bee-stm --test-show-details=streaming

Things to watch out for

Exceptions

haskell-bee uses exceptions to detect issues with jobs and trigger timeouts. Be careful when catching exceptions in your code.

Problem: If your code contains something like:

runSomeAction `catch` (\(e :: SomeException) -> whatever)

and whatever doesn't re-throw e, your job will continue running without noticing Timeout exceptions.

Solution: Always specialize your exception handling to only catch the specific exceptions you want to handle:

runSomeAction `catch` (\(e :: MySpecificException) -> handleMyException e)

Other work

Haskell Job Queues: An Ultimate Guide (2020)

• hasql-queue This implements a custom PostgreSQL schema. On top of that it proviedes some simple strategies for running tasks.
• immortal-queue This is a library that you can use to build an asynchronous worker pool. The Broker can be configurable.
• job Recent addition with Broker, Worker, job metadata. Supports retrying, re-nicing.
• odd-jobs PostgreSQL-backed, has an admin UI, seems mature.
• broccoli Rust library, a message queue system with configurable brokers. Similar to Celery. Has configurable retry strategies.

Brokers

These libraries implement mostly the queueing mechanism only, without any job metadata structure like in haskell-bee. They fall into the Broker category of haskell-bee.

• postgresql-simple-queue (it uses a custom queueing mechanism; it is more like a Broker in haskell-bee)
• stm-queue (in-memory only)
• mongodb-queue (again, similar to our Broker)
• amazonka-sqs (again, a Broker, based on Amazon SQS. Might be worth integrating this in the future)
• redis-job-queue (worth investigating the sorted set approach)
• poolboy (in-memory only)

Credit

All credit goes to the Gargantext team. This work was done as part of my contract at ISC-PIF CNRS.