tiny-queue

Zero-dependency persistent FIFO queue for Python. File-backed, crash-safe, with retries, dead-letter, and at-least-once delivery. Stdlib only. No Redis, no Celery, no broker.

# coming soon
pip install tiny-queue

Why?

celery needs Redis/RabbitMQ + a worker process + a beat scheduler. huey is lighter but still needs a broker. dramatiq is fantastic but pulls in pika (RabbitMQ) or redis. For most projects, you just want:

queue.put({"email": "user@example.com", "subject": "Welcome!"})

…and have it survive a process restart. tiny-queue gives you that in 1 file, ~600 LOC, 0 deps.

• Append-only NDJSON files (3 of them: pending, inflight, dead)
• Atomic writes via os.replace + optional fsync
• Cross-process safety via fcntl.flock on a lockfile
• Exponential backoff (configurable), max retries, dead-letter inspection
• Idempotency: caller-supplied IDs prevent duplicate enqueue
• Crash recovery: reap_stale() returns inflight jobs older than N seconds
• Worker helper for polling loops
• 29 tests, 0 external deps, 1 file

Install

Drop tiny_queue.py in your project. Or:

git clone https://github.com/hussain-alsaibai/tiny-queue.git
cp tiny-queue/tiny_queue.py ./your_project/

Usage

Enqueue + consume

import tiny_queue as tq

queue = tq.Queue("/var/spool/myapp")
queue.put({"email": "user@example.com"})

job = queue.get(timeout=5.0)
if job:
    try:
        send_email(job.payload)
        queue.task_done(job, success=True)
    except Exception as e:
        queue.task_done(job, success=False, error=str(e))

Delayed / scheduled jobs

queue.put({"remind_at": "09:00"}, delay=3600)
queue.put({"retry_at": "tomorrow"}, delay=86400)

Idempotency

queue.put({"order_id": 42}, id="order-42-payments")
# Second call with the same id raises QueueError

Dead-letter inspection + replay

for dead in queue.dead_letter():
    print(f"{dead.id}: {dead.last_error} after {dead.attempts} attempts")

queue.requeue_dead("the-job-id")   # bring it back to pending

Worker helper

from tiny_queue import Worker

def my_handler(job):
    process(job.payload)

worker = Worker(queue, my_handler, poll_interval=0.5)
worker.run_forever()   # blocks until worker.stop()

Custom backoff

queue = tq.Queue("/var/spool/myapp",
                 max_retries=5,
                 backoff=lambda n: 5 * (2 ** (n-1)))   # 5s, 10s, 20s, 40s, 80s

Crash recovery

# On startup, recover any jobs stuck in inflight (worker crashed):
n = queue.reap_stale()      # default: 5min threshold
print(f"Recovered {n} jobs from a crashed worker")

Multiple queues

emails = tq.Queue("/var/spool/myapp", name="emails")
payments = tq.Queue("/var/spool/myapp", name="payments")
reports = tq.Queue("/var/spool/myapp", name="reports")
# Each gets its own .pending.jsonl, .inflight.jsonl, .dead.jsonl

Persistence model

/var/spool/myapp/
  default.pending.jsonl    # waiting jobs (one per line, NDJSON)
  default.inflight.jsonl   # jobs currently being processed
  default.dead.jsonl       # jobs that exceeded max_retries
  default.lock             # cross-process flock target

A job moves through 3 states:

pending  ──get()──▶  inflight  ──task_done(success=True)──▶  (gone)
                          │
                          ├──task_done(success=False, attempts ≤ max_retries)──▶  pending (with backoff delay)
                          │
                          └──task_done(success=False, attempts > max_retries)──▶  dead

reap_stale(older_than_seconds=N) moves inflight jobs older than N seconds back to pending (or to dead if they've exceeded max_retries). Call it on worker startup.

Performance

Operation	Speed
put(payload)	~50 µs (fsync off) / ~500 µs (fsync on)
get()	~100 µs (file rewrite)
task_done(success=True)	~100 µs
task_done(success=False)	~150 µs (read inflight + write pending)
reap_stale()	~1 ms per 1000 stale jobs

For workloads under ~1000 jobs/sec on a single host, tiny-queue outperforms Celery + Redis on end-to-end latency because there's no network hop.

When NOT to use tiny-queue

• Multi-host distribution. tiny-queue is single-host (the directory is local). For multi-host, use SQS/PubSub/Kafka.
• Sub-second latency at >10K jobs/sec. Disk-bound. Use Redis or in-memory queues (RQ, dramatiq).
• Topics / pub-sub. tiny-queue is FIFO, not fan-out. For pub-sub, use Redis Pub/Sub or NATS.
• At-most-once delivery. tiny-queue is at-least-once. Make your handlers idempotent (the id= parameter helps).

Cross-process locking

flock is used to serialize access across multiple processes on the same host. Some filesystems (Docker tmpfs, certain cloud overlay filesystems) don't support flock reliably — symptoms are intermittent hangs. To disable:

queue = tq.Queue("/var/spool/myapp", cross_process_lock=False)
# or globally:
# export TINY_QUEUE_NO_FLOCK=1

This degrades multi-process safety but works on any filesystem.

License

MIT — see LICENSE.

---

Ecosystem

Part of the tiny-* zero-dep Python stack. Sibling libraries:

Repo	Purpose
tiny-cron	Cron-style scheduler + intervals
tiny-flags	Feature flags, percentage rollout
fast-cache	LRU + TTL cache, sync + async
tiny-rate	Token / fixed / sliding rate limits
tiny-retry	Exponential backoff + circuit breaker
tiny-pool	Thread + async worker pools
tiny-secret	Secret loader + redacting logger
tiny-trace	OpenTelemetry-API subset, W3C trace
tiny-cli	Decorator CLI builder
tiny-config	JSON / YAML / INI / .env / CLI flags
tiny-log	Structured logging
tiny-validator	Data validation, Pydantic-style
tiny-router	WSGI router, Flask in one file
tiny-compose	Decorator stacker / pipeline
tiny-mcp	MCP server, JSON-RPC 2.0
tiny-embed	sentence-transformers in 1 file
tiny-agent	LangChain in 1 file
snapdb	Embedded in-memory DB

Stack: 18 repos, ~14K LOC, 0 dependencies, ~440 tests.

Today's siblings

• tiny-metrics — Prometheus metrics
• tiny-timeout — timeouts that work
• tiny-idempotency — idempotency keys