Redis rq by BehRoooz · Pull Request #13 · BehRoooz/proseqgo

BehRoooz · 2026-07-23T12:20:23Z

Summary

Migrate embedding and training async jobs from SQLite + in-process poll loops to durable Postgres (proseqgo_jobs) and Redis/RQ.
APIs enqueue jobs and return immediately; dedicated embedding-worker / trainer-worker containers run GPU work, update Postgres progress/status, and recover orphaned jobs after worker crash.
Compose adds AOF Redis, a Postgres init hook, and job/env wiring.
Monitoring scrapes workers and redis_exporter, with Grafana panels and alerts for Redis down and worker backlog.
Smoke/acceptance checks (including crash recovery) live under tests/smoke/.

… failures from retryable runtime errors so RQ Retry only requeues blips.

Keep the JobStore API but store lifecycle and artifact metadata in proseqgo_jobs instead of SQLite.

Centralize Queue.enqueue(process_job), job_timeout, Retry, and the failure callback for durable job ids.

Create the Postgres row, enqueue to Redis, and expose queue metrics; remove startup/shutdown of the polling daemon.

Run embedding/training work under RQ, update Postgres progress/status, and raise transient errors for retry instead of polling SQLite.

Starts a Prometheus metrics HTTP server, requeues orphaned started jobs (from a previous SIGKILL), then runs an RQ worker with graceful SIGTERM.

Describe the API→Postgres→Redis→worker flow, crash recovery, and env vars after the queue migration.

Expose JOBS_DATABASE_URL, REDIS_URL, queue name, timeouts, and artifact-root overrides for the async worker stack.

Separate non-retryable failures from retryable runtime errors so RQ Retry only requeues blips.

Keep the JobStore API but store lifecycle and artifact metadata in proseqgo_jobs instead of SQLite.

Write queued rows to Postgres, enqueue process_job, refresh queue metrics, and drop the in-process daemon loop.

Centralize Queue.enqueue with job_timeout, Retry, and the failure callback for durable job ids.

Serve worker metrics, requeue started jobs after SIGKILL, and run the embedding-jobs RQ worker.

Drive embedding work from RQ, update Postgres progress/status, and raise transient errors for retry.

Describe API→Postgres→Redis→worker flow, crash recovery, and storage layout after the queue migration.

Create the durable job-history DB on first Postgres volume init, separate from MLflow's mlflow database.

Mount the jobs DB init script, run Redis with AOF, split embedding/training APIs from GPU workers, and wire JOBS_DATABASE_URL plus Redis for async dispatch.

Add Prometheus targets for embedding-worker, trainer-worker, and Redis so queue and broker health are visible beside the APIs.

Extend target-down coverage to workers/redis_exporter and add critical rules for embedding-worker down with a non-empty RQ queue.

Show cafa5_rq_queue_length and redis_memory_used_bytes on the domain pipelines dashboard after the Redis/RQ migration.

Relocate embedding/MLflow smoke scripts out of scripts/ so pipeline CLIs stay separate from acceptance checks.

Add the smoke/acceptance directory to the repo tree so verification scripts are easy to find.

BehRoooz added 26 commits July 23, 2026 00:03

document Postgres job store, RQ workers, and queue alerts

47c7483

add redis, rq, and psycopg for async jobs

436ceff

document Redis/RQ and proseqgo_jobs env vars

d90200f

add Redis, RQ, and Postgres jobs settings

fb6ed14

Add permanent vs transient job error types. Distinguish non-retryable…

fc52756

… failures from retryable runtime errors so RQ Retry only requeues blips.

refactor(job-store): back job history with Postgres

804b39a

Keep the JobStore API but store lifecycle and artifact metadata in proseqgo_jobs instead of SQLite.

feat: enqueue jobs on Redis/RQ with timeout and retry

e1818f1

Centralize Queue.enqueue(process_job), job_timeout, Retry, and the failure callback for durable job ids.

feat(api): enqueue RQ jobs and drop in-process worker thread

af35a80

Create the Postgres row, enqueue to Redis, and expose queue metrics; remove startup/shutdown of the polling daemon.

refactor(worker): make process_job an RQ entrypoint

455ae0e

Run embedding/training work under RQ, update Postgres progress/status, and raise transient errors for retry instead of polling SQLite.

feat: add RQ worker entrypoint for embedding jobs.

9e9db64

Starts a Prometheus metrics HTTP server, requeues orphaned started jobs (from a previous SIGKILL), then runs an RQ worker with graceful SIGTERM.

docs: document Postgres job store and RQ workers

c76fe10

Describe the API→Postgres→Redis→worker flow, crash recovery, and env vars after the queue migration.

chore(embedding-api): add Redis, RQ, and Postgres jobs config

375d2a5

Expose JOBS_DATABASE_URL, REDIS_URL, queue name, timeouts, and artifact-root overrides for the async worker stack.

feat(embedding-api): add permanent and transient job errors

9deee42

Separate non-retryable failures from retryable runtime errors so RQ Retry only requeues blips.

refactor: persist job history in Postgres

55c57b9

Keep the JobStore API but store lifecycle and artifact metadata in proseqgo_jobs instead of SQLite.

feat: enqueue RQ jobs and remove poll-thread worker

977500f

Write queued rows to Postgres, enqueue process_job, refresh queue metrics, and drop the in-process daemon loop.

feat: enqueue embedding jobs on Redis/RQ

a409998

Centralize Queue.enqueue with job_timeout, Retry, and the failure callback for durable job ids.

feat: add RQ worker entrypoint with orphan recovery

f90d307

Serve worker metrics, requeue started jobs after SIGKILL, and run the embedding-jobs RQ worker.

refactor: make process_job an RQ entrypoint

de3ac51

Drive embedding work from RQ, update Postgres progress/status, and raise transient errors for retry.

docs: document Postgres job store and RQ workers

474971f

Describe API→Postgres→Redis→worker flow, crash recovery, and storage layout after the queue migration.

chore(postgres): add init script for proseqgo_jobs database

6cddb7c

Create the durable job-history DB on first Postgres volume init, separate from MLflow's mlflow database.

feat: add Redis, RQ workers, and proseqgo_jobs wiring

b423d62

Mount the jobs DB init script, run Redis with AOF, split embedding/training APIs from GPU workers, and wire JOBS_DATABASE_URL plus Redis for async dispatch.

chore(monitoring): scrape RQ workers and redis_exporter

e872383

Add Prometheus targets for embedding-worker, trainer-worker, and Redis so queue and broker health are visible beside the APIs.

feat(monitoring): alert on Redis down and worker backlog

6acb48d

Extend target-down coverage to workers/redis_exporter and add critical rules for embedding-worker down with a non-empty RQ queue.

feat(monitoring): add RQ queue and Redis panels to Grafana

d30a452

Show cafa5_rq_queue_length and redis_memory_used_bytes on the domain pipelines dashboard after the Redis/RQ migration.

chore(tests): move Compose smoke checks into tests/smoke

138b427

Relocate embedding/MLflow smoke scripts out of scripts/ so pipeline CLIs stay separate from acceptance checks.

docs: document tests/smoke in the project layout

8f35318

Add the smoke/acceptance directory to the repo tree so verification scripts are easy to find.

BehRoooz merged commit c7d2f4f into main Jul 23, 2026

Navigation Menu

Redis rq - #13

Redis rq#13
BehRoooz merged 26 commits into
mainfrom
redis-rq

BehRoooz commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

BehRoooz commented Jul 23, 2026

Summary

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant