-
Notifications
You must be signed in to change notification settings - Fork 206
Open
Description
In my current job, we have an open-source monorepo featuring a steering website, doing the data modelling and more in Ruby, linked to ETL processes ran in Python. We currently use faktory to communicate and send jobs between the two languages, but its limitations brought us to consider a move to SolidQueue, including writing a Python library worker+client/enqueuer.
I've been through the SolidQueue code and did not encounter difficulties to understand the main concepts, but would appreciate a short cross-check of my TODO list / plan to make sure I don't oversee something. I could see that there was some interest in #286 and #679 for that.
Python library to interact with SolidQueue
Basic choices
- Not reimplementing everything, no scheduler, no real dispatcher, almost just a worker.
A thin dispatcher layer for threads, signals, config and co is probably required. - Also no recurring tasks support in a first time
- Recommend to use language-specific queues to not mix Python and Ruby jobs
Worker feature set
- Get jobs by queue
- Get all jobs probably not in a first time
- Feature to have a Process (or Interpreter in Python 3.14+) Pool for concurrency
- Support for threads maybe in a second time
- Sync and async support?
- Add retry logic mimicking ActiveRecord that SolidQueue is relying on?
Client/enqueue feature set
- Possibility to enqueue a job.
- Bulk enqueue capacity maybe later
Metadata
Metadata
Assignees
Labels
No labels