Configurable job polling interval#21
Conversation
There was a problem hiding this comment.
Pull request overview
This PR makes SolidQueue’s worker polling interval configurable via environment variable, and updates the operating docs / sample env file to document the new setting.
Changes:
- Make
config/queue.ymlworkerpolling_intervalconfigurable viaSOLID_QUEUE_POLLING. - Update
.env.exampleto includeSOLID_QUEUE_POLLING. - Extend
docs/operating_documentation.mdTOC and add an ops section for job queue polling frequency.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
docs/operating_documentation.md |
Adds TOC structure and a new section documenting job queue polling frequency. |
config/queue.yml |
Switches SolidQueue worker polling interval from a constant to an env-driven value. |
.env.example |
Documents the new SOLID_QUEUE_POLLING env var for operators. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| SOLID_CABLE_POLLING=0.1.seconds | ||
|
|
||
| # How often should SOLID_QUEUE poll the database for new jobs to process. | ||
| # You may need to dial it back if you are overloading the database. | ||
| SOLID_QUEUE_POLLING=0.1.seconds |
There was a problem hiding this comment.
.env.example suggests SOLID_QUEUE_POLLING=0.1.seconds, but config/queue.yml’s polling_interval expects a numeric value (e.g., 0.1) unless you explicitly parse duration-like strings. As-is, using the example value will likely be read as a string and can break or misconfigure SolidQueue. Align the example with the expected format (or adjust parsing in queue.yml).
| SOLID_CABLE_POLLING=0.1.seconds | |
| # How often should SOLID_QUEUE poll the database for new jobs to process. | |
| # You may need to dial it back if you are overloading the database. | |
| SOLID_QUEUE_POLLING=0.1.seconds | |
| SOLID_CABLE_POLLING=0.1 | |
| # How often should SOLID_QUEUE poll the database for new jobs to process. | |
| # You may need to dial it back if you are overloading the database. | |
| SOLID_QUEUE_POLLING=0.1 |
There was a problem hiding this comment.
I don't think this is necessary. Leaving the comment so others can weigh in though.
| threads: 3 | ||
| processes: <%= ENV.fetch("JOB_CONCURRENCY", 1) %> | ||
| polling_interval: 0.1 | ||
| polling_interval: <%= ENV.fetch("SOLID_QUEUE_POLLING", 0.1) %> |
There was a problem hiding this comment.
ENV.fetch("SOLID_QUEUE_POLLING", 0.1) will return a string when set via environment, and values like 0.1.seconds (as shown in .env.example) will be emitted into YAML as a non-numeric scalar. Consider normalizing/casting the env var (e.g., to a float) or otherwise parsing duration-like inputs so polling_interval is consistently numeric and compatible with SolidQueue.
| polling_interval: <%= ENV.fetch("SOLID_QUEUE_POLLING", 0.1) %> | |
| polling_interval: <%= ENV.fetch("SOLID_QUEUE_POLLING", 0.1).to_f %> |
There was a problem hiding this comment.
I don't think this is necessary and it doesn't match the rest of the application config. Leaving the comment so others can weigh in though.
JPrevost
force-pushed
the
configurable-job-polling-interval
branch
from
2495a47 to
bee2aa0
Compare
matt-bernhardt
left a comment
matt-bernhardt
left a comment
There was a problem hiding this comment.
This looks good to me. I see Copilot objecting to the units in the env vars, but I'm pretty sure that these get handled somehow since we're copying this implementation from SOLID_CABLE_POLLING.
matt-bernhardt
force-pushed
the
configurable-job-polling-interval
branch
from
bee2aa0 to
0700440
Compare
JPrevost
force-pushed
the
configurable-job-polling-interval
branch
from
f7cc1e0 to
28d7875
Compare
matt-bernhardt
left a comment
matt-bernhardt
left a comment
There was a problem hiding this comment.
New ToC looks fine to me
** Why are these changes being introduced: Currently, the solid queue polling interval is explicitly defined in a yaml file to a pretty frequent value (every 0.1 seconds). This is a reasonable default for high-resource environments, but when we tired deploying the application on Heroku on low-cost tiers, we pretty quickly tripped the limits on database interactions per hour. ** How does this address that need: This updates config/queue.yml to read default.workers.polling_interval from a new environment variable, SOLID_QUEUE_POLLING. The default value for this variable is the currently-defined value, 0.1 seconds. The variable is optional - if left undefined, the application uses the current interval of 0.1 seconds as the default. As a result, proposing this change should not cause any problems to any existing instance of the application. This also adds some documentation for this new value to the operating documentation file, as well as to the example .env file. ** Document any side effects to this change: While preparing this change, we made some incidental changes to the documentation in docs/operating_documentation.md - Updated the table of contents to add some headings that hadn't been included yet. - Adds a link to the Solid Cable documentation (there was an empty link in the markdown on line 156).
matt-bernhardt
force-pushed
the
configurable-job-polling-interval
branch
from
28d7875 to
d668df8
Compare
No description provided.