feat!: Postgres-backed queue - #826
Merged
Merged
Conversation
# Conflicts: # Cargo.lock
Remaining work: - Align sidekiq code with pg code - Update book - Add doc comments - Add some tests for ser/de of worker config structs - Check test coverage and add any major missing coverage
spencewenski
force-pushed
the
gh-758-pg-queue
branch
from
July 5, 2025 08:01
7d739de to
714bf21
Compare
spencewenski
added a commit
that referenced
this pull request
Jul 13, 2025
## 🤖 New release
* `roadster`: 0.8.0-alpha -> 0.8.0-alpha.1 (⚠ API breaking changes)
### ⚠ `roadster` breaking changes
```text
--- failure enum_missing: pub enum removed or renamed ---
Description:
A publicly-visible enum cannot be imported by its prior path. A `pub use` may have been removed, or the enum itself may have been renamed or removed entirely.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.41.0/src/lints/enum_missing.ron
Failed in:
enum roadster::config::service::worker::sidekiq::BalanceStrategy, previously in file /tmp/.tmpNberRT/roadster/src/config/service/worker/sidekiq/mod.rs:138
enum roadster::config::service::worker::sidekiq::StaleCleanUpBehavior, previously in file /tmp/.tmpNberRT/roadster/src/config/service/worker/sidekiq/mod.rs:88
--- failure feature_missing: package feature removed or renamed ---
Description:
A feature has been removed from this package's Cargo.toml. This will break downstream crates which enable that feature.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#cargo-feature-remove
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.41.0/src/lints/feature_missing.ron
Failed in:
feature sidekiq in the package's Cargo.toml
--- failure module_missing: pub module removed or renamed ---
Description:
A publicly-visible module cannot be imported by its prior path. A `pub use` may have been removed, or the module may have been renamed, removed, or made non-public.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.41.0/src/lints/module_missing.ron
Failed in:
mod roadster::service::worker::sidekiq::roadster_worker, previously in file /tmp/.tmpNberRT/roadster/src/service/worker/sidekiq/roadster_worker.rs:1
mod roadster::service::worker::sidekiq, previously in file /tmp/.tmpNberRT/roadster/src/service/worker/sidekiq/mod.rs:1
mod roadster::service::worker::sidekiq::app_worker, previously in file /tmp/.tmpNberRT/roadster/src/service/worker/sidekiq/app_worker.rs:1
mod roadster::service::worker::sidekiq::service, previously in file /tmp/.tmpNberRT/roadster/src/service/worker/sidekiq/service.rs:1
mod roadster::service::worker::sidekiq::builder, previously in file /tmp/.tmpNberRT/roadster/src/service/worker/sidekiq/builder.rs:1
--- failure struct_missing: pub struct removed or renamed ---
Description:
A publicly-visible struct cannot be imported by its prior path. A `pub use` may have been removed, or the struct itself may have been renamed or removed entirely.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.41.0/src/lints/struct_missing.ron
Failed in:
struct roadster::config::service::worker::sidekiq::SidekiqServiceConfig, previously in file /tmp/.tmpNberRT/roadster/src/config/service/worker/sidekiq/mod.rs:16
struct roadster::service::worker::sidekiq::builder::SidekiqWorkerServiceBuilder, previously in file /tmp/.tmpNberRT/roadster/src/service/worker/sidekiq/builder.rs:22
struct roadster::config::service::worker::sidekiq::QueueConfig, previously in file /tmp/.tmpNberRT/roadster/src/config/service/worker/sidekiq/mod.rs:161
struct roadster::service::worker::sidekiq::app_worker::AppWorkerConfig, previously in file /tmp/.tmpNberRT/roadster/src/service/worker/sidekiq/app_worker.rs:22
struct roadster::service::worker::sidekiq::service::SidekiqWorkerService, previously in file /tmp/.tmpNberRT/roadster/src/service/worker/sidekiq/service.rs:52
--- failure struct_pub_field_missing: pub struct's pub field removed or renamed ---
Description:
A publicly-visible struct has at least one public field that is no longer available under its prior name. It may have been renamed or removed entirely.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.41.0/src/lints/struct_pub_field_missing.ron
Failed in:
field connect_timeout of struct Database, previously in file /tmp/.tmpNberRT/roadster/src/config/database/mod.rs:22
field connect_lazy of struct Database, previously in file /tmp/.tmpNberRT/roadster/src/config/database/mod.rs:27
field acquire_timeout of struct Database, previously in file /tmp/.tmpNberRT/roadster/src/config/database/mod.rs:31
field idle_timeout of struct Database, previously in file /tmp/.tmpNberRT/roadster/src/config/database/mod.rs:34
field max_lifetime of struct Database, previously in file /tmp/.tmpNberRT/roadster/src/config/database/mod.rs:37
field min_connections of struct Database, previously in file /tmp/.tmpNberRT/roadster/src/config/database/mod.rs:40
field max_connections of struct Database, previously in file /tmp/.tmpNberRT/roadster/src/config/database/mod.rs:42
field test_on_checkout of struct Database, previously in file /tmp/.tmpNberRT/roadster/src/config/database/mod.rs:45
field retry_connection of struct Database, previously in file /tmp/.tmpNberRT/roadster/src/config/database/mod.rs:50
field sidekiq of struct Service, previously in file /tmp/.tmpNberRT/roadster/src/config/service/mod.rs:38
--- failure trait_missing: pub trait removed or renamed ---
Description:
A publicly-visible trait cannot be imported by its prior path. A `pub use` may have been removed, or the trait itself may have been renamed or removed entirely.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.41.0/src/lints/trait_missing.ron
Failed in:
trait roadster::service::worker::sidekiq::app_worker::AppWorker, previously in file /tmp/.tmpNberRT/roadster/src/service/worker/sidekiq/app_worker.rs:48
trait roadster::service::AppServiceAsAny, previously in file /tmp/.tmpNberRT/roadster/src/service/mod.rs:126
trait roadster::service::AppService, previously in file /tmp/.tmpNberRT/roadster/src/service/mod.rs:35
trait roadster::service::AppServiceBuilder, previously in file /tmp/.tmpNberRT/roadster/src/service/mod.rs:70
--- failure type_allows_fewer_generic_type_params: type now allows fewer generic type parameters ---
Description:
A type now allows fewer generic type parameters than it used to. Uses of this type that supplied all previously-supported generic types will be broken.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#trait-new-parameter-no-default
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.41.0/src/lints/type_allows_fewer_generic_type_params.ron
Failed in:
Struct ServiceRegistry allows 2 -> 1 generic types in /tmp/.tmpDDYcDy/roadster/src/service/registry.rs:33
Struct FunctionService allows 4 -> 3 generic types in /tmp/.tmpDDYcDy/roadster/src/service/function/service.rs:44
```
<details><summary><i><b>Changelog</b></i></summary><p>
<blockquote>
##
[0.8.0-alpha.1](roadster-v0.8.0-alpha...roadster-v0.8.0-alpha.1)
- 2025-07-13
### Added
- [**breaking**] Postgres-backed queue
([#826](#826))
This release adds support for using a Postgres database as the backing
store for an async queue in addition to our
existing Sidekiq (Redis-backed) support. This release contains a lot of
breaking changes.
- New features
- Add `Worker` trait to allow enqueuing and handling of jobs
- Add `Enqueuer` trait to decouple application code from the logic
required to enqueue jobs to different queue
backends. This allows easily switching the queue backend if needed
without needing to change any application
code. This also allows consumers to provide the own queue backends
instead of relying on what's built-in to
Roadster.
- Add `PgProcessor` and a wrapping `PgWorkerService` to process jobs
enqueued to the Postgres backend
- Allow adding custom values (extensions) to the `AppContext`. This is
useful for external libraries that might
need custom state.
- New variants were added to the `Error` enum for the new features and
dependencies
- Breaking changes
- Refactored our Sidekiq support to align more closely with the code
added to support Postgres backed queues.
- Add a `SidekiqProcessor` has a very similar API to the `PgProcessor`
- Replace the `SidekiqWorkerService` with a new impl that simply wraps
the `SidekiqProcessor`
- Roadster's `Worker` trait should be used instead of [the one from
`rusty-sidekiq`](https://docs.rs/rusty-sidekiq/latest/sidekiq/trait.Worker.html)
- A lot of the configuration structs were renamed, moved, or
consolidated
- Rename `AppService` to `Service` and remove the `App` type parameter
- Duration config fields were all aligned to millisecond precision, with
the exception of the `max-age` config
for the cache control middleware, as the HTTP header uses second
precision.
</blockquote>
</p></details>
---
This PR was generated with
[release-plz](https://github.com/release-plz/release-plz/).
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Spencer Ferris <3319370+spencewenski@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds support for using a Postgres database as the backing store for an async queue in addition to our existing Sidekiq (Redis-backed) support. This is a large PR with a lot of breaking changes.
New features
Workertrait to allow enqueuing and handling of jobsEnqueuertrait to decouple application code from the logic required to enqueue jobs to different queue backends. This allows easily switching the queue backend if needed without needing to change any application code. This also allows consumers to provide the own queue backends instead of relying on what's built-in to Roadster.PgProcessorand a wrappingPgWorkerServiceto process jobs enqueued to the Postgres backendAppContext. This is useful for external libraries that might need custom state.Errorenum for the new features and dependenciesBreaking changes
SidekiqProcessorhas a very similar API to thePgProcessorSidekiqWorkerServicewith a new impl that simply wraps theSidekiqProcessorWorkertrait should be used instead of the one fromrusty-sidekiqAppServicetoServiceand remove theApptype parametermax-ageconfig for the cache control middleware, as the HTTP header uses second precision.Closes #758