Complete Queue lifecycle safety, durability, and Laravel parity - #457
Merged
Conversation
Record the settled ownership model for pooled queues, deferred dispatch, SQS overflow storage, poison payloads, worker lifecycle, and eager inspection. Capture the approved Laravel parity boundaries, anti-overengineering constraints, performance analysis, regression matrix, and full validation workflow so the implementation remains auditable across compactions.
Document that framework configuration is shallow-merged, with mergeable options reserved for named groups whose individual entries replace matching defaults. Align the configuration bootstrap and package provider docblocks with that contract, and add regressions proving named queue connections merge independently while other nested settings replace wholesale.
Give deferred queue operations an explicit owner-first dispatcher so transaction callbacks never retain a borrowed queue beyond its lease. Configure and clear the dispatcher at pool boundaries, preserve the immediate path for size-one pools, and cover logical connection reuse, rollback cleanup, callback ownership, and end-to-end transaction behavior.
Resolve one transaction manager for each synchronous, background, and deferred after-commit operation and pass that exact owner through rollback and commit registration. Keep execution-time payload behavior intact, return a predictable null for deferred work, and expand the fixtures to model a real active transaction without permissive cache behavior.
Make Beanstalk size include ready, delayed, and reserved jobs from one tube-stats read, matching the active workload represented by other drivers. Retain buried jobs as dead-letter state, correct the contract wording, and verify the total and individual metrics without adding another network request.
Expose eager pending, delayed, and reserved inspection for database-backed queues while preserving queue, attempt, and record identifiers. Validate inspected payloads through the shared Queue boundary and cover status classification, exact queue filtering, ordering, and invalid-payload diagnostics.
Accept static credentials, callable or object providers, and the current ECS and instance provider names while memoizing provider resolution. Keep pool fingerprints explicit for dynamic configuration, omit non-client options, and fail unsupported provider names with a descriptive exception.
Publish SQS jobs through ordered SendMessageBatch chunks while retaining exact queue events, FIFO options, and one deferred lease per batch. Store oversized bodies only for the chunk being sent, preserve ambiguous deliveries, clean explicit rejections, and enforce terminal delete, lease release, and overflow cleanup precedence.
Decode each consumed payload once, cache either the validated array or the exact InvalidPayloadException, and reject malformed or incomplete job envelopes consistently. Preserve raw evidence in diagnostics and extend the database rollback contract to the level-aware behavior already required by timeout cleanup.
Move Redis attempts validation into the existing Lua reservation round trip so malformed JSON, scalar payloads, invalid attempts, and raw zero values are reserved without mutation or loss. Carry the reservation attempt count into RedisJob, make identifiers lazy and non-throwing, add eager raw-mode inspection, and verify exact removal members against real Redis.
Translate malformed queue payloads into the shared Queue exception and contain only that named failure after the underlying queue action succeeds. Preserve valid entries in mixed migrations, skip unavailable timer identifiers, and keep unrelated Horizon listener and transport failures observable.
Stop Telescope failed-job recording when the shared Queue payload validator rejects external data, avoiding a second unsafe payload read. Keep all ordinary watcher failures and valid payload behavior unchanged, with focused coverage for malformed job telemetry.
Add truthful timeout, idle, quiet-stop, completion, pause, and stop metadata while keeping concurrent jobs in daemon-owned state rather than worker singletons. Use one monotonic clock, batch paused-queue reads, preserve in-flight work across stop paths, reset boot-only controls between tests, and distinguish admitted from completed jobs.
Add quiet-period stopping, explicit pause capability errors, current monitor formatting, and complete verbose and JSON worker output. Preserve exact zero values and command option precedence while carrying the real connection, queue, elapsed time, memory, and lifecycle context through each operational path.
Align both database failed-job providers with the current constructor order and expose their configured table. Preserve exact non-null connection and queue filters, including string zero, and cover lookup, pruning, counting, and construction behavior for both identifier strategies.
Write failed-job snapshots through a same-directory temporary file under the existing lock, require complete I/O, preserve file modes where available, and rename atomically. Clean unpublished temporary files without replacing the primary failure and cover false reads, short writes, rename failures, visibility, mode preservation, and cleanup.
Make failed-driver selection explicit and fail unsupported values rather than silently constructing an invalid provider. Keep file path and limit defaults at the provider that consumes them, preserve replace-whole failed and batching configuration, and verify every supported driver and optional file setting.
Add current JSON listing output and preserve malformed failed-job evidence until a retry is successfully re-enqueued. Decode retry payloads strictly, retain arbitrary and FIFO fields, support direct and pooled SQS options, and cover valid, missing, malformed, and failed requeue paths.
Pass the actual framework job through unique-until-processing cleanup and accept enum-backed queue, connection, limiter, and overlap identifiers. Normalize identifiers once, preserve integer-backed zero and explicit zero delays, and retain serialized rate-limiter behavior across both cache implementations.
Add Laravel-shaped Release middleware and callable exception backoff for both cache-backed throttling implementations. Keep conditional release APIs, exact backoff resolution, and Redis behavior aligned with current upstream while covering each public branch through integration tests.
Route DateInterval releases through the existing seconds-until helper already used for date-time values. Keep integer and absolute-date behavior intact and make fake assertions observe the same normalized delay contract.
Add concrete pending, delayed, and reserved inspection to inspectable queues, delegate through failover and facade surfaces, and return empty collections for unsupported drivers. Keep the optional capability out of the core contract and verify eager result behavior and facade metadata without adding a second abstraction.
Record immediate, delayed, and reserved fake jobs as disjoint state while preserving one ordered push history. Carry delays through partial fakes, route push and later through one operation boundary, and add instance-owned before and after hooks whose state publishes only after successful operations.
Reject transformed Redis connections before SafeScan begins because transformed tuples cannot preserve native scan cursors or deletion members. Keep FlushByPattern on the same raw boundary, update the public usage guidance, and verify no scan or deletion is attempted with an incompatible connection.
Reconcile case-insensitive phpredis method signatures, preserve mixed SETNX and HSETNX values, and keep reverse score bounds in native order. Separate pooled connection lifecycle discard from Redis transaction discard, document held-connection ownership, regenerate facade metadata, and cover transformed, raw, and integration behavior.
Remove channel enumeration from the core broadcaster contract because conforming broadcasters are not required to implement it. Keep the concrete command capability through one narrowly documented dynamic call instead of introducing a one-consumer capability contract.
Widen the notification factory boundary to the mixed notifiable shapes already supported by the dispatcher and implementations. Keep sendNow at the factory's two-argument contract and remove the false dependency on a particular collection representation.
Declare Queue's direct context and filesystem requirements so its split package is complete in isolation. Use the shared PHP and Artisan binary helpers in the listener and verify the package manifest rather than relying on root-monorepo dependency leakage.
Document SQS overflow storage, inspection, release middleware, worker lifecycle controls, failed-job output, and malformed-payload handling in the existing task-first style. Record the deliberate owner-first protected callback difference needed for pooled deferred work and keep operational memory and dedicated-store warnings at the user-facing surfaces.
Mark the Queue package complete and record the verified lifecycle, durability, parity, performance, and cross-package findings in the audit ledger. Update dependency routing and completed-package revalidation so future audits can trace every contract and source change made by this work unit.
|
Important Review skippedToo many files! This PR contains 127 files, which is 27 over the limit of 100. To get a review, narrow the scope: Upgrade to a paid plan to raise the limit. This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (127)
You can disable this status message by setting the ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Greptile SummaryThe PR completes queue lifecycle ownership, payload durability, worker behavior, SQS batching and overflow storage, Redis inspection, failed-job persistence, and Laravel-facing API parity.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains.
|
| Filename | Overview |
|---|---|
| src/queue/src/SqsQueue.php | Adds SQS batching and overflow-payload lifecycle handling; the prior single-message cleanup concern is invalid because a thrown SDK request cannot establish that no retry attempt was accepted. |
| src/queue/src/Queue.php | Refactors enqueue ownership and prepared after-commit operations so deferred work does not retain borrowed queue leases. |
| src/queue/src/QueuePoolProxy.php | Supplies owner-first dispatch through fresh pooled leases and clears lease-specific state before returning objects. |
| src/queue/src/Worker.php | Extends timeout, idle, quiet-stop, pause, draining, and completion metadata behavior for concurrent workers. |
| src/queue/src/RedisQueue.php | Adds durable malformed-payload reservation and eager operational inspection behavior. |
| src/queue/src/Failed/FileFailedJobProvider.php | Publishes failed-job records through checked same-directory temporary files and atomic replacement. |
| src/redis/src/Operations/SafeScan.php | Rejects transformed Redis connections before scan-based operations can inspect or delete data. |
| src/support/src/Testing/Fakes/QueueFake.php | Expands fake queue history and hook behavior to match the updated queue-facing API. |
Reviews (2): Last reviewed commit: "docs(queue): explain ambiguous SQS overf..." | Re-trigger Greptile
Collaborator
Author
|
@greptileai I reviewed the SQS overflow finding against the queue implementation and the installed AWS SDK. No runtime change is appropriate. The finding assumes a thrown single-message request is definitively rejected, but the SDK retries connection and transient failures. A surfaced error can therefore follow an earlier attempt that SQS accepted after its response was lost. There is no supported signal proving that no attempt published the pointer, so deleting its body on failure risks losing a live job. The existing batch cleanup remains correct: explicit per-entry I pushed ff49060 to document both call-site invariants and add concise operator guidance for dedicated, monitored, non-evicting overflow storage. Focused SQS coverage, formatting, static analysis, and whitespace validation are green. |
Document why an AWS SDK failure cannot prove a single-message publication was rejected, and why batch cleanup is limited to explicit per-entry failures. Warn operators that non-expiring overflow bodies may outlive failed, expired, or cleared messages, and require a dedicated monitored store that does not evict live payloads. Runtime queue behavior remains unchanged.
binaryfire
force-pushed
the
audit/queue-lifecycles-parity
branch
from
July 28, 2026 14:39
e5ebe5b to
ff49060
Compare
|
The reasoning is correct and the documentation approach is the right call. The key distinction holds: the AWS SDK's built-in retry behavior means a surfaced exception from The batch path's cleanup of explicit The inline comments in |
|
The reasoning is sound and the documentation commit is the right resolution. The core invariant holds: an AWS SDK exception on The asymmetry between the two call sites is correctly handled:
The inline comment in The finding is resolved as documented. No runtime change is warranted. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR completes the Queue lifecycle, durability, and current Laravel parity work.
The central change is ownership. Deferred work no longer retains a borrowed queue object across a database transaction. The prepared payload survives until commit, but the physical queue lease does not. A successful commit borrows a fresh queue for the operation, while the immediate path continues to use the current lease. This keeps size-one pools live and prevents one coroutine from publishing through an object another coroutine has borrowed.
The rest of the work follows that same rule: every queue, connection, payload, transaction callback, temporary file, and cache entry has one clear owner and one terminal cleanup path.
This PR also:
For more details, see:
docs/plans/2026-07-28-0353-queue-pooling-payload-durability-and-current-laravel-parity.mdPooled dispatch
After-commit dispatch now captures a prepared operation rather than a borrowed queue. The pool proxy supplies an owner-first dispatcher when a queue is checked out and clears it before the object returns to the pool.
This distinction matters:
Synchronous, background, and deferred queues resolve one transaction manager and use that same owner for registration and rollback cleanup.
SQS
Bulk publication uses
SendMessageBatchwith ordered chunks bounded by SQS count and byte limits. After-commit bulk work uses one callback and one lease for the batch rather than one checkout per job.Oversized message bodies are written only for the chunk about to be sent. The cleanup rules distinguish the outcomes that matter:
Credential providers now support the current static, callable, ECS, and instance forms. FIFO group and deduplication identifiers preserve exact values, including
"0".Payload durability
Queue payloads are decoded and validated once per job. The job caches either the validated payload or the exact
InvalidPayloadException, so every later accessor observes the same result.Malformed jobs now follow a terminal path:
Redis reservation performs attempts validation inside the existing Lua round trip. Invalid JSON, scalar payloads, missing or nonnumeric attempts, and raw
"0"values are reserved with their exact bytes intact. That prevents loss betweenLPOPand reservation and preserves the real member needed for removal.Horizon and Telescope contain only the named invalid-payload telemetry failure after the queue action succeeds. Ordinary application, listener, database, and transport exceptions continue to propagate.
Worker lifecycle
Workers now distinguish admitted jobs from completed jobs and use one monotonic clock for runtime deadlines, timeout monitoring, and quiet-period stopping.
The worker surface includes:
--timeout=0;WorkerIdleevents;--stop-when-empty-for;The implementation does not add process-global current-job state. Concurrent jobs remain daemon-owned entries, which is safe for Hypervel's coroutine worker.
Queue APIs and storage
This ports or completes current Laravel-shaped behavior for:
DateIntervalreleases;Inspection remains a concrete capability rather than being added to the core Queue contract. It intentionally returns eager collections and is documented as an operational API that should not be used against very large backlogs in latency-sensitive paths.
The file failed-job provider now writes a complete same-directory temporary file and atomically renames it under the existing lock. It preserves the current mode where available, removes unpublished temporary files, and never lets cleanup replace the primary persistence failure.
Redis and contracts
Queue inspection consumes Redis scans through a raw phpredis connection. Transformed scan results have a different shape and are rejected before scanning or deleting anything.
Redis command metadata now gives case-insensitive method names the same truthful signature, preserves native score-range argument order, and separates Redis transaction discard from pooled connection lifecycle discard.
The contract updates remove optional channel enumeration from the core broadcaster contract and allow Notification factories to accept every notifiable shape their implementations already support. Queue package metadata now declares the dependencies required by its split package.
Performance
The ordinary dispatch path remains one pool checkout and does not allocate a dispatcher closure per operation. The retained dispatcher is one bounded closure per worker-cached logical proxy.
Real after-commit work adds one necessary checkout at commit instead of retaining a lease for the duration of the transaction. SQS bulk reduces network requests by sending bounded batches. Valid payloads decode once. Redis poison handling stays inside the existing Lua round trip.
Paused-queue lookup uses one cache
many()call. This replaces sequential reads for empty or later-priority queues; a busy first-priority queue transfers a few additional boolean values but does not add another network round trip. The read size depends on configured queue names, not backlog size.Eager inspection is not on the worker or request hot path. It is explicitly documented as memory-sensitive operational behavior.
Compatibility
Public APIs follow current Laravel naming and shapes unless Hypervel's pooling or coroutine runtime requires a deliberate difference.
The protected
enqueueUsing()callback receives the operation's queue owner first. This is documented in the Queue package README because it is the boundary that prevents deferred pooled work from retaining a borrowed object.Hypervel's concurrent worker, pooling, queue-route, debounce, payload-context, SafeScan, and Redis transformation behavior remain intact.
Validation
Focused Queue, Redis, Horizon, Telescope, database, SQS, worker, middleware, failed-job, facade, package metadata, and integration coverage was run throughout the work.
The final repository gate was run through
composer fix, covering formatting, static analysis, and the full parallel test suite. The final diff also passes whitespace validation.