feat(sqs): rate-limited async message move tasks with cancellation#765
Merged
vieiralucas merged 2 commits intomainfrom Apr 25, 2026
Merged
feat(sqs): rate-limited async message move tasks with cancellation#765vieiralucas merged 2 commits intomainfrom
vieiralucas merged 2 commits intomainfrom
Conversation
Replaces the synchronous-only message-move flow with a rate-aware async mover so callers can observe a moving task in RUNNING state, cancel it mid-flight, and rely on MaxNumberOfMessagesPerSecond as a real ceiling. - StartMessageMoveTask: when MaxNumberOfMessagesPerSecond is set, mint a RUNNING task with a real UUID-derived TaskHandle and spawn a background mover that drains one message per tick at the requested rate. Without a rate, keep the existing fast-path that drains synchronously to COMPLETED. - CancelMessageMoveTask: signal cancellation via an AtomicBool flag on the task; the mover observes it on its next tick and finalizes the task as CANCELLED. The request flips status to CANCELLING immediately so listings reflect the in-flight transition. - Background mover updates messages_moved on each successful move so ListMessageMoveTasks reports real progress; finalizes COMPLETED on drain or FAILED if the source/destination queue disappears. - Critical: the mover acquires the state write lock for exactly one move per tick and drops it before sleeping or finalizing. finalize_move_task also takes the write lock, so holding it across the finalize call would deadlock with parking_lot's non-reentrant RwLock. - MessageMoveTask carries the cancel flag in a #[serde(skip)] field so on-disk snapshots are unaffected; restores get a fresh non-cancelled flag. Adds 2 E2E tests (rate-limited drain + mid-flight cancel) and migrates one unit test to tokio::test now that the rate path spawns.
There was a problem hiding this comment.
1 issue found across 3 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="crates/fakecloud-sqs/src/service.rs">
<violation number="1" location="crates/fakecloud-sqs/src/service.rs:852">
P1: Message loss when target queue disappears mid-flight: `pop_front()` removes the message from the source before confirming the destination exists. If the target queue was deleted (or the URL can't be resolved), `msg` is dropped and the message is silently lost. Push the message back onto the source queue on failure, or verify the target before popping.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
Cubic flagged P1: pop_front removed the message from source before verifying the destination still existed, dropping the message if the destination queue was deleted between ticks. Refactored mover loop to: - Verify both source and destination queues exist before popping. - Pop the source message only after target validation. - If get_mut on the resolved target races and returns None (queue deleted between resolve and commit), push the message back to the front of the source queue so it is retried or remains durable. - Bump the round-robin index only after a successful commit so a failed attempt doesn't skip a target.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the synchronous-only message-move flow with a rate-aware async mover so callers can observe a moving task in RUNNING state, cancel it mid-flight, and rely on
MaxNumberOfMessagesPerSecondas a real ceiling. Part of batch 8/10 of the gap-fill plan.StartMessageMoveTaskwith a rate spawns a background mover that drains one message per tick.CancelMessageMoveTaskflips anAtomicBoolcancellation flag and marks the taskCANCELLING; the mover observes the flag on its next tick and finalizesCANCELLED.messages_movedper move soListMessageMoveTasksreflects real progress.COMPLETED(no behavior change for callers that didn't set a rate).#[serde(skip)]field onMessageMoveTask; snapshots unaffected.Critical detail noted in the commit body: the mover acquires the state write lock for exactly one move per tick and drops it before sleeping or finalizing —
finalize_move_taskalso takes the write lock and parking_lot's RwLock is non-reentrant.Test plan
cargo test -p fakecloud-sqs --lib)crates/fakecloud-e2e/tests/sqs_message_move.rs(rate-limited drain + mid-flight cancel)cargo clippy --workspace --all-targets -- -D warningscleancargo fmt --allcleanSummary by cubic
Adds a rate-aware async worker for moving SQS messages with observable progress and mid-flight cancellation. Also fixes an edge case to prevent message loss if the destination queue disappears during a move.
New Features
StartMessageMoveTaskwith a rate spawns a background worker that moves one message per tick and updatesmessages_moved.CancelMessageMoveTasksets a cancellation flag and marksCANCELLING; the worker finalizesCANCELLEDon the next tick.ListMessageMoveTasks; tasks end asCOMPLETEDon drain orFAILEDif queues/tasks disappear.COMPLETED.#[serde(skip)]onMessageMoveTask, so snapshots remain unchanged.Bug Fixes
Written for commit 911e906. Summary will update on new commits.