(MOT-3944) feat(harness,queue): provision harness turn queue - #464
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (34)
💤 Files with no reviewable changes (1)
🚧 Files skipped from review as they are similar to previous changes (29)
📝 WalkthroughWalkthroughThe queue worker adds durable named function queues with persistent configuration, FIFO and concurrent processing, adapter recovery, runtime queue providers, tracing, and harness startup integration through the dedicated ChangesDurable function queue platform
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant Harness
participant QueueWorker
participant QueueAdapter
participant Function
Harness->>QueueWorker: queue::define harness-turn
QueueWorker->>QueueAdapter: create FIFO consumer
Harness->>QueueWorker: engine::queue::enqueue
QueueWorker->>QueueAdapter: publish durable message
QueueAdapter->>Function: deliver grouped turn
Function-->>QueueAdapter: acknowledge or retry
Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
skill-check — worker0 verified, 42 skipped (no docs/).
Four for four. Nicely done. |
ytallo
changed the title
ytallo
marked this pull request as ready for review
July 10, 2026 13:09
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
queue/src/store.rs (1)
242-256: 🚀 Performance & Scalability | 🔵 Trivial
enqueueperforms blocking file I/O while holding the store mutex.
persist_if_neededruns synchronousstd::fs::write/std::fs::renamewhiledata = shared.inner.lock().awaitis still held. Every concurrent store operation (including consumerdequeue) blocks behind that disk write, and the blocking call also stalls the tokio worker thread. This is the cost of the rollback-on-failure guarantee (which does require holding the lock), but under enqueue load it can meaningfully serialize the queue.Consider persisting via
tokio::task::spawn_blocking(or an async fs) while still under the guard, or acknowledge this is acceptable given expected throughput. Note the other persist paths (dequeue/ack/nack) release the lock before persisting, soenqueueis uniquely affected.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@queue/src/store.rs` around lines 242 - 256, Refactor enqueue so persist_if_needed does not execute synchronous file I/O while holding shared.inner’s mutex. Preserve rollback semantics by snapshotting the previous and updated state, release the guard before persistence, then reacquire it to restore the previous state if persistence fails; alternatively move persistence into spawn_blocking while clearly avoiding Tokio worker-thread blocking. Update enqueue and related persistence handling consistently with dequeue, ack, and nack.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@queue/src/runtime.rs`:
- Around line 424-439: Ensure the prepared adapter is shut down on every
pre-swap failure in the replacement flow. In the replacement block using
`setup_consumers`, explicitly clean up `new_adapter` when setup fails or when
`stop_all_consumers` fails, before propagating the original error; preserve the
existing consumer restart and error-reporting behavior.
---
Nitpick comments:
In `@queue/src/store.rs`:
- Around line 242-256: Refactor enqueue so persist_if_needed does not execute
synchronous file I/O while holding shared.inner’s mutex. Preserve rollback
semantics by snapshotting the previous and updated state, release the guard
before persistence, then reacquire it to restore the previous state if
persistence fails; alternatively move persistence into spawn_blocking while
clearly avoiding Tokio worker-thread blocking. Update enqueue and related
persistence handling consistently with dequeue, ack, and nack.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 1b530321-8080-47d1-ac01-fb3d0d36152c
⛔ Files ignored due to path filters (1)
queue/Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (42)
database/tests/e2e/config.yamlharness/Makefileharness/README.mdharness/engine.config.yamlharness/iii.worker.yamlharness/queue.config.yamlharness/src/functions/mod.rsharness/src/functions/turn.rsharness/src/lib.rsharness/src/locks.rsharness/src/main.rsharness/src/queue.rsharness/src/turn_loop.rsharness/tests/manifest.rsqueue/Cargo.tomlqueue/README.mdqueue/iii.worker.yamlqueue/src/adapter.rsqueue/src/adapters/builtin.rsqueue/src/adapters/memory.rsqueue/src/adapters/rabbitmq/adapter.rsqueue/src/adapters/rabbitmq/topology.rsqueue/src/adapters/redis.rsqueue/src/boot.rsqueue/src/config.rsqueue/src/configuration.rsqueue/src/functions.rsqueue/src/lib.rsqueue/src/main.rsqueue/src/manifest.rsqueue/src/observability.rsqueue/src/runtime.rsqueue/src/store.rsqueue/src/trigger.rsqueue/tests/e2e_durability.rsqueue/tests/e2e_rabbitmq.rsshell/tests/e2e/config-jailed.yamlshell/tests/e2e/config.yamlstorage/tests/e2e/config.all.yamlstorage/tests/e2e/config.yamltech-specs/2026-06-agentic/harness.mdtelegram-bot/iii.worker.yaml
💤 Files with no reviewable changes (5)
- harness/engine.config.yaml
- storage/tests/e2e/config.yaml
- shell/tests/e2e/config-jailed.yaml
- shell/tests/e2e/config.yaml
- storage/tests/e2e/config.all.yaml
Comment on lines
+424
to
+439
| if let Some((new_adapter, name)) = replacement { | ||
| let old_adapter = self.adapter.current().await; | ||
| self.setup_consumers(new_adapter.clone(), &next.queue_configs) | ||
| .await?; | ||
| if let Err(error) = self.stop_all_consumers(old_adapter.clone()).await { | ||
| let restart = self | ||
| .setup_and_start_consumers(old_adapter, &old.queue_configs) | ||
| .await; | ||
| let restart_summary = result_summary(&restart); | ||
| if let Ok(consumers) = restart { | ||
| self.replace_all_consumers(consumers).await?; | ||
| } | ||
| anyhow::bail!( | ||
| "cannot stop previous queue consumers before adapter replacement: {error}; consumer restart: {restart_summary}" | ||
| ); | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Close the prepared adapter on pre-swap failures.
If setup succeeds but stopping old consumers fails, new_adapter is returned without shutdown(). The direct ? on Line 426 has the same leak path. Repeated failed reconfigurations can accumulate adapter connections/resources.
Proposed cleanup
- self.setup_consumers(new_adapter.clone(), &next.queue_configs)
- .await?;
+ if let Err(error) = self.setup_consumers(new_adapter.clone(), &next.queue_configs).await {
+ new_adapter.shutdown().await;
+ return Err(error);
+ }
if let Err(error) = self.stop_all_consumers(old_adapter.clone()).await {
+ new_adapter.shutdown().await;
let restart = self📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| if let Some((new_adapter, name)) = replacement { | |
| let old_adapter = self.adapter.current().await; | |
| self.setup_consumers(new_adapter.clone(), &next.queue_configs) | |
| .await?; | |
| if let Err(error) = self.stop_all_consumers(old_adapter.clone()).await { | |
| let restart = self | |
| .setup_and_start_consumers(old_adapter, &old.queue_configs) | |
| .await; | |
| let restart_summary = result_summary(&restart); | |
| if let Ok(consumers) = restart { | |
| self.replace_all_consumers(consumers).await?; | |
| } | |
| anyhow::bail!( | |
| "cannot stop previous queue consumers before adapter replacement: {error}; consumer restart: {restart_summary}" | |
| ); | |
| } | |
| if let Some((new_adapter, name)) = replacement { | |
| let old_adapter = self.adapter.current().await; | |
| if let Err(error) = self | |
| .setup_consumers(new_adapter.clone(), &next.queue_configs) | |
| .await | |
| { | |
| new_adapter.shutdown().await; | |
| return Err(error); | |
| } | |
| if let Err(error) = self.stop_all_consumers(old_adapter.clone()).await { | |
| new_adapter.shutdown().await; | |
| let restart = self | |
| .setup_and_start_consumers(old_adapter, &old.queue_configs) | |
| .await; | |
| let restart_summary = result_summary(&restart); | |
| if let Ok(consumers) = restart { | |
| self.replace_all_consumers(consumers).await?; | |
| } | |
| anyhow::bail!( | |
| "cannot stop previous queue consumers before adapter replacement: {error}; consumer restart: {restart_summary}" | |
| ); | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@queue/src/runtime.rs` around lines 424 - 439, Ensure the prepared adapter is
shut down on every pre-swap failure in the replacement flow. In the replacement
block using `setup_consumers`, explicitly clean up `new_adapter` when setup
fails or when `stop_all_consumers` fails, before propagating the original error;
preserve the existing consumer restart and error-reporting behavior.
guibeira
approved these changes
Jul 10, 2026
guibeira
left a comment
guibeira
left a comment
Collaborator
There was a problem hiding this comment.
Can we update only the harness and queue? i would like to first update those, and them we can run some tests. If everything works fine we can come back and update them.
| use tracing_opentelemetry::OpenTelemetryLayer; | ||
| use tracing_subscriber::registry::LookupSpan; | ||
|
|
||
| const TRACER_NAME: &str = "iii-queue"; |
Collaborator
There was a problem hiding this comment.
let's switch to just queue
ytallo
force-pushed
the
feat/boot-provisioned-harness-queues
branch
from
July 13, 2026 16:57
3cc60a2 to
63b2bd0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes MOT-3944
Summary
Behavior
Validation
Companion engine PR: iii-hq/iii#1950
Summary by CodeRabbit
queue::define/engine::queue::enqueue).