Umbrella Cursor Loop Mode

Umbrella: Cursor / Claim Loop Mode (#100)

Issue: noetl/ai-meta#100 Status: CLOSED — GKE confirmation in progress Primary repo: noetl/server

Goal

Port the claim-based cursor loop (loop.spec.mode: cursor) to the Rust orchestrator so the patient-fetch flow (test_pft_flow_v2) and the production state_report_generation playbook run on the Rust stack without the server rejecting the playbook at parse time.

Acceptance criteria:

• loop.spec.mode: cursor accepted at playbook register time.
• Entry → claim → fan-out body per row → re-claim until 0 rows → drain arc fires.
• test_pft_flow_v2 runs all_passed: true against the throttling/error-injecting paginated-api test server on kind.
• Four parity gaps surfaced during validation also resolved.

Design

The cursor loop is orchestrator-driven — no worker holds a slot between claim frames. Each claim executes as a normal postgres tool command; the orchestrator drives the loop.

Entry
  ├── emit step.enter
  └── issue claim command (frame 0)
        │
        ▼
Claim result (worker executes claim SQL, returns rows)
  ├── 0 rows  → DRAIN: complete step, route arcs with event.name = loop.done
  └── K rows  → fan out body per row (bounded by frame.row_concurrency)
                    │ all body rows complete
                    └── issue claim command (next frame)

The claim SQL controls the lease via FOR UPDATE SKIP LOCKED … RETURNING. {{ __frame_max_rows }} is injected from loop.spec.frame.max_rows. Stale-row reclaiming is the playbook's own responsibility via a reclaim-stale CTE in the claim SQL.

The four parity fixes

1. mode: cursor loop engine — LoopMode::Cursor, CursorClaim, FrameSpec; orchestrator entry hook + cursor-drive block + reconstruct_cursor_frames; StepInfo.is_cursor so claim/body completions don't prematurely complete the step; drain via __cursor_drained. noetl/server#196 (server v3.8.0).

2. output namespace — arc when: expressions and step set: blocks may now reference the just-completed step's result as {{ output.<field> }}. Without it, the PFT's load_next_facility output-gated arcs stalled. noetl/server#196.

3. Cursor loop-back re-entry — a loop-back re-run (e.g., mark_<type>_done → fetch_<type> when not all types done) resets frame tracking so the re-run's frame 0 does not merge with the prior drained run's frame history. noetl/server#196.

4. Postgres ---comment splitter — an apostrophe in a -- comment swallowed the trailing ;, merging statements and causing "cannot insert multiple commands into a prepared statement" at setup_facility_work. noetl/tools#66 (noetl-tools v3.10.1), pulled by noetl/worker#88.

Validation

test_pft_flow_v2 — the patient-fetch flow that mirrors the production state_report_generation playbook — ran end-to-end on the Rust stack on the local kind cluster against the throttling/error-injecting paginated-api test server:

all_passed: true
  assessments:   5/5
  conditions:    5/5
  medications:   5/5
  vital_signs:   5/5
  demographics:  5/5

The test server injects throttling errors and paginated responses, so the green result proves the playbook handles retries and the cursor loop across multiple frames.

Recent activity

Date	What
2026-06-15	noetl/server#196 merged → v3.8.0 (cursor loop + output namespace + loop-back re-entry).
2026-06-15	noetl/tools#66 merged → v3.10.1 (postgres -- comment splitter fix).
2026-06-15	noetl/worker#88 merged — pulls noetl-tools v3.10.1.
2026-06-15	test_pft_flow_v2 all_passed: true on kind against paginated-api.
2026-06-15	#100 closed on ai-meta; board status → Done.

Next concrete steps

• Confirm the fix on GKE (deploy server v3.8.0 + worker pulling noetl-tools v3.10.1; re-run the PFT fixture against the prod paginated-api endpoint).
• After GKE confirmation close out any remaining pointer bumps.

Related

• noetl/ai-meta#100 — the umbrella issue
• noetl/server wiki — cursor-loop-mode — deep-dive reference page
• noetl/tools wiki — postgres splitter fix context
• Execution model — why the orchestrator drives the loop