[FEAT]: Non-Blocking Async Extraction, Retry, and Observability for /forms/fill

[Acuspeedster]

---

name: 🚀 Feature Request
about: Suggest an idea or a new capability for FireForm.
title: "[FEAT]: Non-blocking async LLM extraction with streaming, retry, and job orchestration"
labels: enhancement
assignees: ''

📝 Description

The current implementation of POST /forms/fill performs LLM extraction synchronously using requests.post() within the FastAPI request lifecycle.

Execution chain:

forms.py → controller.fill_form() → llm.get_data() → requests.post()

Because FastAPI runs on an asyncio event loop (via Uvicorn), this synchronous HTTP call blocks the event loop thread for the entire duration of Ollama inference.

Operational impact:

• Each field extraction may take 2–5 seconds on CPU inference.
• With sequential main_loop() (N fields), total blocking time scales linearly.
• A 10-field form can hold the event loop for 20–50 seconds.
• Even main_loop_batch() still blocks for the full duration of a single inference call.

This results in:

• Event loop starvation
• Inability to serve concurrent requests
• 30+ second client response latency
• No retry mechanism for missed fields
• No per-field confidence visibility
• No observable progress during extraction
• No fault tolerance for partial success scenarios

---

💡 Rationale

PR #151 (currently open) proposes schema enforcement using Ollama’s format parameter and dynamically generated Pydantic models.

While that meaningfully improves output structural reliability, it does not address:

• Transport-layer blocking
• Concurrency limitations
• Retry logic for null/missed fields
• Event loop starvation
• Client-side observability
• Job orchestration

In high-impact public-sector deployments, a system must not only return structured output but must also:

• Remain responsive under load
• Provide progressive feedback
• Handle partial failures gracefully
• Support operational monitoring

This issue proposes a transport and orchestration layer redesign to meet those requirements.

---

🛠️ Proposed Solution

1️⃣ Asynchronous Concurrent Extraction

• Replace requests with httpx.AsyncClient
• Introduce async_extract_all_streaming() in src/llm.py
• Launch per-field extraction tasks via asyncio.create_task()
• Collect results using asyncio.as_completed()

This ensures:

• Wall-clock time is bounded by the slowest field
• Partial results become available immediately
• Event loop remains free to serve other requests

If 9 fields resolve in 3 seconds and 1 resolves in 8 seconds:

• Client receives 9 results at second 3
• Final result at second 8

Prior implementations return nothing until second 8.

---

2️⃣ Two-Pass Auto-Retry Mechanism

After Pass 1 completes:

• Any field returning None enters Pass 2.
• _build_targeted_prompt() constructs a focused single-field prompt.
• Explicit instruction: return -1 if not found.
• Retry tasks launched concurrently.

Confidence scoring:

Confidence	Meaning
high	Extracted in Pass 1
medium	Recovered in Pass 2
low	Missing after both passes

This provides deterministic field-level reliability reporting.

---

3️⃣ Non-Blocking PDF Generation

• Introduce fill_form_with_data() in filler.py
• Offload pdfrw operations to a ThreadPoolExecutor via loop.run_in_executor()
• Prevent CPU-bound PDF writes from blocking event loop

Correctness improvement:

• None values written as empty strings instead of literal "None"

---

4️⃣ New Client-Observable API Surfaces

POST /forms/fill/stream

• Returns text/event-stream
• Emits one Server-Sent Event per field as soon as it resolves
• Event payload includes:
  • field
  • value
  • confidence
  • phase
• Final complete event includes:
  • submission_id
  • output_pdf_path

POST /forms/fill/async

• Returns 202 with job_id
• Full extraction pipeline runs as FastAPI BackgroundTask

GET /forms/jobs/{job_id}

Returns:

• status (pending, running, complete, failed)
• partial_results
• field_confidence
• output_pdf_path
• error_message

---

5️⃣ Database Orchestration Layer

Introduce FillJob SQLModel table:

• UUID primary key
• template_id
• input_text
• status
• output_pdf_path
• partial_results (JSON)
• field_confidence (JSON)
• error_message
• created_at

Repository functions:

• create_job
• get_job
• update_job (**kwargs for partial updates)

This enables incremental persistence of extraction progress.

---

✅ Acceptance Criteria

• Event loop no longer blocked by extraction
• Concurrent per-field extraction implemented
• Two-pass retry operational
• Field-level confidence scoring implemented
• SSE streaming endpoint functional
• Async job queue endpoint functional
• FillJob model implemented with incremental updates
• Original /forms/fill endpoint preserved
• Comprehensive test coverage

---

📌 Additional Context

This redesign is transport- and orchestration-focused.

It is fully compatible with schema enforcement approaches such as PR #151 and does not conflict with model-level validation strategies.

[Acuspeedster]

Hi @vharkins1 @marcvergees

I’ve opened PR #153 as a complementary architectural enhancement to this issue.

PR #151 meaningfully improves structural reliability by leveraging Ollama’s format parameter and dynamic schema enforcement. However, it retains a synchronous HTTP transport model, which continues to block the FastAPI event loop during inference. Under CPU bound conditions, this effectively serializes requests and limits scalability : an architectural constraint independent of schema correctness.

PR #153 focuses on the execution and orchestration layer rather than output structure. It replaces blocking requests calls with httpx.AsyncClient and concurrent per field extraction (asyncio.create_task + as_completed), eliminating event loop starvation and bounding latency by the slowest field instead of the cumulative duration of all fields.

Beyond non blocking transport, it introduces:

• Two-pass targeted retry for missed fields
• Deterministic per field confidence scoring
• SSE streaming for progressive extraction visibility
• Background job orchestration with incremental database persistence
• Non blocking PDF generation via run_in_executor

The intent is to align FireForm with production grade async architecture principles: responsiveness under load, observability, fault tolerance, and deployability in resource-constrained environments.

This design is orthogonal and fully compatible with the schema enforcement strategy proposed in #151 : the async transport layer can pass the same format payload without modification.

From a long term sustainability perspective, this separates:

• Schema reliability (model layer)
• Transport scalability (async execution layer)
• Operational observability (streaming + job state persistence)

I believe this layered separation strengthens FireForm’s architectural clarity and positions it well for real world deployment scenarios.

[marcvergees]

That's a very good point. We already had it in mind to develop it. The project is in an initial development/setup. Asap this turns out to be smthg more professional, it will have a "better" frontend, which will obviously imply handling this kind of asynchronous queries you talk.

[Acuspeedster]

Hi @marcvergees

That’s great to hear glad this aligns with the direction you had in mind.
I’ve opened PR #158 which implements this batch orchestration model, including canonical single-pass extraction and concurrent template mapping to avoid redundant LLM calls across multiple forms.
If you get a chance, I’d really appreciate your feedback on the approach especially around the separation between extraction, template mapping, and batch/audit persistence. Happy to iterate to better align with the project’s design direction.

[utkarshqz]

Hi @marcvergees — flagging that PR #210 and PR #241 address the event loop starvation described here at the model layer.

The primary source of blocking was the per-field extraction loop — N sequential Ollama calls. PR #210 replaced this with a single batch prompt covering all fields simultaneously, reducing Ollama calls from O(N) to O(1). On a 7-field form, blocking time drops from ~35s to ~5s.

PR #241 extends this to the batch level — one LLM call now covers all templates in a batch request, with deterministic Python mapping instead of secondary LLM calls, eliminating hallucination risk at the mapping layer.

These don't implement the async transport layer proposed here, but they eliminate the scaling bottleneck at the source before it reaches the transport layer.

Happy to discuss the approach!

[Acuspeedster]

@utkarshqz

Thanks for sharing your approach.
For context, Issue #152 and PR #153 were opened earlier to address the event-loop blocking and scalability concerns in /forms/fill by introducing an async transport layer (httpx.AsyncClient), concurrent extraction (asyncio.create_task + as_completed), retry logic, streaming observability, and background job orchestration.
Over the past few weeks there have been several parallel PRs opened that partially re-implement aspects of the same issues with different approaches. This makes coordination and review more difficult and fragments the discussion across multiple implementations.
If you'd like to contribute to this feature, it would likely be more productive to collaborate directly on the existing PR rather than opening separate implementations addressing the same issue. You’re very welcome to commit to the branch for PR #153 so we can iterate on the solution together.
It would also help if we refrain from opening redundant PRs for the same feature and instead converge on a single implementation, while letting the maintainers decide if the architectural direction should change.
From a design perspective, the changes in PR #210 / #241 reduce the number of model calls, but they do not address the transport-layer concern raised in this issue namely that synchronous requests.post() still blocks the FastAPI event loop and prevents true concurrency under load. PR #153 focuses specifically on that layer (async transport, streaming progress, and job orchestration), which is orthogonal to prompt batching.
@vharkins1 @marcvergees happy to follow whichever workflow and architectural direction the maintainers prefer here the goal is simply to keep the effort collaborative and aligned with the long-term design of FireForm.
Also please check on him he has been doing the same with my many prs and issues.

[utkarshqz]

Hi @Acuspeedster, @marcvergees, @vharkins1 —

Want to be precise rather than defensive here.

PR #153's async transport layer is a correct fix for a real problem and I am not disputing it. Async transport, SSE streaming, and job orchestration are the right direction for a production multi-user deployment. I have this planned as an explicit next step.

What PR #210 addresses is a different layer entirely — the number of model calls. The original codebase called Ollama once per field, meaning N sequential calls for an N-field form. PR #210 replaced this with a single batch prompt covering all fields simultaneously.

Benchmark on a 7-field form run locally:

Original approach (7 calls): 152s
PR #210 (1 call): 59s
LLM calls reduced: 7 → 1
Speedup: 2.6x on CPU-only hardware

Beyond speed, the per-field approach produced hallucinations on 2 of 7 fields — the model returned meta-commentary instead of values because it lacked full form context. The batch approach extracted all 7 fields correctly because the model sees the complete form at once.

These are genuinely orthogonal fixes. PR #153 addresses how requests travel through the event loop. PR #210 addresses how many requests are made and how accurately values are extracted. Both matter. Neither makes the other redundant.

FireForm is a UN Digital Public Good intended for global deployment — at that scale both the transport layer and model efficiency layer need to be right. I see PR #153 and PR #210 as complementary, not competing.

@marcvergees @vharkins1 — happy to follow whatever direction and workflow you prefer here.

hallucination in 'incident_type' , 'badge_number' ;