DocInsightAI — Document Q&A

Upload PDFs, ask questions, get answers grounded only in the uploaded documents, with citations pointing to the exact source page. If the documents don't contain the answer, the app says so instead of inventing one.

Built as a scoping exercise: SPEC.md defines what v1 is, SPEC_V2.md documents everything deliberately excluded and how it would be built.

Stack

Layer	Choice	Why
Backend	Ruby on Rails 8 (API mode)	Very readable code. I believe that team preference is undervalued metric when choosing stack. More familiarity means faster processing and less surprises during code reviews, plus ruby maps cleanly into python - Claude Code is good at such translations
Jobs	ActiveJob + SolidQueue	DB-backed queue, no Redis; async status lifecycle is idiomatic.
RAG	Hybrid retrieval: vector embeddings + full-text search	Great for both: semantic, but also exact text search
Database	PostgreSQL + pgvector (neighbor)	One store for documents, jobs, chunks, vectors, and full-text search.
PDF parsing	pdf-reader	Pure Ruby, per-page text extraction - page provenance is trivially correct.
Embeddings	bge-m3 via Ollama (1024d)	Self-hosted, battle-tested, fits pgvector HNSW natively.
Generation	Any OpenAI-compatible endpoint via env	One client, one seam: local Ollama (gpt-oss:20b) for dev, claude API for production.
Frontend	Vue 3 + Vite, single page	Explicit JSON API boundary, minimal surface, no framework overhead.
Architecture	Monorepo, but with cleary separated backend and frontend	Monorepo, because that make more sense for small teams and tightly integrated components - otherwise, on the long run, half of commits would be duplicated - one commit per functionality for the frontend and one for the backend. I opted for using API-only rails on the backend and vue on the frontend, instead of just a rails app, mostly to fulfill the task "Clear API design and data flow between frontend and backend"

Everything that can be tuned (chunk size, retrieval k's, similarity threshold, model names) lives in config/qa.yml.

How to run

With docker

This path does not require Ruby on the host. If you have ollama running locally, turn it off first. Running this command will consume around 15 GB (mostly for gpt-oss:20b), and it will be quite slow (a single question can take over 5 minutes to answerwhile if yu run ollama and the app natively, it will be a matter of seconds). Before you run this command bump Docker Desktop's memory limit. Open Docker Desktop → Settings → Resources, and raise the Memory slider from ~7.7GB to at least 16GB (20GB+ gives more headroom), then hit "Apply & restart.".

docker compose up -d --build   # starts Postgres, Ollama, and the app container

Then open http://localhost:5173. The app starts immediately; a separate bootstrap service pulls the default Ollama models in the background on first run.

With ruby, ollama, and postgres installed on your machine

Remember to change PGUSER (and potentially other variables) in .env to match your PostgreSQL environment.

bin/setup

bin/dev starts the API on :3000, the SolidQueue worker, and Vite on :5173.

The generation model defaults to Ollama's gpt-oss:20b (ollama pull gpt-oss:20b). To use a hosted API instead, set in .env:

LLM_BASE_URL=https://api.openai.com/v1   # any OpenAI-compatible endpoint
LLM_API_KEY=sk-...
LLM_MODEL=gpt-4o-mini

Tests (no network, no LLM — all clients stubbed): bin/rails test

Smoke eval against the real models (ingests eval/fixtures/, asks the 8 questions in eval/questions.json, exits non-zero on any wrong page citation or hallucinated answer): bin/rails eval:run

API

All endpoints under /api, JSON in/out. Errors share one shape: { "error": { "code": "...", "message": "..." } }.

Interactive docs: the backend's home page (http://localhost:3000) redirects to Swagger UI at /api-docs, rendered from the hand-maintained OpenAPI document swagger/v1/openapi.yaml (rswag-api + rswag-ui; rswag-specs is skipped because it requires RSpec which is beyond the scope of this project).

Endpoint	Purpose
POST /api/documents	Multipart files[], one or more PDFs. Per-file result list; non-PDFs and files > 25 MB are rejected per-file. 202 if anything was accepted, 422 if everything was rejected.
GET /api/documents	All documents with status (queued/processing/ready/failed), page_count, error_message.
GET /api/documents/:id	Single document — the frontend's 2s polling target.
DELETE /api/documents/:id	Remove one document, its chunks, and its stored file — in any status. 204.
DELETE /api/documents	Remove every document ("start over"). 204.
POST /api/questions (alias POST /api/ask)	{ "question": "..." } over all ready documents. 409 if none are ready. 202 with a persisted question row: { id, question, status: "queued", found: null, answer: null, citations: [], citations_dropped: 0 }. Async — poll the endpoints below for the result.
GET /api/questions/:id	Single question — poll this while status is queued/processing. Once ready: { found, answer, citations: [{ document_id, filename, page, quote }], citations_dropped }. found: false is a normal ready state, not an error — "not found" is a correct outcome.
GET /api/questions	Full conversation history, oldest first — the frontend reloads this on page load.
DELETE /api/questions	Clear the conversation history. 204.

How the citations are guaranteed real

Two independent mechanisms, one structural and one adversarial:

1. Chunks never cross page boundaries (app/services/chunker.rb). Each chunk is built from exactly one page's text, so its page_number cannot be wrong - the page citation is guaranteed rather than probabilistically hoped.
2. Citation verification (app/services/citation_verifier.rb). The LLM must return a verbatim quote per citation. Each quote is checked against the cited chunk's actual text (case/whitespace-normalized, typographic quotes/dashes folded — deterministic, no fuzzy matching). Fabricated quotes are dropped; if every citation of a found: true answer fails verification, the whole answer degrades to found: false — an unverifiable answer is treated as not grounded. If only some citations are dropped, the response reports citations_dropped and the UI shows a "partially verified" badge - verification is citation-level in v1, so a claim whose citation was dropped may remain in the answer text (claim-level binding and NLI entailment are the documented next steps, SPEC_V2.md §5).

On top of that, an honesty guard runs before the LLM: if retrieval returns nothing, or the best dense similarity is below min_similarity, the pipeline returns found: false without paying for a generation call at all.

AI trace log: every ask appends one human-readable block to log/ai.log (app/services/ask_trace.rb) showing what was retrieved (with the best similarity vs the guard threshold), whether the guard refused, the model's raw answer, and — citation by citation - what was verified or exactly why it was dropped (out-of-range excerpt, empty quote, or a quote that isn't a verbatim substring of the cited excerpt). When an answer shows the "partially verified" badge, this file is where you see what the model offered and why it didn't survive.

Async processing and failure states

Document state lives in Postgres (queued → processing → ready | failed), moved by ProcessDocumentJob via DocumentIngestor. Every expected failure sets a specific error_message:

• password-protected PDF → "PDF is password-protected"
• unparseable file → "Could not parse PDF"
• scanned/image-only PDF → "No extractable text — scanned/image-only PDFs are not supported in v1"
• Ollama down → "Embedding service unavailable"

Ingestion is idempotent (re-running replaces the document's chunks in one transaction), so retrying after a transient failure is safe. A page refresh mid-processing is a non-event: the frontend re-fetches GET /api/documents on mount and resumes polling whatever is still non-terminal — the client holds no state worth losing.

Retrieval

Hybrid: pgvector cosine (top 20) + Postgres full-text websearch_to_tsquery (top 20), fused with Reciprocal Rank Fusion (k=60), top 6 into the prompt. Lexical search covers exact terms/numbers that embeddings blur; dense covers paraphrase. Both indexes (HNSW + GIN) live on the same chunks table.

Eval

bin/rails eval:run scores the pipeline on 4 committed fixture PDFs — 2 synthetic ones (regenerable via bin/generate_fixtures.rb) and 2 real-world documents — against 8 hand-verified questions, including an unanswerable one. Expected pages are (file, page) pairs; page-hit requires every cited page to be within the verified set. The report also surfaces partial verification (citations the model offered that failed the verbatim-quote check). Current result with bge-m3 + gpt-oss:20b, all local:

question          found  cited (file:pages)  expected                           page-hit  kw-hit  dropped  latency
------------------------------------------------------------------------------------------------------------------
q-notice-period   true   aurora-serv…:3      aurora-serv…:3                     yes       yes     -        21285ms
q-annual-fee      true   aurora-serv…:2      aurora-serv…:2                     yes       yes     -        11782ms
q-governing-law   true   aurora-serv…:4      aurora-serv…:4                     yes       yes     -        7759ms
q-vacation-days   true   atlas-emplo…:2      atlas-emplo…:2                     yes       yes     -        7462ms
q-probation       true   atlas-emplo…:1      atlas-emplo…:1                     yes       yes     -        7853ms
q-pets            false  []                  []                                 refused   -       -        8829ms
q-unit-gaps       true   Im_thinking…:5      Im_thinking…:5                     yes       yes     -        38687ms
q-llm-superhuman  true   I_think_tha…:1      I_think_tha…:1|2|3 Im_thinking…:2|4 yes      yes     3        46187ms

Partially verified answers (some citations failed verification and were removed): q-llm-superhuman

PASS: all 8 questions

(Latency is dominated by local 20B-model generation; a hosted endpoint is ~10x faster.)

The eval has caught two real behaviors so far:

• gpt-oss:20b wraps its verbatim quotes in literal quotation marks — before the verifier folded those, correct answers were honestly degraded to "not found".
• On the real documents, q-llm-superhuman reliably produces a partially verified answer: the model paraphrases some quotes, those citations are dropped, and the claims they supported remain visible but unsourced — the exact gap the citations_dropped badge surfaces and SPEC_V2 §5's claim-binding would close.

Key decisions & tradeoffs

• Page-bounded chunking over cleverer strategies — cross-page answers are handled by returning multiple citations, not by chunks that span pages. Correctness of the core promise beats recall at the margin.
• Quote-echo + string-match verification over NLI entailment — deterministic, ~40 lines, catches most citation fabrication. NLI is the documented next step.
• "Not found" is a normal ready state, not an error — honesty is a feature; the guard runs pre-LLM so unanswerable questions cost nothing.
• Postgres for everything (queue, vectors, full-text) — one dependency, one backup story, trivially reviewable.
• Asking is async and persisted, but not conversational — each question is a Question row processed by its own job (so a page reload never loses an in-flight or past answer, and the conversation can be cleared), but every question is still answered independently: no history is fed into retrieval or the prompt, so follow-ups aren't decontextualized. Grouping into distinct conversations and history-aware answering (query rewriting) are the remaining V2 §7 work.

Cut corners (known, deliberate)

• Physical page indexes only — printed page labels (roman numerals etc.) are V2.
• Scanned/image-only PDFs are rejected, not OCR'd; tables/multi-column layouts are flattened by pdf-reader (Docling is the documented V2 parser).
• Questions always query all ready documents; per-document scoping is a trivial extension (filter in ChunkRetriever#ready_chunks).
• No auth, no rate limiting, local-disk storage.
• min_similarity (0.35) was tuned against the original 6-question eval only (now 8, still smoke-test scale) — a real corpus needs the V2 eval harness before touching it.

What I'd do next

In order (details in SPEC_V2.md):

1. Production release - migrate towards an architecture using stronger and faster models
2. Reranker (bge-reranker-v2-m3, in-process) - cheapest measured quality win.
3. Full eval harness with run tracking - per-question results stored per config hash, so every later change is measured.
4. Docling parser - tables, multi-column, OCR, and bounding boxes for highlight-in-PDF citations.
5. Grounding hardening - NLI entailment per claim; citation UX with a pdf.js viewer that deep-links to the cited page.
6. Product features - conversation-aware answering (history in the prompt, query rewriting — turns already persist, see Key decisions), streaming, per-document scoping, auth.