Penta Application Screener

A production tool that turns 300+ housing co-op applications into a committee-ready, weighted shortlist — with a human in the loop at every stage and every AI-influenced number traceable back to its evidence. In real use by the Penta Housing Co-op membership committee, hosted at screener.pentacoop.com.

It imports Google Sheets responses, applies deterministic eligibility filters, runs cached AI passes over the eligible pool, and produces a ranked list with per-candidate rationale. Reviewers get a searchable table, candidate detail pages, audit-friendly flags, human overrides, and an interactive tier-list for weighting what matters.

It's both a live application backing a real co-op screening workflow and a portfolio project exploring the craft of AI product design: human-in-the-loop review, cost-aware model use, and the judgment of which decisions to keep deterministic and which to hand to an LLM.

Design Highlights

A few decisions I'm particularly happy with — the ideas that make this more than a wrapper around an LLM call:

• AI suggestions are inert until a human activates them. The model may propose any differentiating dimension, but a discovered one carries weight 0 until a committee member drags it into a tier — nothing the AI says can move a ranking on its own. Safety becomes a property of the workflow rather than of prompt wording, so every junk suggestion is harmless by default, including the ones I never anticipated.

• The LLM extracts features; the math does the ranking. No model is ever asked "who's the best candidate?" — it scores each applicant per dimension (with rationale and evidence), and fit is a pure, inspectable formula, Σ(weight·score) / Σ(weight), so every ranked number traces back to a specific score and a committee-assigned weight.

• Content-addressed caching, so nothing is ever paid for twice. Every result is cached on a key of (application content, pass, model, prompt version), so an unchanged applicant reuses its stored score for free — a cache hit is exempt from the spending cap because it makes no model call. On a ~300-applicant pool this is the difference between re-scoring everyone on every tweak and paying only for what genuinely changed. Two things make the key precise: a pass hashes its own static prompt text into that prompt_version, so editing one prompt re-runs only that pass; and a re-rank runs an LLM identity-match that re-adopts a re-discovered dimension's prior key, so its cached scores carry forward and only genuinely new or edited dimensions are charged. Re-tiering re-sorts with zero spend.

• Cost estimated up front, capped, and attributed. Every run projects its cost before starting (over the uncached work only), is checked against a server-side spending cap, and refuses no-op re-runs — AI spend is a first-class product surface, not a surprise on a bill.

• Every pass persists its reasoning, and the reasoning is a product surface, not a log file. Each AI call's free-text rationale and structured audit trail (what discovery found, how decomposition settled it, why consolidation merged a pair) are stored per run and rendered in the Observability tab. That trace pays for itself three ways: it turns a red ✗ into a root cause when a pass misbehaves (one pet-flag miss became a three-line prompt fix because the captured reasoning explained why the model held back); it's the audit trail that lets a committee trust an AI-influenced ranking; and it's mined to grow the eval sets — real runs become labelled test cases instead of hand-fabricated ones.

What It Does

The workflow is three single-verb steps — Import → Screen → Rank — each gated behind a confirmation card with an up-front cost estimate, plus a "View ranking" action.

• Google OAuth login with signed server-side sessions, least-privilege by design: members sign in with identity only (no Drive/Sheets scope). An admin links the one response sheet via the Google Picker, granting drive.file access to that single file — so the app never asks a member for broad Drive access.
• Google Sheets sync into a SQLite database, read with the linking admin's designated token.
• Configurable application settings for unit size, move-in date, income range, household rules, pets, and disabled deterministic rules.
• Deterministic hard filters for clear eligibility issues, applied at import.
• Application dashboard, searchable/sortable table, facets, pagination, and candidate detail pages.
• Screen: AI integrity pass flagging suspicious, AI-boilerplate, or low-quality submissions (informational input to human review, never auto-disqualifying).
• Rank: one orchestrated AI chain over eligible applicants — parallel pattern discovery → decomposition into one non-overlapping set → identity-match onto prior runs → per-dimension scoring → post-score duplicate consolidation — feeding a weighted ranked list with relative fit bands and per-driver rationale. (Detailed in The AI Pipeline below; the ranking math is in The LLM extracts features; the math does the ranking above.)
• Interactive tier-list weighting: drag discovered criteria into Critical/Important/Minor/Ignore tiers to instantly re-sort. Re-ranking carries tier placements forward and reuses cached scores (see Prompt identity as a cache key above).
• Reports: browser print-to-PDF of the ranked view and candidate detail pages, with an @media print stylesheet and a text importance-tiers summary.
• Provider-agnostic AI interface with Amazon Bedrock/Strands as the concrete provider and a deterministic mock provider for tests.
• Raw source row and raw AI output debug panels, on the candidate detail page (open to any logged-in member — every committee member is a trusted screener).
• Human status overrides with stale-finding indicators when machine findings change later.

The AI Pipeline

Every AI call is a named, single-purpose pass — never a general "agent" deciding what to do next. The orchestration is deterministic code and human-gated workflow steps; no model chooses which pass runs, and no pass calls another. Each is a structured-output call with its own prompt, schema, cache/version, cost line, and reasoning trace. Model tier is chosen per job: cheap-and-fast Haiku where call count drives cost, stronger Sonnet for the once-per-run judgment calls.

Screen (one pass, runs on its own):

• Screening integrity flags (Haiku) — reads each application and flags placeholder/suspicious names, spam or AI-boilerplate essays, internal inconsistencies, and contact/pet-policy issues. Informational only; never auto-disqualifies.

Rank (one button, five passes chained deterministically over the eligible pool):

1. Pattern discovery (Sonnet, ×K in parallel) — reads the whole pool and discovers the dimensions it actually varies on. Runs K times on fresh contexts; their cross-call disagreement is the diversity the next step needs. Each call is blind except for committee proposals seeded into one worker.
2. Decomposition (Sonnet) — settles the K overlapping discovery reports into one finest, non-overlapping set: collapses re-carvings of one concept, keeps genuinely distinct axes apart, protects committee-requested axes.
3. Identity matching (Sonnet) — maps this run's dimensions onto prior runs' by meaning, so a re-discovered concept re-adopts its old key and carries its tier placement + cached scores forward. A high bar (a wrong match corrupts a reused score), so it errs toward "new."
4. Dimension scoring (Haiku, per candidate) — scores each applicant on each dimension from −1 (low end) to +1 (high end), 0 neutral, with a rationale and grounding evidence. Silence scores 0, never negative — absence of evidence isn't a weakness. The only per-applicant pass; everything above is pool-level.
5. Consolidation (Sonnet) — post-score cleanup: since every dimension now has a per-applicant score vector, near-identical vectors nominate suspected duplicates the definition-only match pass missed, and one confirm call merges genuine ones (aliasing the newer key to the older, so the key space converges instead of growing). Distinct axes that merely correlate are kept apart.

Then the ranking itself is pure deterministic math over the cached scores and committee tier weights — no model call. Two invariants hold across all of it: AI output is inert until a human activates it (a discovered dimension has weight 0 until tiered), and every pass persists its reasoning + cost so any number traces back to its evidence.

Two tabs make the AI legible. Observability surfaces each run: what each discoverer found, how decomposition settled them, which duplicates consolidation merged and why, how dimensions carried forward, per-pass cost attribution, and operational-metrics trends. Evals is an in-app cockpit — property-based invariants, per-pass live evals, and a blind label-auditing LLM judge (evaluation design is documented in docs/ai-evals.md).

The spec lives in SPEC.md; developer architecture notes in docs/app-architecture.md, with deeper references in docs/ai-screening.md, docs/api.md, and docs/form-field-reference.md. Significant design decisions live in docs/adr/. Shared agent guidance lives in .clinerules, with AGENTS.md pointing agents there.

Privacy And Test Data

Applicant data is sensitive. Do not commit real application exports, local SQLite databases, OAuth credentials, raw AI traces, exported/printed reports with applicant data, or .env files.

The sample CSV in test-data is synthetic and intentionally realistic so import logic and AI quality checks can be exercised locally. See test-data/README.md for the directory policy.

Tech Stack

• Backend: Python, FastAPI, SQLAlchemy, Alembic, SQLite
• Python tooling: uv, project-local virtual environment, pytest
• Frontend: Vite, React, TypeScript, npm
• Authentication: Google OAuth with signed server-side session cookies
• Google integration: Google Sheets import/sync via the Picker (drive.file, least-privilege)
• AI integration: provider-agnostic interface; Strands + Amazon Bedrock provider; mock provider for tests
• Hosting: Fly.io (single instance, auto-suspend, persistent-volume SQLite); single-origin — FastAPI serves the built SPA; deployed manually with fly deploy --remote-only

Setup

1. Install prerequisites:

   • uv
   • Node.js 20+ with npm
   • PowerShell 7 on Windows if using dev.ps1

2. Initialize dependencies and the local database:

   bash ./setup.sh       # macOS/Linux

   ./setup.ps1           # Windows PowerShell

3. Configure Google OAuth.

   Place the downloaded OAuth client JSON from Google Cloud Console in backend/secrets/:

   mkdir -p backend/secrets
   # copy or move the downloaded client_secret_*.json file into backend/secrets/

   The backend auto-discovers any client_secret_*.json file in that directory. The directory is ignored by Git.

   See docs/google-cloud-oauth-setup.md for full Google Cloud and OAuth details.

   The setup script has already run the database migrations.

Local Development

Start both servers:

./dev.sh        # macOS/Linux

./dev.ps1       # Windows PowerShell

The backend runs at http://localhost:8000. The frontend runs at http://localhost:5173. On Windows, dev.ps1 writes per-service output and errors to .dev-logs/. If either service exits, it prints the last log lines; it also retries the frontend twice before leaving the backend running for diagnosis.

If local screening data looks stale or inconsistent, reset the local SQLite database before starting dev:

./reset-db.sh
./dev.sh

./reset-db.ps1
./dev.ps1

Backups

A Rank's output is expensive (paid Bedrock calls) and non-deterministic — it cannot be regenerated identically — so the local database is snapshotted. Snapshots use SQLite VACUUM INTO (a consistent hot copy, safe while the backend is running) and land in backend/data/backups/, which is gitignored (the snapshots hold applicant PII and must never be committed).

• Automatic: every completed Rank snapshots the DB (best-effort — a backup failure never fails the run).
• Manual: ./backup-db.sh [tag] (./backup-db.ps1 -Tag <label> on Windows) — e.g. before anything risky. The newest ~50 snapshots are kept.
• Restore: ./restore-db.sh lists snapshots and prompts for one (--latest for the most recent); ./restore-db.ps1 on Windows. The current DB is snapshotted first (tag pre-restore), so a restore is itself reversible. Stop the backend before restoring.

Or run services individually:

Backend:

cd backend
uv run fastapi dev app/main.py

Frontend:

cd frontend
npm run dev

Tests

Backend:

cd backend
uv run pytest

Frontend build/type check:

cd frontend
npm run build

Manual AI Quality Audit

Run the evals — the blind label-auditing judge, the per-pass live evals (single run + stability, ?mode=stability), and the deterministic invariants — from the in-app Evals tab. The spending evals make paid Bedrock calls only when you confirm a run; nothing here runs during Rank or the test suite. Each subtab shows its cases, runs whole-set or per-case, streams the model's reasoning, and persists every run; re-baselining the invariant fixture is the tab's "Re-baseline from current Rank" action. Growing the golden case sets from a real Rank is done with the harvest scripts under backend/scripts/ (co-authored, then labelled by hand).

Status

Live in production. The app is deployed on Fly.io at screener.pentacoop.com and used by the Penta Housing Co-op membership committee (a small, trusted group of screeners) to review real application cycles. It runs as a single instance with SQLite on a persistent volume — deliberately right-sized for a ~5-member committee rather than built for scale it doesn't need. Hosting decisions and the platform tradeoff analysis are in docs/adr/0012-hosting-platform-m17.md; the deploy/runbook is docs/deploy.md, and suspend-to-zero recovery controls are in ops/fly-watchdog/README.md.

It remains an actively developed project, and everything below still works for fully local screening workflows — the same codebase runs on a laptop with no hosting at all.

License

This project is licensed under the Apache License 2.0. See LICENSE.