1.2 harness

1.2. The Harness

The harness is 3F's conversation engine — the part that turns "user typed something with two images" into a streamed, validated, persisted assistant answer. Module root: server/src/modules/harness/.

The step engine

Jobs from the harness queue execute through a deterministic pipeline of steps (services/steps/), each backed by an action (actions/):

sanitize ──▶ interpret ──▶ execute ──▶ respond

Step / Action	Role
sanitize	Normalise and harden the incoming prompt/images/metadata before any model sees them.
interpret	Classify intent (prompts/intent-selection.prompt.ts, templates/intent.schema.ts) — e.g. chat vs. describe/compare/OCR/news/article/product/media-list — and choose a response variant (variant-instructions.registry.ts).
execute	Perform the work: model call via the AI SDK, tool invocations when the intent requires grounding (Serper + YouTube Data API sources, webpage scrape/fetch), structured generation against the chosen JSON schema.
respond	Shape the final answer, validate it (response-validator.service.ts) against the variant schema, and stream/persist it.

Steps run under a shared HarnessContext (harness-context.service.ts) carrying identity (requestId, sessionId, conversationId, roomId), model parameters (numCtx, think, stream), and the accumulated state. step-engine.service.ts drives the sequence; step-registry.service.ts maps step names to implementations so the pipeline stays declarative and each step stays testable in isolation (all steps/actions have spec files).

The HarnessStepLogger reports semantic progress (receive, per-step events) with truncated prompt previews — long logs without leaking full user content.

Grounding tools

Tools live in server/src/modules/ai-sdk/tools/sources/ — one factory per tool, created only when its provider is enabled. Tool factories are split per provider (serper/, bright-data/, youtube/):

• Serper (serper/): serperWebSearch, serperImageSearch (720p min, 1440p preferred), serperNewsSearch, serperPlacesSearch, serperShoppingSearch, serperBusinessReviewsSearch (Google Maps reviews of a business, keyed by cid/placeId), serperVideoSearch, plus serperWebpageScrape (rendered page text via the scrape endpoint).
• Bright Data (bright-data/): brightDataWebSearch, brightDataImageSearch, brightDataNewsSearch, brightDataPlacesSearch, brightDataShoppingSearch, brightDataVideoSearch — an alternative search provider enabled via SysCtl.
• YouTube Data API v3 (youtube.ts): youtubeVideoSearch — search.list + batch videos.list enrichment (duration, view count, channel, direct thumbnail), enabled by YOUTUBE_API_KEY/SysCtl.
• Browser automation (Playwright MCP sidecar) — browser_navigate, browser_snapshot, browser_click, browser_type, browser_fill_form, browser_tabs, browser_network_requests, browser_console_messages, browser_take_screenshot, browser_wait_for, browser_verify_*, browser_select_option, browser_press_key, browser_navigate_back. A headless chromium driven through the playwright-mcp compose sidecar (PLAYWRIGHT_MCP_ENABLED); page content arrives as accessibility snapshots, so no vision model is required. Dangerous tools (browser_run_code*, browser_evaluate, browser_file_upload) are deny-listed. Browser intents get an 8-step budget (BROWSER_MAX_STEPS); search tools stay single-step.
• Built-ins: webSearch (SearXNG), webFetch, image-variant request tools.

Media ordering rules in the sanitize step: web-article videos rank first, then YouTube results outrank Serper videos, and images sit last in the tool context. Before the respond step, a lightweight CLD3 language detector (detect-language.helper.ts) tags articles/videos with their language; items in a language different from the user's are moved into internationalArticles/internationalVideos pools so nothing is lost — the templates render them as an international coverage aside next to the primary-language content.

Prompts & structured outputs

• System prompts: prompts/base-system.prompt.ts, prompts/content-system.prompt.ts, plus reusable instructions/ and shared/ fragments. Variant instructions are selected per intent from variant-instructions.registry.ts.
• Structured output schemas (schemas/): ocr, describe, compare, summary, article, evaluation, imagelist, news, product, shoplist, video-gallery-item, videolist.

These schemas are the contract between the model and the UI: the dashboard renders exchanges from structured JSON (lists, galleries, articles, comparisons) rather than trying to parse prose. normalize-think helpers in ai-sdk/helpers/ keep model quirks (thinking tags etc.) out of the final payload. Response validation is split per variant (services/response-validators/): article, news, describe, compare, ocr, summary, evaluation, product, shoplist, imagelist, videolist, and free-form validators each enforce their own schema before the answer reaches the UI.

Streaming & persistence

harness-chat-streaming.service.ts owns emission to the Socket.IO room: incremental stream chunks while the model produces, role/phase payloads (assistant, original, clarification), terminal result, and error. Successful runs persist the exchange into HarnessConversation (sessionId + conversationId scoped, JSON content) — see 1.5.

Cancellation

harness-cancellation.service.ts provides cooperative cancellation: POST /harness/cancel sets a cancellation marker for the requestId; the step engine checks it at step boundaries and the streaming service sends cancel_result back. Cancellation is async and best-effort by design — it stops work between steps/chunks, not mid-syscall.

Media ingestion

• cloud-image-ingestion.service.ts — fetches image payloads referenced by URL/metadata instead of direct upload.
• media-url-validator.service.ts — validates media URLs (schemes, SSRF-guarded targets) before fetching.
• Media sanitization helpers (helpers/) verify media URLs against live endpoints (1280×720 floor), collect image/video/page URLs, scrub broken URLs from messages, rewrite candidates with ingested media, build ingested-by-URL maps and user fingerprints, and limit cloud-reference images.
• shown-media.service.ts — records and deduplicates already-shown media (HarnessShownMedia) so media-list follow-ups never repeat content across history.

Intake flow recap

1. Controller validates DTOs and the x-harness-llm header (400 fast).
2. Image parts → buffers + metadata, deduped against session hashes.
3. HarnessQueueService.emit enqueues the job envelope (buffers, meta, filters) → 202 with realtime coordinates.
4. HarnessProcessor (harness.processor.ts) picks the job off the queue and drives the step engine.
5. Failure at any point → envelope copied to the DLQ (see 1.3), error event to the room.

Why a queue at all?

Because models are slow and flaky and users are not. Queueing decouples HTTP latency from inference latency, gives us retries/backoff for transient Ollama hiccups, enables cancellation and re-instatement, and lets horizontal workers scale against KeyDB without touching the API process.