1.2 harness

1.2. The Harness

The harness is 3F's conversation engine — the part that turns "user typed something with two images" into a streamed, validated, persisted assistant answer. Module root: server/src/modules/harness/.

The step engine

Jobs from the harness queue execute through a deterministic pipeline of steps (services/steps/), each backed by an action (actions/):

                                           ┌─▶ memoryWrite ──▶ memoryProfile
interpret ──▶ execute ──▶ sanitize ──▶ respond ┤
                                           └─▶ vectorize

Step / Action	Role
interpret	Classify intent (prompts/intent-selection.prompt.ts, templates/intent.schema.ts) — one of 14 templates: article, news, describe, compare, ocr, summary, evaluation, product, shoplist, imagelist, videolist, stockmarketitem, stockmarketlist, text — and choose a response variant (variant-instructions.registry.ts). The classifier also picks tools, image/video counts, recency anchoring, a context summary, and can ask a clarification question when the request is too ambiguous.
execute	Perform the work: model call via the AI SDK, tool invocations when the intent requires grounding (Serper + YouTube Data API sources, webpage scrape/fetch), structured generation against the chosen JSON schema.
sanitize	Verify URLs, partition non-user-language results into international pools, scrub broken media — and assemble the final messages. This is where the AI's cognition is injected as always-on system context when memory is enabled: the structured profile always (it doubles as the probe routing map — profile values that token-match the prompt sharpen the insight query: "cars" → likes.cars facet), plus the top-3 insights when the space holds any (identity-key memoryCognition → memoryPartition → sessionId).
respond	Shape the final answer, validate it (response-validator.service.ts) against the variant schema, and stream/persist it.
memoryWrite	Post-response fact write — enqueue-only: when the classifier picked memoryRemember, the step summarizes the turn's gathered results and enqueues a memory-write job; the LLM tool loop (prior-memory search → remember decisions) runs in the vectorize worker (see 1.5 memory).
memoryProfile	Post-response cognition — enqueue-only: after every answered turn (never classifier-gated; "subconscious formation") the step enqueues a memory-profile job; the worker rewrites the structured cognition profile / upserts insight records via one dedicated model call (tolerantly parsed and validated; profile: null = nothing durable learned).
vectorize	Fire-and-forget enqueue of both turn-sides into the vectorize queue (fact extraction + embedding off the request path). The same queue also runs the memory-write and memory-profile cognition jobs enqueued by the two steps above — one worker, three job kinds (processors/vectorize.processor.ts branches on the job name).

Steps run under a shared HarnessContext (harness-context.service.ts) carrying identity (requestId, sessionId, conversationId, roomId), model parameters (numCtx, think, stream), and the accumulated state. step-engine.service.ts drives the sequence; step-registry.service.ts maps step names to implementations so the pipeline stays declarative and each step stays testable in isolation (all steps/actions have spec files).

The HarnessStepLogger reports semantic progress (receive, per-step events) with truncated prompt previews — long logs without leaking full user content.

Grounding tools

Tools live in server/src/modules/ai-sdk/tools/sources/ — one factory per tool, created only when its provider is enabled. Tool factories are split per provider (serper/, bright-data/, youtube/):

• Serper (serper/): serperWebSearch, serperImageSearch (720p min, 1440p preferred), serperNewsSearch, serperPlacesSearch, serperShoppingSearch, serperBusinessReviewsSearch (Google Maps reviews of a business, keyed by cid/placeId), serperVideoSearch, plus serperWebpageScrape (rendered page text via the scrape endpoint).
• Bright Data (bright-data/): brightDataWebSearch, brightDataImageSearch, brightDataNewsSearch, brightDataPlacesSearch, brightDataShoppingSearch, brightDataVideoSearch — an alternative search provider enabled via SysCtl.
• YouTube Data API v3 (youtube.ts): youtubeVideoSearch — search.list + batch videos.list enrichment (duration, view count, channel, direct thumbnail), enabled by YOUTUBE_API_KEY/SysCtl.
• EODHD (eodhd/): eodhdQuote, eodhdSearch, eodhdHistory, eodhdIntraday, eodhdNews, eodhdFundamentals, eodhdTechnical — market-data tools for stock-market intents, enabled by EODHD_API_KEY/SysCtl. The execute step wraps them with chart-streaming events and builds a fallback input when the provider is unavailable.
• Browser automation (Playwright MCP sidecar) — browser_navigate, browser_snapshot, browser_click, browser_type, browser_fill_form, browser_tabs, browser_network_requests, browser_console_messages, browser_take_screenshot, browser_wait_for, browser_verify_*, browser_select_option, browser_press_key, browser_navigate_back. A headless chromium driven through the playwright-mcp compose sidecar (PLAYWRIGHT_MCP_ENABLED); page content arrives as accessibility snapshots, so no vision model is required. Dangerous tools (browser_run_code*, browser_evaluate, browser_file_upload) are deny-listed. Browser intents get an 8-step budget (BROWSER_MAX_STEPS); search tools stay single-step.
• Built-ins: webSearch (SearXNG), webFetch, image-variant request tools.
• Memory (memory/, gated by MEMORY_ENABLED): memoryRemember (store one self-contained fact record), memoryRecall (multi-variant semantic recall of the user's fact partition; its verbatim quotes are the delete identity — records expose no ids), memoryDelete (delete an exact verbatim record, or cognition:true to wipe the AI's understanding of the user when asked to forget them). Delete intents get a 3-step budget (MEMORY_DELETE_MAX_STEPS) so recall → delete can chain. Write/delete details live in 1.5.

Media ordering rules in the sanitize step: web-article videos rank first, then YouTube results outrank Serper videos, and images sit last in the tool context. Before the respond step, a lightweight CLD3 language detector (detect-language.helper.ts) tags articles/videos with their language; items in a language different from the user's are moved into internationalArticles/internationalVideos pools so nothing is lost — the templates render them as an international coverage aside next to the primary-language content.

Prompts & structured outputs

• System prompts: prompts/base-system.prompt.ts, prompts/content-system.prompt.ts, plus reusable instructions/ and shared/ fragments. Variant instructions are selected per intent from variant-instructions.registry.ts.
• Snippet-composed templates (snippets/): the news, article, and evaluation templates are composed from reusable snippets (header, lead, key findings, sources, gallery, video gallery, related stories, …) via snippet-presets.constant.ts. Each preset composes its own Zod schema (compose-snippet-schema.helper) and validation (create-snippet-validator.helper), so the model contract stays declarative per template. Every other template keeps its rigid instructions/schema.
• Structured output schemas (schemas/): ocr, describe, compare, summary, article, evaluation, imagelist, news, product, shoplist, stockmarketitem, stockmarketlist, video-gallery-item, videolist.

These schemas are the contract between the model and the UI: the dashboard renders exchanges from structured JSON (lists, galleries, articles, comparisons) rather than trying to parse prose. normalize-think helpers in ai-sdk/helpers/ keep model quirks (thinking tags etc.) out of the final payload. Response validation is split per variant (services/response-validators/): article, news, describe, compare, ocr, summary, evaluation, product, shoplist, imagelist, videolist, and free-form validators each enforce their own schema before the answer reaches the UI.

Streaming & persistence

harness-chat-streaming.service.ts owns emission to the Socket.IO room: incremental stream chunks while the model produces, role/phase payloads (assistant, original, clarification), terminal result, and error. Successful runs persist the exchange into HarnessConversation (sessionId + conversationId scoped, JSON content) — see 1.5.

Cancellation

harness-cancellation.service.ts provides cooperative cancellation: POST /harness/cancel sets a cancellation marker for the requestId; the step engine checks it at step boundaries and the streaming service sends cancel_result back. Cancellation is async and best-effort by design — it stops work between steps/chunks, not mid-syscall.

Media ingestion

• cloud-image-ingestion.service.ts — fetches image payloads referenced by URL/metadata instead of direct upload.
• media-url-validator.service.ts — validates media URLs (schemes, SSRF-guarded targets) before fetching.
• Media sanitization helpers (helpers/) verify media URLs against live endpoints (1280×720 floor), collect image/video/page URLs, scrub broken URLs from messages, rewrite candidates with ingested media, build ingested-by-URL maps and user fingerprints, and limit cloud-reference images.
• shown-media.service.ts — records and deduplicates already-shown media (HarnessShownMedia) so media-list follow-ups never repeat content across history.

Intake flow recap

1. Controller validates DTOs and the x-harness-llm header (400 fast).
2. Image parts → buffers + metadata, deduped against session hashes.
3. HarnessQueueService.emit enqueues the job envelope (buffers, meta, filters) → 202 with realtime coordinates.
4. HarnessProcessor (harness.processor.ts) picks the job off the queue and drives the step engine.
5. Failure at any point → envelope copied to the DLQ (see 1.3), error event to the room.

Why a queue at all?

Because models are slow and flaky and users are not. Queueing decouples HTTP latency from inference latency, gives us retries/backoff for transient Ollama hiccups, enables cancellation and re-instatement, and lets horizontal workers scale against KeyDB without touching the API process.