# 1.2. The Harness

The harness is 3F's conversation engine — the part that turns "user typed something with two images" into a streamed, validated, persisted assistant answer. Module root: `server/src/modules/harness/`.

## The step engine

Jobs from the harness queue execute through a deterministic pipeline of steps (`services/steps/`), each backed by an action (`actions/`):

```
                                           ┌─▶ memoryWrite ──▶ memoryProfile
interpret ──▶ execute ──▶ sanitize ──▶ respond ┤
                                           └─▶ vectorize
```

| Step / Action | Role                                                                                                                                                                                                                              |
| ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `interpret`   | Classify _intent_ (`prompts/intent-selection.prompt.ts`, `templates/intent.schema.ts`) — one of 14 templates: `article`, `news`, `describe`, `compare`, `ocr`, `summary`, `evaluation`, `product`, `shoplist`, `imagelist`, `videolist`, `stockmarketitem`, `stockmarketlist`, `text` — and choose a response variant (`variant-instructions.registry.ts`). The classifier also picks tools, image/video counts, recency anchoring, a context summary, and can ask a clarification question when the request is too ambiguous. |
| `execute`     | Perform the work: model call via the AI SDK, tool invocations when the intent requires grounding (Serper + YouTube Data API sources, webpage scrape/fetch), structured generation against the chosen JSON schema.               |
| `sanitize`    | Verify URLs, partition non-user-language results into international pools, scrub broken media — and assemble the final messages. This is where the AI's cognition is injected as always-on system context when memory is enabled: the structured profile always (it doubles as the probe routing map — profile values that token-match the prompt sharpen the insight query: "cars" → `likes.cars` facet), plus the top-3 insights when the space holds any (identity-key `memoryCognition` → `memoryPartition` → `sessionId`). |
| `respond`     | Shape the final answer, validate it (`response-validator.service.ts`) against the variant schema, and stream/persist it.                                                                                                          |
| `memoryWrite` | Post-response fact write — enqueue-only: when the classifier picked `memoryRemember`, the step summarizes the turn's gathered results and enqueues a `memory-write` job; the LLM tool loop (prior-memory search → remember decisions) runs in the vectorize worker (see **1.5** memory). |
| `memoryProfile` | Post-response cognition — enqueue-only: after every answered turn (never classifier-gated; "subconscious formation") the step enqueues a `memory-profile` job; the worker rewrites the structured cognition profile / upserts insight records via one dedicated model call (tolerantly parsed and validated; `profile: null` = nothing durable learned). |
| `vectorize`   | Fire-and-forget enqueue of both turn-sides into the vectorize queue (fact extraction + embedding off the request path). The same queue also runs the `memory-write` and `memory-profile` cognition jobs enqueued by the two steps above — one worker, three job kinds (`processors/vectorize.processor.ts` branches on the job name). |

Steps run under a shared `HarnessContext` (`harness-context.service.ts`) carrying identity (`requestId`, `sessionId`, `conversationId`, `roomId`), model parameters (`numCtx`, `think`, `stream`), and the accumulated state. `step-engine.service.ts` drives the sequence; `step-registry.service.ts` maps step names to implementations so the pipeline stays declarative and each step stays testable in isolation (all steps/actions have spec files).

The `HarnessStepLogger` reports semantic progress (`receive`, per-step events) with truncated prompt previews — long logs without leaking full user content.

## Grounding tools

Tools live in `server/src/modules/ai-sdk/tools/sources/` — one factory per tool, created only when its provider is enabled. Tool factories are split per provider (`serper/`, `bright-data/`, `youtube/`):

- **Serper** (`serper/`): `serperWebSearch`, `serperImageSearch` (720p min, 1440p preferred), `serperNewsSearch`, `serperPlacesSearch`, `serperShoppingSearch`, `serperBusinessReviewsSearch` (Google Maps reviews of a business, keyed by `cid`/`placeId`), `serperVideoSearch`, plus `serperWebpageScrape` (rendered page text via the scrape endpoint).
- **Bright Data** (`bright-data/`): `brightDataWebSearch`, `brightDataImageSearch`, `brightDataNewsSearch`, `brightDataPlacesSearch`, `brightDataShoppingSearch`, `brightDataVideoSearch` — an alternative search provider enabled via SysCtl.
- **YouTube Data API v3** (`youtube.ts`): `youtubeVideoSearch` — `search.list` + batch `videos.list` enrichment (duration, view count, channel, direct thumbnail), enabled by `YOUTUBE_API_KEY`/SysCtl.
- **EODHD** (`eodhd/`): `eodhdQuote`, `eodhdSearch`, `eodhdHistory`, `eodhdIntraday`, `eodhdNews`, `eodhdFundamentals`, `eodhdTechnical` — market-data tools for stock-market intents, enabled by `EODHD_API_KEY`/SysCtl. The execute step wraps them with chart-streaming events and builds a fallback input when the provider is unavailable.
- **Browser automation (Playwright MCP sidecar)** — `browser_navigate`, `browser_snapshot`, `browser_click`, `browser_type`, `browser_fill_form`, `browser_tabs`, `browser_network_requests`, `browser_console_messages`, `browser_take_screenshot`, `browser_wait_for`, `browser_verify_*`, `browser_select_option`, `browser_press_key`, `browser_navigate_back`. A headless chromium driven through the `playwright-mcp` compose sidecar (`PLAYWRIGHT_MCP_ENABLED`); page content arrives as accessibility snapshots, so no vision model is required. Dangerous tools (`browser_run_code*`, `browser_evaluate`, `browser_file_upload`) are deny-listed. Browser intents get an 8-step budget (`BROWSER_MAX_STEPS`); search tools stay single-step.
- **Built-ins**: `webSearch` (SearXNG), `webFetch`, image-variant request tools.
- **Memory** (`memory/`, gated by `MEMORY_ENABLED`): `memoryRemember` (store one self-contained fact record), `memoryRecall` (multi-variant semantic recall of the user's fact partition; its verbatim quotes are the delete identity — records expose no ids), `memoryDelete` (delete an exact verbatim record, or `cognition:true` to wipe the AI's understanding of the user when asked to forget them). Delete intents get a 3-step budget (`MEMORY_DELETE_MAX_STEPS`) so recall → delete can chain. Write/delete details live in **1.5**.

Media ordering rules in the sanitize step: web-article videos rank first, then **YouTube results outrank Serper videos**, and images sit last in the tool context. Before the respond step, a lightweight CLD3 language detector (`detect-language.helper.ts`) tags articles/videos with their language; items in a language different from the user's are moved into `internationalArticles`/`internationalVideos` pools so nothing is lost — the templates render them as an **international coverage** aside next to the primary-language content.

## Prompts & structured outputs

- System prompts: `prompts/base-system.prompt.ts`, `prompts/content-system.prompt.ts`, plus reusable `instructions/` and `shared/` fragments. Variant instructions are selected per intent from `variant-instructions.registry.ts`.
- **Snippet-composed templates** (`snippets/`): the `news`, `article`, and `evaluation` templates are composed from reusable snippets (header, lead, key findings, sources, gallery, video gallery, related stories, …) via `snippet-presets.constant.ts`. Each preset composes its own Zod schema (`compose-snippet-schema.helper`) and validation (`create-snippet-validator.helper`), so the model contract stays declarative per template. Every other template keeps its rigid instructions/schema.
- Structured output schemas (`schemas/`): `ocr`, `describe`, `compare`, `summary`, `article`, `evaluation`, `imagelist`, `news`, `product`, `shoplist`, `stockmarketitem`, `stockmarketlist`, `video-gallery-item`, `videolist`.

These schemas are the contract between the model and the UI: the dashboard renders exchanges from structured JSON (lists, galleries, articles, comparisons) rather than trying to parse prose. `normalize-think` helpers in `ai-sdk/helpers/` keep model quirks (thinking tags etc.) out of the final payload. Response validation is split per variant (`services/response-validators/`): `article`, `news`, `describe`, `compare`, `ocr`, `summary`, `evaluation`, `product`, `shoplist`, `imagelist`, `videolist`, and `free-form` validators each enforce their own schema before the answer reaches the UI.

## Streaming & persistence

`harness-chat-streaming.service.ts` owns emission to the Socket.IO room: incremental `stream` chunks while the model produces, role/phase payloads (`assistant`, `original`, `clarification`), terminal `result`, and `error`. Successful runs persist the exchange into `HarnessConversation` (sessionId + conversationId scoped, JSON `content`) — see **1.5**.

## Cancellation

`harness-cancellation.service.ts` provides cooperative cancellation: `POST /harness/cancel` sets a cancellation marker for the `requestId`; the step engine checks it at step boundaries and the streaming service sends `cancel_result` back. Cancellation is async and best-effort by design — it stops work _between_ steps/chunks, not mid-syscall.

## Media ingestion

- `cloud-image-ingestion.service.ts` — fetches image payloads referenced by URL/metadata instead of direct upload.
- `media-url-validator.service.ts` — validates media URLs (schemes, SSRF-guarded targets) before fetching.
- Media sanitization helpers (`helpers/`) verify media URLs against live endpoints (1280×720 floor), collect image/video/page URLs, scrub broken URLs from messages, rewrite candidates with ingested media, build ingested-by-URL maps and user fingerprints, and limit cloud-reference images.
- `shown-media.service.ts` — records and deduplicates already-shown media (`HarnessShownMedia`) so media-list follow-ups never repeat content across history.

## Intake flow recap

1. Controller validates DTOs and the `x-harness-llm` header (`400` fast).
2. Image parts → buffers + metadata, deduped against session hashes.
3. `HarnessQueueService.emit` enqueues the job envelope (`buffers`, `meta`, `filters`) → `202` with realtime coordinates.
4. `HarnessProcessor` (`harness.processor.ts`) picks the job off the queue and drives the step engine.
5. Failure at any point → envelope copied to the DLQ (see **1.3**), `error` event to the room.

## Why a queue at all?

Because models are slow and flaky and users are not. Queueing decouples HTTP latency from inference latency, gives us retries/backoff for transient Ollama hiccups, enables cancellation and re-instatement, and lets horizontal workers scale against KeyDB without touching the API process.