feat: introduce agentic search pipeline with live trace streaming

.gitignore

@@ -30,10 +30,19 @@ coverage
 # Rust build output
 target/
 
+# Python dev environments (reader sandbox pytest venv, etc.)
+.venv/
+__pycache__/
+*.pyc
+.pytest_cache/
+
 # Git worktrees
 .worktrees
 .claude/worktrees
 .gstack/
 
-# Superpowers generated docs (design specs, implementation plans) — never commit
+# Superpowers generated docs (design specs, implementation plans): never commit
 docs/superpowers/
+# Superpowers brainstorming visual-companion working files
+.superpowers/
+*.profraw

README.md

@@ -1,5 +1,3 @@
-
-
 <h1 align="center">
   Thuki - WIP
 </h1>
@@ -16,7 +14,6 @@
   A floating AI secretary for macOS. Fully local, completely free, zero data ever leaves your machine.
 </p>
 
-
 <p align="center">
   <img src="https://img.shields.io/badge/status-beta-yellow.svg" alt="Beta" />
   <a href="LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License" /></a>
@@ -48,7 +45,6 @@ Double-tap Control <kbd>⌃</kbd> to summon Thuki from anywhere. Ask a question,
 
 https://github.com/user-attachments/assets/57df0efe-24eb-4875-a83d-e605e0c6f8b4
 
-
 ### Overlay Mode
 
 Thuki floats above every app, including fullscreen ones. Highlight text anywhere, double-tap Control <kbd>⌃</kbd>, and Thuki opens with your selection pre-filled as a quote, ready to ask about.
@@ -137,7 +133,30 @@ bun run sandbox:stop
 
 For the full architecture and security philosophy behind the sandbox, see [`sandbox/README.md`](sandbox/README.md).
 
-### Step 2: Install Thuki
+### Step 2: Setup the search sandbox (Optional, required for /search)
+
+The `/search` command uses an agentic search pipeline that depends on two local Docker containers: a **SearXNG** meta-search engine and a **Trafilatura** reader. This setup ensures that your search queries and the content you read remain entirely local.
+
+**Prerequisite:** [Docker Desktop](https://www.docker.com/get-started) must be running.
+
+1. **Start the search services**
+
+   ```bash
+   bun run search-box:start
+   ```
+
+2. **Verify services (Optional)**
+
+   ```bash
+   # Search Engine check:
+   curl "http://127.0.0.1:25017/search?q=thuki&format=json"
+   ```
+
+   Without this service running, the `/search` command will be disabled in the chat, but all other features will remain available.
+
+   For more details on the agentic search pipeline, see [docs/agentic-search.md](docs/agentic-search.md).
+
+### Step 3: Install Thuki
 
 #### Download (Recommended)
 
@@ -210,8 +229,8 @@ Contributions are welcome! Read [CONTRIBUTING.md](CONTRIBUTING.md) to get starte
 
 Thuki is macOS-only, but the community has been busy bringing it to other platforms. Huge shoutout to these contributors 🎊🚀!
 
-| Platform | Repo | Author |
-|----------|------|--------|
+| Platform      | Repo                                               | Author                                       |
+| ------------- | -------------------------------------------------- | -------------------------------------------- |
 | Windows 10/11 | [ThukiWin](https://github.com/ayzekhdawy/thukiwin) | [@ayzekhdawy](https://github.com/ayzekhdawy) |
 
 > Each port is independently maintained by its author. For issues or questions about a specific port, head to that repo directly.

package.json

@@ -23,8 +23,10 @@
     "format": "prettier --write \"src/**/*.{ts,tsx,css}\" && cd src-tauri && cargo fmt",
     "format:check": "prettier --check \"src/**/*.{ts,tsx,css}\" && cd src-tauri && cargo fmt -- --check",
     "typecheck": "tsc --noEmit",
-    "sandbox:start": "docker compose -f sandbox/docker-compose.yml up -d",
-    "sandbox:stop": "docker compose -f sandbox/docker-compose.yml down -v",
+    "llm-box:start": "docker compose -f sandbox/llm-box/docker-compose.yml up -d",
+    "llm-box:stop": "docker compose -f sandbox/llm-box/docker-compose.yml down -v",
+    "search-box:start": "docker compose -f sandbox/search-box/docker-compose.yml up -d --build",
+    "search-box:stop": "docker compose -f sandbox/search-box/docker-compose.yml down",
     "test": "vitest run",
     "test:watch": "vitest",
     "test:coverage": "vitest run --coverage",

sandbox/README.md → sandbox/llm-box/README.md

@@ -20,7 +20,7 @@ The sandbox separates model initialization from the inference runtime, keeping c
 | **Breakout Mitigation** | Active | `cap_drop: ALL` strips every Linux kernel capability |
 | **Privilege Control** | Active | `no-new-privileges: true` blocks setuid/setgid escalation |
 | **Read-Only Filesystem** | Active (inference only) | `sandbox-server` root filesystem is read-only; only `/tmp` is writable. `sandbox-init` requires write access to pull the model. |
-| **Ephemeral Lifecycle** | Active | `bun run sandbox:stop` runs `down -v`, permanently destroying all model weights |
+| **Ephemeral Lifecycle** | Active | `bun run llm-box:stop` runs `down -v`, permanently destroying all model weights |
 | **Non-Executable Weights** | Active | GGUF format is math-only; no Python/Pickle code execution risk |
 
 > **Note on network egress:** The sandbox does not use `internal: true` on the Docker network. On macOS, Docker Desktop's networking layer does not support `internal: true` alongside host port binding, so the isolation strategy relies on `127.0.0.1` ingress restriction, `cap_drop: ALL`, and the read-only filesystem instead. Outbound connections from the container are not hard-blocked at the network level.
@@ -38,15 +38,15 @@ The sandbox is intended for:
 **Start the sandbox:**
 
 ```bash
-bun run sandbox:start
+bun run llm-box:start
 ```
 
 The first run pulls the model inside the init container, which may take several minutes depending on your connection. Subsequent starts are instant.
 
 **Stop and wipe the sandbox:**
 
 ```bash
-bun run sandbox:stop
+bun run llm-box:stop
 ```
 
 This runs `docker compose down -v`, which destroys the Docker volume and permanently removes all downloaded model weights from disk. Nothing persists after this command.

sandbox/search-box/docker-compose.yml

@@ -0,0 +1,65 @@
+# ==============================================================================
+# Search Sandbox: SearXNG Local Search Engine
+#
+# Provides a privacy-respecting, locally-hosted meta-search engine for Thuki's
+# /search command. Aggregates results from Google, Bing, DuckDuckGo, Brave,
+# and many specialized engines without rate limiting (local use only).
+#
+# SECURITY CHECKLIST:
+# [x] NETWORK INGRESS: 127.0.0.1 binding - no external access
+# [x] PRIVILEGE ESCALATION: no-new-privileges enforced
+# [x] CAPABILITY RESTRICTION: only CHOWN/SETGID/SETUID retained (required by uwsgi)
+# [ ] RATE LIMITING: intentionally disabled for local performance
+# ==============================================================================
+
+services:
+  searxng:
+    image: searxng/searxng:latest
+    container_name: thuki-searxng
+    restart: unless-stopped
+    ports:
+      - "127.0.0.1:25017:8080"
+    volumes:
+      - ./searxng:/etc/searxng:rw
+    environment:
+      - SEARXNG_BASE_URL=http://127.0.0.1:25017/
+    cap_drop:
+      - ALL
+    cap_add:
+      - CHOWN
+      - SETGID
+      - SETUID
+    security_opt:
+      - no-new-privileges:true
+    networks:
+      - search_net
+
+  reader:
+    build:
+      context: ./reader
+    image: thuki-reader:local
+    container_name: thuki-reader
+    restart: unless-stopped
+    ports:
+      - "127.0.0.1:25018:8000"
+    networks:
+      - search_net
+    cap_drop:
+      - ALL
+    security_opt:
+      - no-new-privileges:true
+    read_only: true
+    tmpfs:
+      - /tmp:size=16m
+    mem_limit: 512m
+    cpus: 1.0
+    healthcheck:
+      test: ["CMD", "python", "-c", "import urllib.request,sys;sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8000/healthz',timeout=4).status==200 else 1)"]
+      interval: 30s
+      timeout: 5s
+      retries: 3
+      start_period: 10s
+
+networks:
+  search_net:
+    driver: bridge

sandbox/search-box/reader/Dockerfile

@@ -0,0 +1,25 @@
+FROM python:3.12-slim AS base
+
+ENV PYTHONDONTWRITEBYTECODE=1 \
+    PYTHONUNBUFFERED=1 \
+    PIP_NO_CACHE_DIR=1 \
+    PIP_DISABLE_PIP_VERSION_CHECK=1
+
+WORKDIR /app
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+      ca-certificates \
+    && rm -rf /var/lib/apt/lists/* \
+    && addgroup --system --gid 10001 reader \
+    && adduser --system --uid 10001 --gid 10001 --home /nonexistent --no-create-home reader
+
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+COPY main.py .
+
+USER reader:reader
+
+EXPOSE 8000
+
+CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "2"]

sandbox/search-box/reader/README.md

@@ -0,0 +1,132 @@
+# Reader service
+
+Trafilatura-based URL-to-markdown extractor. Second stop of Thuki's agentic `/search` pipeline.
+
+## What it does
+
+Takes a URL, fetches the page, strips boilerplate (navigation, ads, footers, cookie banners), and returns clean markdown the synthesis LLM can cite against.
+
+```
+POST /extract { "url": "https://example.com/article" }
+  -> { "url": "...",
+       "title": "Page title",
+       "markdown": "# Article text\n\nCleaned body...",
+       "status": "ok" | "empty" }
+```
+
+## Why Thuki needs it
+
+SearXNG returns URLs plus short snippets (usually the first 150-200 chars of the page). For many queries, snippets are enough. For questions like "compare tokio vs async-std benchmarks in 2026," the answer lives deep inside blog posts and docs pages that snippets never surface.
+
+The pipeline's judge decides snippet sufficiency after the initial SearXNG round. When the judge returns `Partial` or `Insufficient`, the reader is called to fetch the top URLs in full and hand rich text to the next judge round. This is the classic "RAG reader" pattern from Perplexity, Exa, and the CRAG / Self-RAG literature.
+
+## Why Trafilatura
+
+HTML boilerplate removal is a surprisingly hard problem. Naive approaches (strip `<nav>`, `<footer>`) fail on modern SPAs where everything is `<div>`. Getting it right requires heuristics built over years of research. Two parallel research agents independently landed on Trafilatura as the best-in-class open-source solution:
+
+- **F1 ~0.95** on the ScrapingHub article extraction benchmark, top of the field.
+- Apache 2.0 license.
+- Production use at HuggingFace, IBM, Microsoft Research, Stanford, EU Parliament.
+- Pure Python, no browser, tiny attack surface.
+
+We considered and rejected: Firecrawl (AGPL-3.0 blocks bundling), Jina Reader cloud (proxies every URL through Jina's servers, violates privacy), Crawl4AI (Chromium in container, 4 GB RAM, CVE history), ScrapeGraphAI / ReaderLM-v2 (LLM per page, wrong shape), DIY Playwright (SSRF surface without extraction value), most Rust readability crates (weaker extraction, Jan 2025 benchmark showed many return empty strings on real pages).
+
+## How it fits into the pipeline
+
+```
+snippets judge returns Partial / Insufficient
+  -> reader.fetch_batch_cancellable(&top_urls, &cancel)
+    -> POST /extract for each URL in parallel (semaphore-bounded, 5 in flight)
+      -> Trafilatura extraction per page
+        -> chunker splits markdown into ~500-token chunks
+          -> BM25 rerank picks top chunks for the query
+            -> chunks judge decides sufficiency
+              -> synthesis OR gap-query loop
+```
+
+The Rust `search::reader::ReaderClient` calls this service over HTTP. It races each call against a cancellation token and degrades gracefully when the reader container is unreachable (emits `ReaderUnavailable` warning, pipeline falls back to snippets).
+
+## Architecture
+
+Single-file FastAPI app (`main.py`, ~90 lines). One endpoint (`/extract`) and a healthz probe. Entire service fits in your head:
+
+```
+main.py
+├── _validate_url        -> SSRF guard (scheme + private-host blocklist)
+├── fetch_html           -> httpx stream with 8s timeout + 2MB byte cap
+├── trafilatura.extract  -> boilerplate removal, markdown conversion
+└── trafilatura.extract_metadata -> page title
+```
+
+The Dockerfile is standard Python-slim hardening: non-root user, minimum install, no build tools in the final layer.
+
+## Security posture
+
+Enforced at three layers:
+
+**App layer (`main.py`):**
+- SSRF guard rejects non-http(s) schemes plus private, loopback, link-local, multicast, and reserved IP ranges (both IPv4 and IPv6) plus the literal string `"localhost"`.
+- Byte cap: upstream fetch aborts once 2 MB is buffered. Prevents hostile servers from exhausting memory.
+- Timeout: 8s hard ceiling on upstream fetch.
+- Request body limits (URL max length 2048 chars, validated via Pydantic).
+
+**Container layer (`docker-compose.yml`):**
+- `cap_drop: ALL` (no capabilities, not even the reduced set SearXNG needs)
+- `no-new-privileges: true`
+- `read_only: true` root filesystem
+- `tmpfs: /tmp:size=16m` for the minimal writable scratch area
+- `mem_limit: 512m`, `cpus: 1.0`
+- Bound to `127.0.0.1:25018` only
+
+**Image layer (`Dockerfile`):**
+- Runs as `reader:reader` (uid/gid 10001, system user, no home directory)
+- Only `main.py`, `requirements.txt`, and pinned runtime deps land in the image
+- No pytest, no dev tools, no compilers
+
+## Files in this directory
+
+| File | Purpose | Shipped? |
+|---|---|---|
+| `main.py` | The service code (FastAPI app) | Yes (production) |
+| `Dockerfile` | Container build recipe | Yes (production) |
+| `requirements.txt` | Pinned runtime deps (6 packages) | Yes (production) |
+| `requirements-dev.txt` | Pinned test deps (pytest only) | No (local-only) |
+| `test_main.py` | Unit tests (5 cases) | No (local-only) |
+
+Dev artifacts like `.venv/` and `.pytest_cache/` are gitignored and never enter the image.
+
+## Local development
+
+```bash
+# Bring up the reader container (also pulls the image on first run):
+bun run sandbox:start
+
+# Exercise the endpoint:
+curl -sS -X POST http://127.0.0.1:25018/extract \
+  -H 'Content-Type: application/json' \
+  -d '{"url":"https://example.com/"}' | jq
+
+# Healthcheck:
+curl -sS http://127.0.0.1:25018/healthz
+
+# Tear down:
+bun run sandbox:stop
+```
+
+### Running pytest without Docker
+
+```bash
+cd sandbox/search-box/reader
+python -m venv .venv
+.venv/bin/pip install -r requirements.txt -r requirements-dev.txt
+.venv/bin/python -m pytest test_main.py -v
+```
+
+`.venv/` and `.pytest_cache/` are gitignored.
+
+## What the reader is not
+
+- Not a browser. It does not render JavaScript. Pages that rely on client-side rendering come back as `status: "empty"`. This is tracked in pipeline telemetry; if empty-body rate gets high in production we add a Playwright fallback in v2.
+- Not a crawler. One URL in, one markdown blob out. No link following, no sitemap parsing, no depth-limited traversal.
+- Not a cache. Every call fetches fresh. Caching belongs upstream in the Rust pipeline if we ever need it.
+- Not a general-purpose service. The endpoint accepts only http(s) URLs pointing at public hosts. Private networks and non-web schemes are rejected 400.