pi-search-boost v0.1.3

pi-search-boost v0.1.3

Correctness and security release for deep research, page fetching, cancellation, and parallel child-agent reliability.

Security

• Closed the redirect-based SSRF gap in fetch_page: untrusted page redirects are followed manually, each Location target is revalidated, and redirect depth is bounded.
• Untrusted local extraction now uses a DNS-pinned node:http(s) transport: the address approved by the guard is the address used by the socket, while the original hostname remains available for Host/SNI. This closes the DNS rebinding check/connect gap.
• Loopback, RFC1918, link-local/cloud metadata, numeric hosts, private DNS answers, and unsafe schemes remain blocked. Clash fake-IP TUN access is security-first opt-in with PI_SEARCH_ALLOW_TUN_FAKEIP=1 and requires every DNS answer to be in 198.18/15.

Deep research correctness

• goal now drives initial searches, focus filtering, excerpt selection, evidence gaps, follow-up queries, and stopping conditions.
• Coverage is computed only from selected evidence excerpts, never from an arbitrary match elsewhere on the page.
• Goal subject terms require evidence from at least two independent domains.
• Time-sensitive goals/queries require the same aligned claim segments on two domains to carry recent dates; unrelated footer/copyright years cannot freshen stale claims.
• Results explicitly report semanticGoalCheck: not_performed; lexical evidence coverage is no longer presented as proof that the semantic goal was satisfied.
• Corroboration now compares claim-sized segments using boundary-aware query anchors and exact version/date/measure facts, rejects conflicting version/lifecycle-status claims, and is clearly labeled heuristic claim alignment rather than proof.
• Custom continuation queries retain a goal-bearing search variant, and domain stagnation must persist for two rounds before stopping while evidence gaps remain.

Fetch and search reliability

• Tavily, Exa, Brave, and Exa MCP now combine caller cancellation with their engine timeout. Cancelling deep_research stops in-flight API work instead of consuming credits until timeout. One Exa MCP deadline covers its whole multi-stage session and bounded empty-result retry.
• Focus filtering caps repeated-word influence, rewards unique concept coverage, uses Latin token boundaries (LTS no longer matches results), preserves borderless/outer-pipe Markdown tables, and retains bounded heading/table context with complete relevant rows.

Parallel research / WebSocket fixes

• Child Pi processes now use -ne before explicitly loading this extension, preventing duplicate tool registration when the npm package is already installed.
• Captures bounded child stderr so provider and startup failures are no longer hidden.
• Transient WebSocket/socket/provider transport failures retry once serially after the parallel wave, reducing concurrent connection pressure; persistent failures include an actionable transport: auto/sse diagnostic.
• Fixed abort cleanup, TERM→KILL escalation, killed-child success misclassification, stale parent session metadata, and Windows Pi CLI resolution without unsafe shell execution.
• Audit records now count actual cited URLs and distinct domains instead of model turns / hard-coded zero domains.

Packaging and tests

• Published package now includes test/ and a Node.js 22.6+ copy-out runner, so npm test, npm run test:blackbox, and npm run test:all work even when the package lives under node_modules.
• Cross-process cache read/merge/rename is protected by a stale-recovering file lock; a true concurrent child-process regression test verifies both writes survive.
• Added regression coverage for redirect SSRF policy, API cancellation, stale-claim/current-footer dates, conflicting lifecycle/version facts, Latin boundaries, borderless Markdown status tables, balanced-parenthesis URL auditing, serial WebSocket retry, and killed-child classification.

v0.1.1

v0.1.0 — npm release

What changed in this release

• Published to npm — pi-search-boost is now installable directly:

  pi install npm:pi-search-boost@0.1.0     # install
  pi -e npm:pi-search-boost               # try once without installing

• Version jumped 0.0.4 → 0.1.0 to mark the npm milestone. All v0.0.3/v0.0.4 features are included (see RELEASE_NOTES_v0.0.3.md / RELEASE_NOTES_v0.0.4.md).

Feature summary (v0.0.3 + v0.0.4)

• fused_search — multi-engine fusion: Tavily + Brave + Exa in parallel (api layer) or keyless exa-free MCP (free layer), URL dedupe, cross-engine scoring, complexity routing, recency decay, domain filters
• fetch_page — focus-filtered reading (~95% token savings)
• deep_research / research_parallel — multi-round loop with coverage checks / parallel subagents
• x_search — real-time X/Twitter search:
  • keyword / semantic / user / thread modes (all four Grok Build sub-tools covered)
  • parallel instant search: hosted x_search (grok login / XAI_API_KEY) ∥ fused multi-engine, merged + deduped
  • works with no credentials (multi-engine + oEmbed ~2s; structured user profiles via anonymous guest GraphQL)
  • /x-login enables the official hosted path (imports grok login or stores an API key); /x-logout disables it and returns to the fallback chain only

npm package contents

index.ts, lib/ (xsearch, xauth, xfallback, engines, extract, research, parallel, cache, audit, layer, util), README (EN/ZH), release notes, AGENTS.md, LICENSE.

Verified

• 33/33 unit tests pass
• type check: 0 new-region errors
• live x_search: keyword 8 results (grok session), parallel merge 13 = x 8 + engines 5, no-credential path ~2s
• credential lifecycle: login → available → logout → fallback-only → login → available

v0.1.0

v0.1.0 — npm release

What changed in this release

• Published to npm — pi-search-boost is now installable directly:

  pi install npm:pi-search-boost@0.1.0     # install
  pi -e npm:pi-search-boost               # try once without installing

• Version jumped 0.0.4 → 0.1.0 to mark the npm milestone. All v0.0.3/v0.0.4 features are included (see RELEASE_NOTES_v0.0.3.md / RELEASE_NOTES_v0.0.4.md).

Feature summary (v0.0.3 + v0.0.4)

• fused_search — multi-engine fusion: Tavily + Brave + Exa in parallel (api layer) or keyless exa-free MCP (free layer), URL dedupe, cross-engine scoring, complexity routing, recency decay, domain filters
• fetch_page — focus-filtered reading (~95% token savings)
• deep_research / research_parallel — multi-round loop with coverage checks / parallel subagents
• x_search — real-time X/Twitter search:
  • keyword / semantic / user / thread modes (all four Grok Build sub-tools covered)
  • parallel instant search: hosted x_search (grok login / XAI_API_KEY) ∥ fused multi-engine, merged + deduped
  • works with no credentials (multi-engine + oEmbed ~2s; structured user profiles via anonymous guest GraphQL)
  • /x-login enables the official hosted path (imports grok login or stores an API key); /x-logout disables it and returns to the fallback chain only

npm package contents

index.ts, lib/ (xsearch, xauth, xfallback, engines, extract, research, parallel, cache, audit, layer, util), README (EN/ZH), release notes, AGENTS.md, LICENSE.

Verified

• 33/33 unit tests pass
• type check: 0 new-region errors
• live x_search: keyword 8 results (grok session), parallel merge 13 = x 8 + engines 5, no-credential path ~2s
• credential lifecycle: login → available → logout → fallback-only → login → available

v0.0.4

v0.0.4 — /x-logout: explicit credential switch for x_search

New

• /x-logout — removes the pi-local credential copy (~/.pi/agent/xsearch-auth.json). The official hosted x_search path is then disabled and x_search uses only the multi-engine / guest-GraphQL / oEmbed fallback chain. grok CLI's own login is never touched; /x-login re-enables the official path.

Behavior change

• The official path must be explicitly enabled. ~/.grok/auth.json is no longer auto-consumed by x_search: previously a grok login on disk was silently picked up; now only XAI_API_KEY env or a pi-local copy written by /x-login (or /x-login -k) unlocks the hosted x_search tool. Without either, x_search routes straight to the fallback chain (~2s multi-engine + oEmbed, structured users via guest GraphQL).
• /x-login status now marks a present-but-not-imported grok file as NOT imported; run /x-login to enable the official x_search path.

Why

An explicit switch: you decide when the official (xAI-hosted) path is used. /x-logout = "my implementation only" (multi-engine route, guest GraphQL, oEmbed); /x-login = "official hosted x_search also available" (parallel instant search).

Verified

• lifecycle: no creds → unavailable → /x-login → available → /x-logout → unavailable (even with grok's file present) → re-/x-login → available
• fallback chain still works after logout: keyword engines+oembed ~3s; user guest-graphql (structured profile)
• type check: 0 new-region errors

v0.0.3

v0.0.3 — x_search: real-time X/Twitter search

New

• x_search tool — real-time X (Twitter) search with four modes:
  • keyword — X advanced syntax (from:user, since:YYYY-MM-DD, min_faves:N, lang:xx)
  • semantic — natural-language relevance search
  • user — structured account profile + recent timeline (followers, bio, verified, posts with engagement) via X's anonymous guest GraphQL
  • thread — full conversation by post id (or x.com/.../status/<id> URL)
• Parallel instant search — keyword/semantic run two channels concurrently and merge, deduped by status id/URL:
  1. the hosted x_search tool (grok login / XAI_API_KEY) — live in-app search
  2. the fused multi-engine route (Tavily/Brave/Exa or exa-free, site-restricted to x.com) — returns in seconds
• Works with no credentials at all — a fast synchronous preflight routes straight to the multi-engine path (~2s instead of waiting out a timeout), with oEmbed full-text enhancement for the top hits; user falls back from guest GraphQL to engine profile links.
• /x-login command — imports your grok login into pi's own directory (~/.pi/agent/xsearch-auth.json), or stores an XAI_API_KEY; status shows the credential chain. OIDC access tokens auto-refresh (discovery + form POST) with best-effort write-back to grok's own auth file.

How it works (no subprocess)

pi itself POSTs the Responses-API request to the sampling endpoint with tools: [{"type": "x_search", ...}]:

• API key → https://api.x.ai/v1/responses (public, docs)
• grok login → https://cli-chat-proxy.grok.com/v1/responses (the CLI's internal endpoint; x-grok-client-version gate satisfied)

Fallback routing (by type)

keyword/semantic → hosted x_search ∥ multi-engine (merged) → multi-engine + oEmbed
user             → hosted x_search → guest GraphQL (structured) → multi-engine profiles
thread           → hosted x_search → oEmbed single-post full text

Fixes / improvements

• search-audit recent now renders xsearch events
• AuditFetchEvent.via accepts "search" (matches extract.ts reality)
• IPv4-forced DNS for direct-to-X fetches (Windows undici IPv6-first connect timeouts)
• <search_balance> tool-routing table now routes X-specific questions to x_search

New files

• lib/xsearch.ts — hosted-tool direct HTTP client + credential preflight
• lib/xauth.ts — credential chain (env → pi-local copy → grok file) + OIDC refresh
• lib/xfallback.ts — multi-engine/oEmbed/guest-GraphQL fallback router

Upgrade

pi install update git:github.com/Mr-remon219/pi-search-boost
# or, if installed from a local clone:
git pull && pi install .

Then /reload in pi, and optionally /x-login to import your grok login.

v0.0.2

v0.0.2 — Layers: keyless free tier via /web_change

Search layers, engine retirement, and hardening. The big change: two switchable layers — a keyless free layer (single engine, no API keys) and the multi-engine api layer — toggled at runtime with /web_change and persisted across reloads.

Features

• /web_change [free|api|show] — switch the active search layer at runtime; persisted to ~/.pi/agent/search-boost-layer.json (default api). fused_search output now reports the active layer; deep_research and research_parallel inherit it automatically (subprocesses read the same state file)
• Free layer: keyless Exa MCP — new exa-free engine adapter speaking the minimal MCP Streamable HTTP protocol (initialize → initialized → tools/call web_search_exa) against mcp.exa.ai; measured 4/4 correct-entity results in side-by-side probing; no API key required
• Layer-aware complexity routing — api: simple = 1×2 (tavily+brave) / medium = 2×3 / complex = 3×3 + tavily advanced; free: all tiers use exa-free with 1/2/3 variants
• fused_search is now the single search entry point — quick lookups pass complexity: "simple"; the companion-package references (web_search / web_fetch) were removed from the policy, tool descriptions, and docs

Removed

• Bing HTML engine — retired. The channel never failed (0% HTTP errors over the audited window), but its entity resolution was wrong on every ambiguous probe: tokio → Tokyo Wikipedia, pi → π, linux.do → linux.org, and one clean query returned an entire page of Australian medical clinics. The en-US market pin reduced but did not fix the wrong-entity pollution
• Brave HTML engine — retired. 80% measured 429 rate from this IP (persistent IP-level rate limiting); unusable as a routing engine

Fixed

• exa-free MCP notification path — notifications/initialized is answered with 202 + empty body; the old parser unconditionally called resp.json() and threw (silently swallowed). Now: read text first, treat empty body on notifications as success, parse SSE last-data:-line or JSON body explicitly
• Dead-pool silent empty results — with the api layer selected but zero API keys configured, fused_search returned nothing with no explanation. Now degrades to keyless exa-free and surfaces a WARNING: line (new warnings[] on the result)
• Audit layer field — search events now record layer; the AuditSearchEvent interface was updated to match
• Tavily credit estimate — /search-audit stats now counts only searches where tavily actually ran (free-layer searches no longer inflate the estimate) and shows a layer distribution line

Other

• research_parallel subtask prompts now instruct subagents to drop to 1 variant on 429 instead of hammering
• Cache keys: exa-free is optionless (unfragmented keys, like the retired HTML scrapers)
• Tests: dropped bingMarketForQuery, added a web-layer state round-trip test, updated cache-key and cross-process cache tests

Upgrade notes

• If you relied on keyless mode: it is now the free layer (/web_change free) — single engine, no cross-engine scoring, expect occasional 429
• If a caller passes engines: ["bing"] or ["bravehtml"], the request is ignored and the active layer's default engines are used (with a warning if that leaves the pool empty)
• Companion package @bytetrue/pi-web-search is no longer referenced; pi remove npm:@bytetrue/pi-web-search if installed

pi-search-boost v0.0.1

v0.0.1 — Multi-engine search enhancement for pi

Initial release. Turns pi's web search into a research-grade capability: fused multi-engine retrieval, deep research loops, parallel subagents, focus-filtered page reading, caching, and full auditability.

Features

• Distributed as a pi package — pi install git:github.com/Mr-remon219/pi-search-boost (or clone + pi install .); pi.extensions manifest in package.json
• fused_search — 5-engine parallel search (Bing + Brave HTML keyless, Tavily, Exa, Brave), URL dedupe, cross-engine scoring, engine provenance per result; keyless mode keeps two independent free channels for cross-checking
• Complexity routing — simple (1×2 engines, 1 credit) / medium (2×3) / complex (3×4 + advanced extraction, 2 credits); simple lookups stop costing like deep research
• fetch_page with focus — dynamic filtering keeps only query-relevant paragraphs: 95% token savings (1136 → 61 words measured); Jina Reader → local extractor → headless-browser fallback chain
• deep_research — multi-round loop (search → fetch → coverage check → follow-ups → converge), per-source corroboration (≥2 independent domains), temporal freshness, mode=step for agent-driven iteration
• research_parallel — 2-4 independent subagent processes (own context, own search budget); measured 3 subtasks in ~65s vs ~160s serial
• x-algorithm-inspired ranking — per-domain soft diversity decay (each further hit from the same domain × 0.7 down to a 0.35 floor) instead of a hard 2/domain cut; score parameters centralized (SCORE_PARAMS)
• <search_balance> policy injection — proactive-search ruleset: when to search / skip / stop (anti-over-search), autonomy rules (curl fallback), coding-time triggers (search before writing against an unsure API), doubt-triggered search ("I'm not sure" is the signal); live search-budget state injected on agent start
• Caching & audit — search cache 6h / page cache 24h (hot hits ~1ms, cross-process); JSONL audit log with tier distribution, Tavily credit estimation, engine errors, repeated-query loop detection; /search-audit and /search-cache TUI commands

Measured performance

Metric	Value
Simple query	~1.0s
Medium query	~3.2s
Complex query	~3.6s
Deep research round	9.8s (converged)
Hot cache hit	1-3ms
Focus filtering	95% token savings
research_parallel (3 subtasks)	~65s (vs ~160s serial)

Install (pi-driven)

# one-command install from git (recommended)
pi install git:github.com/Mr-remon219/pi-search-boost

# optional companion: web_search / web_fetch
pi install npm:@bytetrue/pi-web-search

Or clone and pi install .; manual fallback: install.bat / install.sh. Full guide: README.md.

Optional API keys: Tavily (1000 free credits/mo), Exa, Brave — keyless mode (Bing + Brave HTML + Jina) works without any.

Known limitations

• Without API keys: two free engines (Bing + Brave HTML) — quality slightly lower, and free channels are subject to anti-bot rate limits
• Proactive search is policy-driven (system prompt), not RL-trained — empirically equivalent to model-decided triggering on mainstream agents
• No X/Twitter data source (paid API; guest-token scraping is dead)
• Bing/Brave HTML parsing depends on page structure (changes detected, fail loudly)