WebVector 0.3.0

WebVector 0.3.0

package	install
webvector	npm i webvector
webvector-mcp	claude mcp add webvector -- npx -y webvector-mcp
webvector-cli	npx -y webvector-cli search "…"

New: the markets module — news, SEC filings, an event calendar, crowd sentiment and a market pulse for trading and research agents, built only from free, keyless sources that were probed and classified (see docs/MARKETS.md and the research notes). Nothing in the research pipeline changed; everything here is additive and opt-in.

npx -y webvector-mcp --tools research,fetch,markets   # five extra tools

const news = await wv.markets.news({ symbols: ['NVDA'], windowMs: 12 * 3_600_000 });

tool / call	what you get
webvector_news · wv.markets.news()	headlines per ticker (or a market briefing) from Yahoo, Seeking Alpha, Bing, CNBC, MarketWatch, Benzinga, GlobeNewswire, PR Newswire and the Fed — relevance-filtered, deduped across sources with a spread count, event-tagged (earnings, guidance, M&A, FDA, analyst, insider, offering, legal, macro…), newest first; read: N attaches article bodies through the normal robots-respecting fetch
webvector_filings · wv.markets.filings() / searchFilings()	SEC EDGAR filing history and full-text search: decoded 8-K item codes, Form 4, S-3/424B, 13D/G, and the primary-document URL on sec.gov/Archives for every hit
webvector_calendar · wv.markets.calendar()	scheduled macro prints with ET times, impact and forecast/previous/actual, recent Fed press releases, and earnings dates
webvector_sentiment · wv.markets.sentiment()	StockTwits bull/bear skew, posting velocity and top posts, plus FINRA daily short-sale volume share
webvector_pulse · wv.markets.pulse()	Cboe VIX and FRED yields / fed funds, with index and futures quotes when the gray Yahoo source is enabled

Source policy, stated up front. Every source in markets/sources.ts is classified open (documented API/feed, robots allows it), feed (syndication endpoint on a host whose robots targets page crawlers) or gray (keyless, but robots/terms/UA requirements argue against automation). Gray sources — Google News RSS, Nasdaq, the Yahoo chart API — are off unless you set markets.graySources. Reuters, Bloomberg, Dow Jones pages, FT, Seeking Alpha articles, TradingView, Stooq, CME and Reddit JSON are deliberately absent; a headline from a feed may be shown, but the page behind it is never scraped.

How it behaves. Requests reuse the existing polite Fetcher (SSRF guard, per-host pacing, bot-wall detection, size caps) plus a TTL cache with ETag/Last-Modified revalidation in the page-cache SQLite. One deadline per call (markets.deadlineMs, default 12 s) with every source's outcome reported (ok / cached / stale / failed / timeout, and n/m failed for partial failures); a slow or broken source never fails the call. SEC requests declare WebVector/<ver> (<contact>) — set WEBVECTOR_CONTACT_EMAIL, which the fair-access policy requires.

Core additions: markets config section (graySources, feedRobots, disableSources, contact, deadlineMs; WEBVECTOR_MARKETS_* env), the webvector/markets subpath export, FetchInit.robots: 'skip' for syndication endpoints (default respect), Fetcher.setHostMinInterval(host, ms), and Bing News apiclick redirector unwrapping in cleanUrl.

Security: the ToolGuard domain policy now also applies to the unwrapped target of redirect-wrapper URLs (google.com/url, bing.com/news/apiclick, …), closing an allow/blocklist bypass for webvector_fetch. Markets tool output is capped to --max-tokens, every returned string is sanitised and length-capped, and body reads honour --allowed-domains / --blocked-domains. See the new row in SECURITY.md.

No new dependencies; the default tools/list is unchanged unless you opt in with --tools markets.

WebVector 0.2.0

WebVector 0.2.0

package	install
webvector	npm i webvector
webvector-mcp	claude mcp add webvector -- npx -y webvector-mcp
webvector-cli	npx -y webvector-cli search "…"

Breaking (MCP): tools are now webvector_research / webvector_fetch / webvector_search (they collided with Anthropic's built-in web_search/web_fetch and Claude Code's tools). Pass --legacy-tool-names to keep the old names as aliases for one release. New: webvector_verify.

Defaults that changed: retrieval.fusion: rsf, retrieval.lexicalWeight: 1.5, retrieval.lexicalRelativeCutoff: 0.3, ingestion.cache.dir: auto (SQLite page + embedding cache under ~/.cache/webvector), ingestion.acceptMarkdown: prefer, ingestion.useProviderContent: auto, ingestion.html.strategy: auto.

Measured on the bundled eval (32 recorded cases): lexical MRR .84 → .96, hit .95 → 1.0; hybrid MRR .74 → .95; extraction F1 on the 40-fixture corpus .88 → .96; second run of the same research: 0 HTTP requests.

Research-driven 0.2: retrieval, extraction, fetch robustness, agent-facing MCP surface, persistence.

Retrieval — BM25F fields (title/heading/body) with proximity, quoted phrases and identifier-aware tokens; BM25+ δ; relative-score fusion (retrieval.fusion: rsf, default) with lexicalWeight 1.5; per-source candidate cap (fixes long pages starving primary sources); lexical relative cutoff, lexical MMR, autocut; per-domain preference; adjacent-chunk merge; query-focused highlights; token-budget packing with an explicit omission footer; xQuAD aspect coverage for relatedQueries (result.coverage); recency boost tied to freshness; corroboration count; LLM-free evidence gate (result.evidence) with suggested queries and optional autoRetry; source-authority priors; verifyCitations() quote-grounding check; explain option.

Extraction / fetch — markdown-first content negotiation (ingestion.acceptMarkdown) with a served-markdown cleaner; bot-wall classifier (FETCH_BLOCKED_BOT, FETCH_PAYMENT_REQUIRED, never retried); Content-Signal etiquette; early abort on non-content types + maxHtmlBytes; URL hygiene (redirect unwrap, AMP/mobile folding, #:~:text= hint); fast paths (arXiv HTML, GitHub README/blob/issues, Google Docs, npm/PyPI, Hacker News and Stack Exchange APIs); provider-content quality gate; opt-in Wayback fallback; extractor ensemble with page-type routing and a recall guard (ingestion.html.strategy); code/table fidelity pre-pass; JS-shell detection (PARSE_NEEDS_JS) + pluggable render hook (ingestion.render); __NEXT_DATA__/RSC recovery; paywall-guarded JSON-LD articleBody; ranking-grade metadata (dates, canonical URL, kind, language, word count); same-host boilerplate suppression; PDF page citations (#page=N); a 40-fixture extraction regression corpus.

MCP / agent surface — tools renamed webvector_research, webvector_fetch, webvector_search (+ webvector_verify, webvector_status); --legacy-tool-names keeps the old names as aliases for one release; server instructions (≤ 2 KB); rewritten descriptions; response_format: concise|detailed, slim structuredContent, max_tokens with omission footer, depth presets, objective, category, deadline_ms, auto_retry, max_age_ms/cache_mode; fetch pagination (start_index), include_links, selector; errors that teach; server-minted sessions; --max-uses / --allowed-domains / --blocked-domains / --user-location guardrails; research and verify_claim prompts; MCP registry files; Anthropic search_result blocks; link stripping and text-fragment deep links.

Persistence / DX — SQLite page cache on by default (~/.cache/webvector/pages.sqlite, ETag/Last-Modified revalidation, per-call maxAgeMs/cacheMode), persistent embedding cache, store.provider: sqlite, single-flight + negative cache, stats.usage (+ opt-in cost estimate), webvector cache stats|ls|clear|prune, doctor --fix/--json, config JSON Schema + webvector init, opt-in OpenTelemetry spans, ingestion.maxCrawlDelayMs, self-describing User-Agent.

Testing — recordingFetch() in webvector/testing; offline retrieval eval (npm run eval, 32 recorded cases, both tiers, baseline gate in CI).

WebVector 0.1.0

First public release of WebVector — web research for AI agents in one tool call: search → read the full pages → rank → cited passages.

Packages (npm)

Package	Install	What it is
webvector	npm i webvector	the library — new WebVector().research(query) + AI SDK / Anthropic / OpenAI / LangChain bindings
webvector-mcp	npx -y webvector-mcp	MCP server: web_research, web_fetch, web_search, webvector_status (stdio + Streamable HTTP)
webvector-cli	npx -y webvector-cli search "…"	CLI: search, fetch, serp, doctor, init, config, mcp

Try it in 30 seconds

npx -y webvector-cli search "what changed in the MCP spec in 2026?" --stats
claude mcp add webvector -- npx -y webvector-mcp

No API keys required. ~12 MB install (lexical BM25 tier); add @huggingface/transformers or any embedding API key and it switches to on-device / hosted semantic search automatically.

Highlights

• 11 search providers (DuckDuckGo keyless, Brave, Serper, SerpAPI, Google CSE, SearXNG, Tavily incl. keyless, Exa, Perplexity, Wikipedia, custom) with automatic fallback chains
• Embeddings: none (BM25), local Transformers.js presets, OpenAI, OpenAI-compatible, Gemini, Voyage, Cohere, Mistral, Jina, Ollama, Vercel AI SDK models
• Stores: in-memory (sessions with TTL), Chroma, Qdrant, pgvector · Rerankers: local cross-encoder, Cohere, Voyage, Jina, LLM
• Retrieval: multi-query expansion, BM25 + vector hybrid (weighted RRF), near-duplicate removal, per-source caps, MMR
• Ingestion: polite concurrent fetcher (robots.txt, per-host limits, size/redirect/time caps), SSRF guard enforced at connect time, Readability→Markdown, PDF, charset sniffing, markdown-aware chunker with breadcrumbs and offsets
• Every passage carries URL, title, offsets and score; failures are reported per URL, never thrown

Docs: README · Configuration · Providers · Security