Skip to content

Product PRD

Jump to bottom
Nitin Khanna edited this page Jun 14, 2026 · 4 revisions

veda — PRD

Status: LIVING · Updated: 2026-06-15 · Owner: Nitin Deep dive: Technical PRD — repo layout, tool contract, parser specs, tasks.

Client integration: Integration PRD — how clients connect to veda and use the tools safely.

The zoom-out picture of veda: what it is, the one job it does, and what "done" looks like. veda is a self-contained service; it knows nothing about who calls it or what they do with the data.

1. What this is

veda is a standalone platform-read service. Give it a URL, a username, or a subreddit; it returns clean, structured data. Starting with Reddit (veda.reddit), with generic web (veda.external) and future platforms (veda.x, …) as sibling sub-scrapers.

  • Repo: https://github.com/BarterX-Tech/veda-mcp
  • Interface: an MCP server over Streamable HTTP exposing a few read tools.
  • Runs as one long-lived server (locally via launchd; a real host later).

veda is step 1 of modularising a larger system into independent services. Strict isolation is the point: veda has no idea what a caller is building, how the data is used, or that any particular client exists. Its entire contract is requested resource → clean data. It follows its instructions and returns the data — nothing more.

2. The one job, done well

2a. One way to read a platform

Reddit has two front doors: the .json route (clean, but Cloudflare-blocks our traffic) and the HTML page (a stealth browser still loads it; veda parses it).

Use the route that is currently reliable, and return the same canonical shape.

As of 2026-06-10, live probes show Reddit .json returning blocked HTML from this environment, so veda disables Reddit .json by default and uses old.reddit HTML. The .json implementation stays behind a central config flag for future probes.

   any MCP client
        │  calls a tool: fetch_thread / fetch_user / fetch_rules / fetch_url
        ▼
   ┌──────────────────────────────────────┐
   │  veda  (MCP server, Streamable HTTP)  │   configured route ─► scrape HTML
   │   core: transport + parsers           │                                         │
   │   + one rate-limiter + cache          │   ◄────────── same canonical shape ◄────┘
   └──────────────────────────────────────┘
        knows nothing about the caller

The non-negotiable: the HTML route returns the same fields as the structured route. If HTML hands back less, the data is silently degraded. That shape-parity is veda's core quality bar.

2b. Isolation

veda is its own repo + its own process. It imports only the standard library, scraping/extraction dependencies (scrapling, trafilatura, lxml, requests), and the MCP server library. It depends on no caller code and is unaware of any caller. One running server means one rate-limiter against the platform — the single coordination point that keeps us un-blocked.

Internally, veda = scraping core (transport, parsers, the read functions) + a thin MCP shell (registers the tools). Because it's MCP-over-HTTP, veda is the deployable service — no extra web layer later.

3. Why it matters

Whatever consumes veda is only as good as the data veda returns. Two correctness bars veda must keep meeting:

  • Completeness: a thread's comments must come back with their scores, OP-flags, ids, and timestamps — so a caller can tell the top/relevant comments from the rest. The current HTML parser is field-complete and parity-tested.
  • Reliability: rules, share-links, user pages, and generic web pages must resolve over the currently working routes — not silently return empty because one route is blocked.

A clean, well-tested read service is reusable by any client and across platforms, and a single choke point makes rate-limiting + reliability solvable in one place.

4. Fixed correctness gaps

These were the original defects veda was built to eliminate. They are now guarded by unit, parity, contract, and MCP-tool tests.

# Former problem Current implementation
1 HTML thread parse was lossy — dropped comment score / OP-flag / id / timestamp. HTML thread parsing now extracts score, created_utc, is_op, id, permalink, edit/gilding flags, and nested replies.
2 Rules were fetched only over blocked .json → empty list. fetch_rules tries structured JSON only when enabled, then uses old.reddit /about/rules/ HTML.
3 Mobile share links (/s/...) were not resolved before normalization. fetch_thread resolves share links through the stealth HTML path before canonical URL normalization.
4 Post metadata gaps on the HTML route (score=0, wrong comment count). The HTML route extracts post score, created time, comment count, permalink, flair, and state flags.

The remaining policy decision is operational, not a missing implementation: Reddit .json tiers are disabled by default because live probes return blocked HTML from this environment. The implementation remains behind config for explicit probes or a future route recovery.

5. What "done" looks like

  • ✅ A standalone veda repo + MCP server exposing four data tools: fetch_thread, fetch_user (activity + profile sidebar), fetch_rules, fetch_url — plus health_status.
  • ✅ structured-first → HTML-fallback → identical shape inside veda; field-complete parsers (scores, OP-flags, ids, timestamps).
  • ✅ Strict isolation — zero knowledge of any caller; no caller-specific imports or stored caller context (boundary tests enforce it).
  • ✅ One server, one rate-limiter + cache against the platform; robust parsing (deleted bodies, missing nodes, deep trees, pagination).
  • ✅ Health monitoring — per-tool/route success rates, .json-vs-HTML ratio, latency, fetch-tier availability.
  • ✅ External fetch reliability — every host tries the cheap request route first and escalates through working browser tiers; extraction returns markdown-style structured text with title/truncated metadata; a live canary (scripts/veda-canary) guards against silent degradation.
  • ✅ Central scrape-route config — Reddit JSON/HTML and external fetch tiers can be enabled or disabled without code changes.
  • ✅ Local security posture — loopback-only service, bearer-token-gated MCP traffic, private-network URL blocking, and conservative tool throttling.
  • ✅ Ops: always-updated README + a /veda-server start/stop/restart/status command.
  • ✅ Standalone Python client — veda-client under clients/python, importable as veda_client, with contract-lockstep tests against server tools/errors.
  • ✅ Service-shaped locally: remote deployment should keep the same MCP endpoint/tool contract, but production hosting/auth remains a separate effort.
  • ✅ Open source — Apache-2.0, public repo (BarterX-Tech/veda-mcp), protected main branch, community files, no secrets or caller references in code or history.
  • ✅ Proactive ops — scheduled live canary with desktop alerts plus a daily email health report; markup drift is noticed by the operator, not by callers.

6. Scope

In: the standalone veda repo; its MCP server (Streamable HTTP) + tools; the scraping core (transport, parsers, rate-limiter) ported and made field-complete + robust; health monitoring; ops (README, /veda-server); a small standalone Python client package that speaks the server protocol.

Out: anything about callers or how the data is used; remote/production deployment (runs locally first); a generic cross-platform interface (Reddit + external concrete now, generalise when X arrives); authenticated platform APIs / OAuth (public reads only).

7. Roadmap (milestones)

Granular checkboxes in Technical PRD §9:

  • M0 — Stand up the veda repo + MCP server. Init repo, port the scraping core, build the MCP server (5 tools, Streamable HTTP) + rate-limiter, README, /veda-server + launchd, boundary + parity-test harness.
  • M1 — Field-complete thread parser (#1, #4). HTML thread parse returns score/OP/id/created; shape-parity tests. The keystone.
  • M2 — Rules + share links (#2, #3). fetch_rules over HTML; /s/ resolution.
  • M3 — fetch_user with profile fields. Profile parsing is merged into fetch_user; there is no separate fetch_profile MCP tool.
  • M4 — fetch_url (external) tool.
  • M5 — Health monitoring. Success rates, route ratio, latency; status surface.
  • M5.5 — Central scrape-route config. Disable Reddit .json by default, keep route toggles for future probes.
  • M6 — Local security hardening. Loopback-only operation, local bearer token, private-network URL blocking, rate limits, and token-aware service scripts.
  • Route policy decision: keep Reddit .json disabled by default while retaining the implementation behind config.
  • Later: deploy the server to a real host with production auth, managed secrets, HTTPS, and rotation.

Each milestone ships test-first, one reviewable PR, with your sign-off before the next.

8. Decisions (confirmed 2026-06-09)

  • Standalone repo (BarterX-Tech/veda-mcp) · MCP server over Streamable HTTP · one server, one rate-limiter.
  • Strict isolation — veda knows nothing about callers; no caller-specific imports, persisted caller state, or workflow-specific behavior.
  • Scope = pure platform reads; Reddit-concrete now, generalise later.
  • .json policy: Reddit .json is disabled by default (VEDA_REDDIT_JSON_ENABLED=0) because current live probes return blocked HTML. Keep the implementation behind config for future probes.
  • Local auth policy: run/veda-token is a long-lived local bearer token. It does not expire automatically; it remains valid until deleted/replaced and the server is restarted. This is acceptable for local-only usage; remote deployment needs stronger production auth with explicit rotation/expiry.
  • Ops: always-updated README + /veda-server command.

Caller-side adoption (MCP client config, repointing, removing any caller's own scraping) is tracked in each consuming project's own docs — not here, because veda must stay caller-agnostic.

Current state: local service implementation is complete for the documented tools; remote production deployment remains out of scope for this repo phase.

Clone this wiki locally