v1.13.5

Changes

Prompture could talk to almost any model, but an agent built on it still had to bring its own way to search, read a page or watch a video, and nobody could easily tell which of those actually worked on a given machine. This PR adds that capability layer: web search, page and URL readers, transcription, an MCP hub, wrapped CLI tools, domain packs and a research agent, all built on a shared foundation of ordered backend chains that fail over instead of failing. prompture doctor reports what works, which backend is active and the exact fix for anything that doesn't. It also fixes GPT-6 models (including gpt-6-astra) being rejected, and Ollama ignoring max_tokens.

Highlights

• Web search, page fetching and URL readers work with no API keys at all; adding keys raises quality and limits, and every result says which backend served it and whether it fell back
• read_url understands YouTube, GitHub, Hacker News, arXiv, Wikipedia, RSS/Atom feeds and podcasts, and falls back to a plain page fetch for anything else
• Repeat searches and static pages come back instantly from a local cache, even in a new process; weather, prices and news expire in minutes, papers and transcripts in days, and cached results say how old they are
• prompture transcribe and summarize_media turn a video, podcast or audio file into a timestamped transcript and a summary with key points; audio is never sent to a second provider without opting in
• prompture research "question" produces a cited report from pages it actually opened, within fetch, token, cost and time budgets
• Agents mount tools by name: tools=["web:all", "mcp:exa", "pack:finance", "cli:gh"]
• prompture mcp add --preset exa registers MCP servers once and reuses them by name; secrets are stored only as ${ENV} references
• prompture doctor shows every provider, tool, media piece, MCP server and binary with its status, active backend and fix; prompture setup / configure store keys in an owner-only file with profiles
• prompture skill install gives coding agents a skill describing all of this, and prompture check-update / watch report new releases
• GPT-6 models (gpt-6-astra, gpt-6-luna, gpt-6.1-sol) no longer fail with "Unsupported parameter: max_tokens", and Ollama now honors max_tokens, temperature and the other sampling controls

Technical changes

Capability foundation (prompture/capabilities/)

• BackendChain tries Backends in order, skips unavailable ones and classifies failures through classify_error: transient errors retry once, auth/quota/rate-limit/challenge errors move to the next backend, and fatal errors re-raise with the attempts attached as exc.attempts. Results are ChainResults with route = {served_by, fallback, attempts}
• An env override (e.g. PROMPTURE_SEARCH_PROVIDERS) moves named backends to the front and keeps the rest in place after them; unknown names are ignored so a stale value can't hide working backends. only= restricts the chain
• classify_backend_error pins UnsafeURLError (fatal), ChallengePageError and ResponseTooLargeError (failover) to fixed actions, so reset_error_rules() can't turn an unsafe URL into a failover
• probe_command runs a side-effect-free command with a UTF-8 child env and stripped interpreter variables and classifies ok | missing | broken | timeout | error; exit 126/127, exec failures and "No Python at" style output count as broken. cached_probe shares results for 60 s
• HealthStatus and a capability registry (register_capability, check_capabilities) that doctor walks; a crashing check becomes an error row instead of breaking the report
• normalize_public_http_url allows only public http(s): it blocks private, loopback, link-local, metadata, CGNAT, multicast and reserved ranges, IPv4-mapped / 6to4 / Teredo / NAT64 wrappers, decimal/octal/hex IPv4 spellings and internal hostnames, and checks every resolved address. PROMPTURE_WEB_ALLOW_PRIVATE is the escape hatch
• GuardedHTTPAdapter checks the address each socket actually connected to before any request bytes are sent, closing the DNS-rebinding gap between validating a hostname and connecting to it. safe_get mounts it on its own session and on any requests.Session passed in; connections to a configured proxy are exempt
• safe_get follows redirects manually and re-validates each hop, streams the body with a byte cap, detects challenge pages and raises typed HTTPStatusErrors that carry status and headers. Proxies come from PROMPTURE_<BACKEND>_PROXY or PROMPTURE_PROXY
• is_challenge_page scores combined Cloudflare / captcha markers in the first 4 KB, requiring more evidence on 2xx responses
• scrub_url_credentials / scrub_secrets remove URL userinfo, bare user:pass@host, secret-looking query/fragment parameters, known API key shapes and bearer tokens. They're applied to DriverHTTPError messages, every capability error and health row, and a new SecretScrubbingFilter installed by configure_logging, which also scrubs rendered tracebacks (exc_text) and stack info
• prompture/tools/named.py resolves "namespace:name" tool specs; Agent, AsyncAgent and DeepAgent accept them next to callables and ToolDefinitions

Web tools (prompture/tools/web/)

• Search backends tavily, exa, serper, brave, jina, searxng and keyless exa_mcp (a minimal JSON-RPC client over streamable HTTP, no mcp dependency) behind web_search / asearch, with domain filters, recency, URL dedupe and a SearchResponse carrying the route
• web_fetch chains jina_reader and direct (safe_get + stdlib HTML to Markdown, trafilatura when installed), validates the URL before any backend sees it, pages long content with start= and caches for 10 minutes
• read_url routes to readers (YouTube transcript API, then yt-dlp subtitles, then transcription; GitHub REST, then gh; HN Firebase; arXiv API; Wikipedia REST; podcast enclosures; feeds), with per-reader step overrides via PROMPTURE_READER_<NAME>_BACKENDS
• search_platform for YouTube, GitHub (repositories, issues, code), Hacker News and arXiv; WebToolkit and the web: namespace build agent tools that return strings and never raise
• WebSearchTool keeps its signature; with nothing configured it now falls back to keyless search instead of raising
• New prompture/tools/web/cache.py: search, fetch, read_url and platform search share a cache built on the existing MemoryCacheBackend / SQLiteCacheBackend (default SQLite at ~/.prompture/cache/web_cache.db, 5,000 entries; PROMPTURE_WEB_CACHE=memory|off, PROMPTURE_WEB_CACHE_PATH). Lifetimes come from ttl_for: volatile queries (weather, prices, scores, news, recency_days <= 1) and fast-moving hosts 10 min, feeds 15 min, platform search 30 min, ordinary pages 1 h, other searches 6 h, Wikipedia and podcasts 1 day, papers, PDFs, DOIs, commit-pinned GitHub files and YouTube transcripts 7 days
• read_url caches the full reader result and pages from it, so walking a long transcript never refetches. Errors and empty results are never stored, values over 2 MB stay out of the disk cache, and hits carry route["cached"] / cache_age_s plus a "cached N min ago" footer. Every entry point takes use_cache= and cache_ttl=; clear_web_cache(), cache_info() (read-only, never creates the database) and a web_cache doctor row round it out

Media understanding (prompture/media/understand/)

• transcribe accepts a URL or file, downloads direct audio through safe_get and video/podcast pages through yt-dlp, then downmixes and chunks long media with ffmpeg and stitches the timestamps. It enforces caps on source size, chunk count and upload bytes, and cleans up temp files in finally
• auto picks the first configured STT provider (Groq, OpenAI, ElevenLabs; reorder with PROMPTURE_STT_PROVIDERS); a second provider is used only with allow_provider_fallback=True
• New GroqSTTDriver, STT usage recorded through record_media_usage, summarize_media map-reduce summaries with timestamped key points, transcribe_media / summarize_media agent tools, and prompture transcribe

Doctor and updates

• prompture.doctor.check_all() returns a DoctorReport with a versioned prompture.doctor/1 JSON schema. The providers section checks key presence, base URLs, SDK importability and driver import offline, and --live lists models only, never generating anything paid. Unconfigured providers collapse into one row
• prompture doctor, check-update (one PyPI call cached for 24 h, with changelog highlights) and watch (exit 1 when something is broken, 2 for an update only with --fail-on-update)
• The companion info payload gains a health summary, cached and never blocking

Setup and credentials

• CredentialStore writes ~/.prompture/credentials.yaml only on first write, owner-only (0600, or a user-only ACL via icacls on Windows), atomically, refusing symlinks and junctions, with a 1 MB read cap and profiles
• Settings reads the store below env vars and .env; get_config_value gives the same precedence to proxies and backend overrides
• detect_environment tells local machines from servers (container, CI, SSH, headless); prompture setup, configure and reset all support --dry-run, validate keys live and never echo them

MCP hub (prompture/mcp/hub.py)

• Named servers in ~/.prompture/mcp.json plus a project .prompture/mcp.json that wins by name. Literal secrets are refused in favor of ${ENV} references resolved at connect time
• resolve_mcp_tools backs mcp:<name> with pooled sessions on a background loop so sync agents can call them, per-server timeouts, <server>__<tool> prefixing and usage-ledger entries per call
• http_session for streamable HTTP, presets (exa, deepwiki, github, fetch, time, git, memory, filesystem, sequential-thinking), opt-in import from editor configs, and prompture mcp add|list|remove|enable|disable|test|import
• build_mcp_server works on both mcp 1.x (FastMCP) and 2.x (MCPServer)

CLI tools and packs

• CLITool turns a declared argument schema into an argv list with no shell. Only declared subcommands are allowed, flag injection is blocked, every run has a timeout and output cap, injected env values are never logged, and shells and Windows batch launchers are refused
• Built-in gh (read-only; gh api forced to GET) and yt-dlp tools, plus user definitions from .prompture/tools.yaml
• Tools defined in a project's .prompture/tools.* are untrusted: health checks and cli:all only locate their executable and never run version_args / live_check_args, so running prompture doctor in a cloned repository can't execute its code. Shipped tools and ~/.prompture definitions are still probed, and a config file can't mark itself trusted
• finance, news, dev and places packs built on Tukuy plugins, the web readers and the PyPI/npm JSON APIs; tools missing a key are left out and listed in the pack's health row

Research agent (prompture/research/)

• ResearchAgent plans sub-questions, fans out searches, platform queries and packs in parallel, then ranks and dedupes URLs with per-domain caps and reads the top pages, transcribing media when useful
• Under a token or cost budget, pack LLM calls take turns: the budget check and usage recording happen under one lock, so parallel packs can't all start against the same remaining budget. The default model reads PROMPTURE_RESEARCH_MODEL / PROMPTURE_DEFAULT_MODEL from the credential store too, so the model chosen in prompture setup is used
• Synthesis cites only opened sources and strips any other [n]. Budgets are enforced across threads; when a limit hits, the agent answers from what it already read, or falls back to cited excerpts
• prompture research, research_tool() and ResearchEvent progress events

Agent skill and docs

• Bundled SKILL.md with standing rules and a routing table to per-capability references. It states which capabilities are keyless (web search and reading) and that failover is per capability, with transcription never moving audio to a second provider without consent; prompture skill install|uninstall|show writes only to an explicit target and removes only directories it created
• llms.txt, docs/agent-install.md and docs/agent-update.md

Fixes

• _default_tokens_param picks max_completion_tokens by model generation (gpt-N with N ≥ 5, plus the o-series) for both openai and azure. GPT-6 models previously got max_tokens and a 400, and Azure sent max_tokens even for gpt-5.5
• Ollama drivers move temperature, top_p, top_k, num_ctx, seed, repeat_penalty, min_p and num_predict (from max_tokens) into the options object Ollama reads, and forward think
• The CLI replaces characters the console can't encode instead of crashing on piped Windows output, and loads capability commands from their own modules
• The MCP tool-name digest uses sha1(..., usedforsecurity=False), since it only keeps names unique
• The companion router stores a reusable non-streaming answer before writing the body to the client. It used to store it afterwards, so a near-identical request sent the moment the first one finished could miss the cache and reach the vendor again

Packaging and config

• New web (feedparser, youtube-transcript-api, trafilatura) and media (yt-dlp) extras, both included in all; skill files shipped as package data
• New EXA_API_KEY setting; GITHUB_TOKEN now also serves the GitHub reader
• .env.copy documents every new variable. The README gains an intro section right after the quick start, a Capabilities group in Key Features and a full Capabilities guide; top-level exports include the new APIs, with doctor and research loaded lazily so import prompture stays as fast as before

Details

• Version: v1.13.5
• PR: #114 by @jhd3197
• PR URL: #114