v0.11.0

SSRF gate + size caps for fetches (builtin, crawl4ai entry points, PDF and image downloads)

• The builtin provider, the crawl4ai entry points, and PDF and image downloads now go through an SSRF gate (web/safe_http.py): http/https only; hostnames resolving to private, loopback, link-local, multicast, reserved, CGNAT (RFC 6598), or unspecified addresses are refused (cloud metadata, RFC1918, [::1], 100.64.0.0/10, …). Not covered: the fixed-host API calls in core/embed.py and core/scholar.py, which take no attacker-influenced URLs. On the safe_get paths (builtin provider, PDF and image downloads) redirects are followed manually and every hop is revalidated. The browser lanes (crawl4ai fetch(), fetch_many(), and the visible-window path) are gated at entry AND re-checked on the URL the browser actually landed on — the browser follows redirects internally, so the entry check alone cannot vouch for the destination. A post-hoc recheck cannot stop the request that already fired (blind SSRF survives it), but it keeps private-network content out of the vault; the same-host case skips the extra DNS lookup. Refused URLs in a batch are logged and skipped, not fatal. The hostname check is best-effort against DNS rebinding (httpx re-resolves at connect time); the residual window is documented in the module docstring.
• [fetch] allow_private_hosts — escape hatch for self-hosted mirrors and intranet sources: hostnames (exact, case-insensitive) or CIDRs that may be fetched despite resolving to private space. Empty by default; consulted by check_url everywhere, including redirect hops and the final-URL recheck. A malformed CIDR entry errors loudly instead of being silently ignored.
• Response-size caps, streamed and enforced mid-body so a lying or chunked server cannot exhaust memory. Configurable via new [fetch] settings max_html_bytes (10 MiB), max_pdf_bytes (25 MiB), max_image_bytes (2 MiB). A gate refusal on an image download is printed, not silently swallowed.
• No unverified-TLS retry. A certificate failure on a PDF fetch is a refusal that names the existing pdf_verify_tls = false opt-out — never an automatic verify-off retry, which a MITM could force with a bad cert. The refusal is its own exception type (CertVerificationError), and a cert-refused PDF is never handed to the browser fallback lane: that lane runs with TLS errors ignored, so "falling back" there would be the automatic unverified retry by another name. In a batch it is a loud skip instead.

Scholarly discovery: eight sources through one client layer

Academic discovery used to be four URL templates rendered into the agent's instructions by core/agent_docs.py, which the model was trusted to assemble and call by hand. No retry, no rate limiting, no dedup, no offline tests — and one of those templates shipped mailto=research@example.com, a shared placeholder on every install, which is exactly the anti-pattern the open-access resolver refuses to commit for Unpaywall. It is now a real package.

• hpr scholar search and hpr scholar sources. One query hits every configured provider, merges records that are the same work, and returns one list. sources lists what is wired, what each covers, and why anything is unavailable — so a user is never guessing which source to reach for.
• Dedup is by DOI first, then normalized title within ±1 year. The year tolerance is deliberate: providers disagree systematically about online-first versus print year for the same article. Two different DOIs never merge regardless of title, which is what stops four 2025 reprints of a famous paper from collapsing into one record. A work confirmed by several providers carries them in also_in, with the highest citation count and the longest abstract.
• --limit is per provider, not a cap on the merged list. A post-merge cap would show only the first provider's records at small limits and make every other source look empty.
• Providers: OpenAlex, Crossref, CORE, DOAB, ClinicalTrials.gov, SEC EDGAR, FRED. Each is a real client through one cache-first, rate-limited HTTP seam (scholar/base.py), with fixtures matching live response shapes. OpenAlex's abstracts arrive as an inverted word-position index and are reconstructed; Crossref's arrive as JATS XML and are stripped; ClinicalTrials.gov and EDGAR were verified against the live services, including EDGAR's User-Agent gate.
• Non-STEM coverage is a first-class goal. OpenAlex and DOAB return books and book chapters, which matters because in the humanities the book is the unit of publication and nothing in the stack could find one before. DOAB is the only source that finds the book rather than a review of it.
• RePEc is listed as unavailable, on purpose. Their API documents that it has no search function. Shipping an honest "cannot search" with a pointer to OpenAlex for the DOI-bearing series beats silently omitting the field.
• HYPERRESEARCH_CONTACT_EMAIL replaces the placeholder. Set it and OpenAlex and Crossref serve you from their polite pools, and SEC EDGAR — which rejects any request without a contact address — becomes available. Unset, no address is sent at all.
• Specialist records are tagged, not disguised. Trials, filings and economic series come back with work_type set so the pipeline never treats a 10-K as a paper.
• FRED_API_KEY is never cached. FRED authenticates by query parameter and the cache keys on URL, so FRED requests bypass the cache rather than write the key into the vault's SQLite in plaintext.

CORE is the third open-access resolver

Recovery used to ask Unpaywall, then Europe PMC. But contact_email is empty by default, which disables Unpaywall, and Europe PMC is biomedical only — so a stock install's open-access recovery covered almost nothing outside biomedicine while the 0.10.0 notes presented it as a headline feature. CORE now runs third: it is the largest full-text open-access aggregator, and unlike Unpaywall it hosts the plain text directly rather than pointing at a repository that may 403.

• Activates when CORE_API_KEY is set; skipped silently otherwise, the same way Unpaywall is skipped without a contact address.
• Every existing invariant holds and is tested: a candidate is accepted only if it is longer than what we had and clears oa_min_full_text_chars; failure is soft; resolver URLs go through check_oa_url; oa_max_attempts is honoured; the four-place disclosure contract is populated; oa_source gains the value core.
• Version is recorded honestly. CORE does not reliably say which version it holds, so oa_version stays unset unless CORE marks the record a preprint — and the banner then says the version is unrecorded and tells the reader to quote with care, rather than implying version of record.

Agent prose now points at the client

The "Academic APIs before web search" section of the injected agent instructions tells agents to run hpr scholar search and not to hand-assemble API URLs, and explains what each source is for.

Contributed fixes

• run finish no longer blocks every light-classified run started on the installed gear (@maximilliangrand in #95). verify_run() took its required-artifact step set from the manifest's profile_steps, so a run initialized with --profile full whose step-1 decomposition classified it light was asked for critic-findings-*.json and patch-log.json — artifacts the light tier correctly never writes, because steps 12 and 14 are skipped by the tier gate. The run did all its light-tier work and then sat at blocked (verify) with no legitimate way to pass. The gate now resolves the tier declared in prompt-decomposition.json when it disagrees with the manifest profile, which is what the router already documents ("the manifest's profile field is informational — the decomposition's tier rules"). A missing, unreadable or unknown tier still falls back to profile_steps.
• The MCP fetch_url tool works again, and a deleted note's URL can be fetched again (#84, fixed by @AmirF194 in #86). Two bugs on the same path. Crawl4AIProvider.fetch() and fetch_many() called asyncio.run(), which raises before the coroutine runs when the caller already has a loop — and the MCP server dispatches sync tools on its own loop thread, so the tool failed on every call while the CLI never noticed. Both now go through a wrapper that runs the coroutine on a dedicated thread when a loop is present. Separately, sources.note_id is ON DELETE SET NULL, so deleting a note leaves its row behind with a NULL id; every duplicate-URL check tested row truthiness, so that URL was DUPLICATE_URL ("already fetched as note 'None'") forever. All four fetch paths (fetch_and_save, hpr fetch, hpr fetch-batch, hpr research) now share one orphan-aware check. The three CLI paths also record the source with INSERT OR IGNORE, which was a no-op on the orphaned row too — so the new note was left with no source record, and hpr fetch's duplicate-race detector then deleted the note it had just written and reported note_id: null. Those paths now reclaim the orphaned row after the insert; the guard on note_id IS NULL leaves a genuine race winner untouched.
• The PreToolUse reminder now reaches the model (#94; @dajiaohuang in #97 and #98). The installed hook wrote its reminder to stderr and exited 0, and for PreToolUse that channel is discarded — so the "check the research base before searching the web" nudge shipped since 0.x had never once been delivered. It is emitted as hookSpecificOutput.additionalContext now, which is the documented injection channel. The matcher is narrowed to WebSearch|WebFetch: on Glob and Grep the advice is noise, and delivering it there for the first time would have made every local vault operation pay for a reminder about the web.
• Parallel Search is a fifth web provider (@georgeatparallel in #57). [web] provider = "parallel" searches through Parallel's keyless Search MCP endpoint — the first search-capable provider that needs no account, since builtin cannot search at all and exa and tavily both need a key. Search only: bulk fetch waves degrade to per-URL, so it complements the crawl4ai fetch path rather than replacing it. The README gained a Web providers section listing all five on equal footing; none had been documented there before. This PR had been red on CI since August for an asyncio.run()-inside-a-running-loop error — the same class of bug #86 fixed — and got the same loop-safe runner.