v0.8.0
Pluggable data & literature providers. Specialists now discover data
sources in light of the research question — FRED and yfinance reach the
tool loop via list_data_sources + a unified fetch_data, and Allium sits
behind a Warehouse capability (its 5 guardrails unchanged). They also
pull the researcher's own reference library (local .bib, LOCAL_DATA_DIR,
and Zotero via the Web API) and read full-text PDFs (read_reference).
Both lanes are now registry-pluggable, so new providers are drop-in.
Cross-lane
scripts/live_check.py— live smoke harness. Exercises the real
data/literature provider paths (yfinance, FRED, Allium connectivity,
OpenAlex search,read_referenceon an OA PDF, Zotero library) against
live services, auto-skipping providers without credentials. No LLM calls
(free). Complementsmake smoke(offline/mocked) andmake smoke-paid
(full LLM run). Run:python scripts/live_check.py.
Lane C — Data
- Allium folded behind a
Warehousecapability (M3b of
docs/MODULARIZATION_PLAN.md). Allium is now a first-class registered
provider:AlliumWarehouseowns itscard(),tools()(→ALLIUM_TOOLS)
andhandler()(→DeferredAlliumToolHandler);_run_pipelineassembles
it by iteratingwarehouses(settings)instead of hardcoding, and the
catalog builds its card from the warehouse. Pure refactor — same condition
(Allium key present), same tools, the 5QueryValidatorguardrails and
approval flow are untouched, andhas_allium/data_module_enabledare
unchanged. Completes the Lane-C registry (series + warehouse). - Series data in the agent loop + RQ-aware discovery (M3a of
docs/MODULARIZATION_PLAN.md). FRED and yfinance are no longer
CLI-only — specialists reach them in the tool loop. NewSeriesFetcher
capability + data registry (providers.py,registry.py) mirror the
Lane-B pattern. Two new tools:list_data_sources(serves the registry
catalog so the agent picks the right source for the research question)
and a unifiedfetch_data(provider, method, params). Allium is unchanged
— it keeps its guardedquery_alliumtool and is advertised in the
catalog (the 5 guardrails are untouched). Series tools are always on
(yfinance needs no key); budgeted (_MAX_FETCHES=20). M3b will fold
Allium behind aWarehousecapability into the same registry.
Lane B — Literature
- Fix: literature search crashed on OpenAlex/S2 explicit nulls. A live
search returned 0 papers becauseopenalex._parseraised
'NoneType' object has no attribute 'get'on a result whose
primary_location.source(oropen_access/authorships) was an
explicitnull—.get(k, default)doesn't apply the default for a
present-but-null value. Both parsers now guard withor {}/or [].
Regression tests added (the mocked payloads previously only used
well-formed fields, so the bug only surfaced live). - Full-text
read_referencetool (M2.5 ofdocs/MODULARIZATION_PLAN.md).
Specialists can now read a reference's PDF in full to deepen the lit
review, not just its abstract. Newread_referenceliterature tool takes
apdf_url(surfaced in search/fetch results and on[PDF]-marked
reference-list entries, incl. Zotero attachments) or adoi(resolves an
open-access PDF). Downloads (auth'd for Zotero hrefs,/file/view→
/file), extracts text via pypdf (pdf.py), and returns it
truncated to ~20K chars. Tightly budgeted (_MAX_READS=6+ per-read char
cap) given the prior 522K-token literature blowup.fetch_bytesgained a
max_bytesoverride (PDFs exceed the 2 MB default). Newpypdfdep. New
ZoteroLibraryReferenceLibraryreads the researcher's Zotero library
via the Web API's native JSON (zotero.py), maps items to
PaperMetadata, and captures each item's primary PDF attachment href
(for the planned on-demandread_referencetool, M2.5). Config:
ZOTERO_API_KEY+ one ofZOTERO_USER_ID/ZOTERO_GROUP_ID; merged
into the reference summary after local.bib, deduped by (title, year).
Unset → no-op. Syncfetch_text_synchelper added for the (sync)
reference-library path. Degrades to[]on any Zotero error — can't
break paper creation. - Provider interface + registry (M1 of
docs/MODULARIZATION_PLAN.md).
Formalized the de-facto interface the source modules already shared into
capability sub-types —SearchSource(web discovery; OpenAlex, arXiv,
Semantic Scholar) andReferenceLibrary(the researcher's own corpus;
LocalBibLibraryoverLITERATURE_BIBTEX_FILE+LOCAL_DATA_DIR) — in
newproviders.py/registry.py.LiteratureToolHandlerand
_load_reference_summarynow iterate the registry instead of hardcoding
provider names. Pure refactor: the search (OpenAlex→arXiv) and DOI-fetch
(OpenAlex→S2) fallback chains are reproduced exactly; +13 tests, no
behaviour change. This is the seam Zotero (M2) and Citavi (M4) plug into.