Skip to content

edaphos 0.8.0 — Pillar 1 at literature scale

Choose a tag to compare

@HugoMachadoRodrigues HugoMachadoRodrigues released this 22 Apr 13:30
· 177 commits to main since this release

Summary

Promotes Pillar 1 (Causal AI) from a ten-abstract demo to a
literature-scale extractor. The LLM + Knowledge-Graph pipeline now
handles tens of thousands of abstracts end-to-end.

New features

  • Paginated corpus clients. causal_corpus_openalex() cursor-pages
    through the OpenAlex Works API transparently past the 200-per-page
    cap. causal_corpus_deduplicate() collapses DOI + title overlap
    between SciELO and OpenAlex.

  • Resumable cached ingestion. causal_llm_ingest_corpus() gains
    cache_dir + max_retries. Per-abstract tools::md5sum() keys each
    JSON cache entry; cache hits short-circuit the LLM call so
    interrupted runs resume exactly where they stopped. Malformed
    responses trigger exponential backoff; exhausted rows land in
    attr(kg, "failed"). A txtProgressBar tracks runs of ≥ 5 rows.

  • Live AGROVOC SPARQL alignment. causal_ontology_agrovoc_align()
    and causal_kg_alignment(vocab = "agrovoc") resolve KG labels
    against FAO's live AGROVOC endpoint, picking the
    Levenshtein-nearest skos:prefLabel and caching resolutions to disk.

Bundled live corpus

  • inst/extdata/cerrado_claims_real_corpus.jsonl — 210 causal claims
    extracted by Ollama + Gemma-4 from 100 deduplicated OpenAlex
    abstracts across three Cerrado queries, aligned against AGROVOC.
    Reproduced end-to-end by data-raw/run_large_corpus.R.

Documentation

  • Vignette pilar1-causal gains §10 "Scaling to tens of thousands of
    abstracts" deriving the pagination, cache, and AGROVOC alignment
    primitives from first principles.
  • README + roadmap + package doc updated to reflect the upgrade.

Quality

  • 105 / 105 causal tests pass (17 new scale tests covering pagination,
    cache hits, retry on malformed responses, AGROVOC Levenshtein
    selection, and kg_alignment(vocab = "agrovoc") plumbing).
  • R CMD check --as-cran clean: 0 errors / 0 warnings / 1 harmless
    "unable to verify current time" NOTE.

🤖 Generated with Claude Code