Skip to content

v0.1.0

Choose a tag to compare

View all tags
@camharris93 camharris93 released this 03 Jun 12:15
· 17 commits to main since this release
release
29cda34

Sediment v0.1.0 — first public release

A dataset-agnostic analytics-engineering framework: drop in any tabular dataset and run it through a complete modern data stack — ingestion → warehouse → transformation → testing → docs → visualization — with AI layered at the build, orchestration, and consumption seams, and deliberately kept out of the deterministic transform path.

One command, no cloud accounts, no API keys for the core data flow, runs offline.

The design stance

The raw → staging → marts flow is hand-written, version-controlled, tested dbt SQL — same result on every run, on every clone. AI assists in building, operating, and querying the pipeline, but never executes the transforms. No LLM regenerates SQL at runtime. Reproducibility is the product.

What's in it

  • Generic ingestion + profiler — any CSV/TSV/Parquet/JSON lands in DuckDB and is profiled to profile.json, with no per-dataset code.
  • dbt Core transform layer on DuckDB — staging, curated marts, and tests, with a single portable warehouse.duckdb file as the entire warehouse.
  • AI build-time scaffolding — profiles a source and proposes a typed/tested staging model plus mart stubs for human review; a deterministic baseline runs with no API key, and the LLM enhances it. scaffold --write installs staging + tests idempotently and never auto-installs marts.
  • Orchestration agent — drives load → run → test, explains test failures in plain English with the offending rows, and flags row-count drift. Try python run.py orchestrate anage --break to watch it catch an injected failure.
  • Trust-first NL→SQL agent — a layered L1–L7 pipeline (intent → generation → static check → dry-run EXPLAIN → read-only execution → plausibility → plain-English answer with a trust badge), with a bounded self-correction loop. A query that can't be validated fails transparently instead of guessing.
  • Streamlit dashboard — Report, Ask (live L1→L7 trust-trace), and Build tabs, with an AI chart builder and an editable, persistable report.
  • Build-a-model-from-chat — promote a validated chat answer into reviewable dbt SQL; preview in a sandbox schema, save as a mart, or run it straight into marts. The chat never silently mutates curated marts.
  • View vs. build governance — build is a property of the deployment, refused server-side (not just hidden) in view mode, so a shared report exposes charts and chat but never model creation.
  • Multi-dataset + multi-table support — derived relationships, join-mart scaffolding, per-dataset schema namespacing (<dataset>_raw/_staging/_marts), and dataset-prefixed model names, so multiple datasets coexist in one warehouse even when they share table names.

Reference dataset

AnAge (the Animal Ageing & Longevity Database), ~4,600 species. The marts answer questions like "which animals live far longer than their body size predicts?" — the body-size→lifespan power law is fit in log-log space with DuckDB's regr_slope/regr_intercept (deterministic SQL, no AI), surfacing the rougheye rockfish, olm, and box turtle as top over-performers.

Getting started

pip install -r requirements.txt
python run.py up           # build the whole pipeline, green
python run.py dashboard    # http://localhost:8501

The core (up, dashboard, docs) needs no API key. The AI layers (scaffold's LLM tier, orchestrate explanations, ask, and the dashboard's Ask/Build/chart features) each use an Anthropic API key — set once via ANTHROPIC_API_KEY.

Known limitations

  • v1 visualization is Streamlit (pure-Python, offline-green); Evidence.dev is the documented upgrade path for a slicker static site.
  • Orchestration is a Python runner rather than a DAG engine (Dagster/Prefect) — a clean later swap behind the same targets.
  • Clone-and-run only; no hosted instance yet.
Assets 2
Loading