Skip to content

ods 0.0.2

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 06 Oct 03:54
· 108 commits to main since this release
v0.0.2
f18ed9a
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

The second release: dashboard screens backed by real signals, configurable health
checks with a CI gate, what reuse saves, and failures explained node by node. It
migrates the state store and changes the dashboard's JSON APIs: see Breaking.

  • Health: each node gets a health badge (failing, warning, healthy or
    unknown) from checks [health] configures, with the findings and evidence behind
    it; ods health check gates CI on them, and the health_check SDK contract lets
    checks be added. Home shows coverage of tests, descriptions, constraints and source
    freshness.
  • New dashboard screens: Freshness evidence, the ERD page and the Impact simulator.
  • What reuse saved: every run says so, and ods state savings totals it, as time
    and, with [state.cost], as cost, from a run ledger in the state store.
  • Why it failed: ods state explain-failure explains one node's failure, and
    Databricks failures are checked against messages recorded from a real warehouse.
  • Install: Homebrew and Chocolatey packages, beside PyPI and the release archives.

Breaking

  • The dashboard's JSON view models (/api/home, /api/catalog, /api/shell and the
    other page APIs) are now at schema_version 3 (#354): a Catalog row's health is an
    object (health, reasons) instead of null, Home's coverage rows replace
    placeholder with how and uncovered, and its count rows gain of, how, href
    and note. What to do: a script reading these APIs reads the new fields; the
    live stream (/api/runs/live, live_schema_version) is unchanged.
  • The SQLite state database migrates to version 2 the first time an ods state
    command that opens it for use (run, build, plan, history, test, export, …)
    runs (a new runs table, the run ledger, ADR-0029); a copy of version 1 is kept beside
    it. ods state savings and ods state doctor read it without migrating. What to
    do:     nothing, unless you go back to an older ods, or the database is read-only: an
    older ods refuses the migrated database, so restore the copy (ods state doctor
    lists it); a read-only one must be migrated where it is writable first.
  • SDK_VERSION is now 0.8 (0.0.1 shipped 0.6). The state_store contract is 0.3, with
    record_run and runs (the run ledger) behind the new capability run_ledger; both
    default to Unsupported (#210). The health_check contract 0.1 is new (#392, under
    Added). What to do: rebuild out-of-process plugins against SDK 0.8; a store
    that keeps no ledger needs no other change.

Added

  • ods health check runs the health checks [health] configures and gates CI on them:
    exit 5 when a check at severity error fails, and with --strict also when one
    couldn't decide; --json for one document. Each run is kept beside the state store as
    a versioned health record (<state db>.health/, the newest 20), unless --no-record
    (docs) (#392)
  • The health_check SDK contract 0.1 (capability health_check), with a fake and a
    conformance suite: checks registered through it run beside the built-ins, under a
    timeout, and one that errs, times out, skips a node or answers about one it wasn't
    asked about is unknown, never a pass. Findings carry machine-readable evidence
    (sorted keys and values) beside their reason, in the dashboard's findings too.
    SDK_VERSION is now 0.8 (#392)
  • [health] in ods.toml tunes the dashboard's health checks
    (ADR-0030): turn each
    built-in check (built, last_run_failed, last_run_skipped, tests_required,
    tests_passed) off or set its severity (error, warn or info), scope it with
    select/exclude by resource type, tag, path glob or name, and choose whether a check
    that couldn't decide counts as unknown or a warning. Each badge now lists its
    findings, one per check (docs) (#394, #392)
  • Health and coverage on the dashboard from real signals, replacing the [n]
    placeholders. Each node in the Catalog has a health badge (failing, warning,
    healthy or unknown), with its reasons and a health filter. Home counts nodes by
    health, stale nodes and runs with failures, and shows coverage of tests, descriptions,
    column constraints and source freshness, each listing what it misses. What ODS can't
    measure reads not measured, never 0
    (docs) (#391, #354)
  • The dashboard's Freshness evidence screen (/catalog/sources, under Catalog):
    for each source and seed, the evidence ODS has about its data now, its grade
    (exact, semantic, proxy, inferred, unknown), what its readers were last
    built from, their decisions with a link to Why, and every node downstream. Unknown evidence always means build. Its numbers are the plan's, as
    ods state explain gives them; /api/catalog/sources returns the same view
    (docs) (#386, #350)
  • In a terminal that follows hyperlinks, the run's journal path (ods state run,
    build, seed, snapshot, test, retry, ods state history --run) is a link to
    the file. ODS only links paths and pages it builds itself; set FORCE_HYPERLINK=0 to
    turn links off, or 1 to force them (#379)
  • Conformance suites for the sql_lineage and observed_lineage contracts, so every
    contract in ods-sdk now has one; the SQL parser (in every dialect), the Unity
    Catalog lineage export reader and the fakes run them. A new guide, Writing a provider,
    shows how a provider built outside this repository runs the suites
    (docs) (#373, #99)
  • What reuse saved, as a cost: [state.cost] rate_per_hour and unit in ods.toml
    make ods state savings add the cost avoided at that rate, per run and in total (an
    estimate, like the time). The dashboard's Runs page shows what reuse saved, from the
    run ledger (savings in /api/state/runs) (docs)
    (#371, #210).
  • ods state savings [--since DATE] [--limit N]: what reuse saved, per run and in
    total, as estimates from each build's last measured time. It reads a new run ledger in the state
    database, to which every run that goes ahead adds an entry, a run with nothing to build
    included (docs,
    ADR-0029) (#370, #210).
  • ods state explain-failure <node> [--run <id>]: why one model, seed, snapshot or test
    failed in the last run (or any run whose journal is kept), with the evidence and what
    to try, as ods state history --run explains it. A node that didn't fail says how it
    ended, and a skipped one what blocked it. ods mcp gains the matching
    ods_explain_failure tool and a --state-db option
    (docs) (#368, #348).
  • ods state run, build, seed, snapshot and retry say what reuse saved:
    saved: ~2.3s of build time (estimate, serial: 9 of 13 nodes reused), and savings
    in JSON (also in ods state plan, for what a run would save). Each build's time is
    recorded with its state and kept while it is reused; reused nodes without a time are
    counted, not guessed (docs,
    ADR-0029) (#369, #210).
  • The ERD page on the dashboard (/erd): the project's entities, keys and relationships
    as ods erd generate --infer finds them. Each edge is drawn by its evidence (tested,
    declared, joined in SQL, or inferred), with cardinality where keys prove it. Click an
    edge for its evidence, scope it with --select and a depth, hide inferred edges or
    non-key columns, and export SVG. Each untested relationship comes with the
    relationships test to paste. /api/erd returns the same as JSON
    (dashboard) (#366, #64).
  • The Impact simulator on the dashboard (/lineage/impact): propose a rename, type
    change or drop of one or more columns and see which models must run (the same set as
    ods lineage impact), which would break because their SQL names a column that goes
    away, which only lose it through select *, and which can't be told because their
    lineage is unknown; plus what can be skipped, the column trail, the ods state build -s …
    command for exactly what must run, and the tests that run. Opened from a column's Simulate link on its Model page or the
    explorer's Impact tab; /api/lineage/impact returns the same as JSON
    (dashboard) (#365, #347).
  • Homebrew: brew install buchochelliq-labs/tap/ods installs the release binary on
    macOS and Linux (Install) (#362, #212).
  • Chocolatey: each release is packaged as opendatasuite and pushed to the Chocolatey
    community repository, where it is listed once Chocolatey approves it
    (Install) (#363, #212).

Changed

  • ods state plan and ods state build --dry-run list what is built first, then what
    is reused, each in plan order, as separate sections of the table, with a footer of
    totals in the terminal. Run results tables end with a totals row too. Plain output
    keeps its table shape; only the row order of plans changes (#377)
  • Each failure under Why it failed is framed in the terminal, titled with what failed
    (customer_segments failed, test not_null on customers.customer_id failed). In plain
    output that title replaces the failed:/failed test: line (#378)
  • Failure explanations on Databricks are checked against messages recorded from a real
    dbt-databricks run on a SQL warehouse (error catalogue version 6). A model contract's
    not_null or check constraint that the model's rows break is now recognised as a
    constraint violation, and a Spark error's position (line 31 pos 4) gives the failing
    line. A table in a missing schema is explained as a missing table, as Databricks
    reports it (reference) (#383, #349)

Fixed

  • A carriage return in a value shown in the terminal (a node name, an engine's message or
    output line) is now shown as a line break: on its own it moved the cursor back, so a
    value could overwrite what was printed before it (#375, #192)
  • Error summaries no longer keep a quoted value written straight after a word: a prefixed
    literal (X'…', r'…', E'…') or a value glued to a word (v'…') can't be told from
    an apostrophe, so the rest of the line is removed from there. Property tests over ods-core's graph order, fingerprints,
    timestamps, strategy choice and redaction found it (#374, #192)

Install

pip install opendatasuite==0.0.2        # or: pipx install / uv tool install opendatasuite
cargo binstall --git https://github.com/buchochelliq-labs/open-data-suite ods-cli

Homebrew, Chocolatey, direct downloads and verification: https://buchochelliq-labs.github.io/open-data-suite/install/

Checksums

8f4dac9f25b0cc496fd2b0c2277b4e4c8ccbd20b2b452ffa14cdf7af56427f65  ods-v0.0.2-aarch64-apple-darwin.tar.gz
4289c26a66e1847dc421447e235f8458cd6f9e804cb5b26655742c8fe53459ad  ods-v0.0.2-aarch64-unknown-linux-musl.tar.gz
c058d05899fe82ec37ca8c6c9173d8f9466b03bc201228bbf57b35abbd82cf96  ods-v0.0.2-x86_64-apple-darwin.tar.gz
fd3598dcf70312641e124aea7db82e053b03821909a62ec119312083094e7a8e  ods-v0.0.2-x86_64-pc-windows-msvc.zip
31b0ef2ec94aaee3b393f5560eec6d505036d5a62c3e2278bd93892a24626eba  ods-v0.0.2-x86_64-unknown-linux-musl.tar.gz
86a8af7a36c7354fa0baf25c0aacb829d58d7513a10109ec9a100ea39890b9e1  ods.rb
2e50cb3895a418399bfcae223fdd36fc61a0339ecfc6e401d69b38e387a7ae4d  opendatasuite-0.0.2-py3-none-macosx_10_12_x86_64.whl
d1bcfca467eaf5871b45cc834024edbbd6d3539838c92161f97b97ce923bb635  opendatasuite-0.0.2-py3-none-macosx_11_0_arm64.whl
1a3b8d9419540984914667c69b7c70bf5ee78ebb41b6e6ca7040492ab9b5fbec  opendatasuite-0.0.2-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
9f1579230d02534d0edd6fc034fe7bb00134ad3573e846857bc7a6ef37c1beaf  opendatasuite-0.0.2-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
bc16c6bf88f8d382f9921c6f85efd9aec040060b6f99763efda863f5a5da876d  opendatasuite-0.0.2-py3-none-musllinux_1_2_aarch64.whl
92f30f379fb515492b37f7becbf540579b42b865c7be7e7db19302767ffb3dd0  opendatasuite-0.0.2-py3-none-musllinux_1_2_x86_64.whl
576786e2fb29baf0abc14528286d0206a8f498638768f586f6b3276febafaa8e  opendatasuite-0.0.2-py3-none-win_amd64.whl
Assets 16
Loading