fray

A rope frays upstream before it snaps. fray is a production-ML guard agent for DataHub: it compiles each deployed model's upstream contract out of DataHub's end-to-end ML lineage, re-scans it on a schedule, and catches the silent breakage that takes models down — schema drift, freshness stalls, volume cliffs, null spikes, lineage rewires — before anyone notices predictions going bad. Every break is ranked by blast radius (which production deployments inherit it), and everything fray learns is written back into DataHub as assertions, run events, tags, institutional memory, and a handoff brief saved as a DataHub Document — so the context graph itself gets smarter with every scan.

Built for the Build with DataHub: Agent Hackathon — Production ML Agents track (overlaps Agents That Do Real Work).

Why this is not another data-quality agent

Generic DQ agents check datasets someone configured, against rules someone wrote. fray starts from the other end:

1. The contract is derived, not configured. fray walks mlModelDeployment → mlModel → mlFeatures → source datasets → full upstream dataset closure through DataHub lineage and freezes what production actually depends on: fields and types, freshness, row counts, per-column stats, lineage edges. Zero config per model.
2. Findings are temporal. fray diffs today's graph against the frozen baseline, so it catches changes — a column that quietly became VARCHAR, a table that stopped landing, a pipeline rewired under the feature store — not just static rule violations.
3. Every finding carries blast radius. A null spike on a staging table is scored by which models and which production deployments transitively consume it. On-call sees "fraud_detector prod endpoint at risk", not "table X has nulls".
4. It writes back. Assertions + FAILURE run events with evidence on broken datasets, SUCCESS heartbeats on clean monitored ones, fray:contract-broken / fray:at-risk / fray:monitored tags, institutional-memory links on at-risk models, and a markdown handoff brief stored as a DataHub Document via the Agent Context Kit. The next agent (or human) that opens DataHub inherits the full picture.
5. Optional Claude triage. With ANTHROPIC_API_KEY set, a tool-use loop groups raw findings into incidents by probable root cause and writes the paging message. The deterministic engine works without it.

60-second demo (no infrastructure)

Requires CPython 3.10-3.13. acryl-datahub 1.6+ needs 3.10 or newer, and pydantic-core has no 3.14 wheels yet, so 3.14 falls back to a Rust build that fails on PyO3's version cap. macOS ships 3.9 as python3, so use a venv:

brew install python@3.12                # macOS; any 3.10-3.13 works
python3.12 -m venv .venv && source .venv/bin/activate

git clone https://github.com/seekdaseek/fray && cd fray
pip install datahub-agent-context==1.7.0
pip install acryl-datahub==1.7.0        # see "upstream pin bug" below
pip install -e . --no-deps
fray demo

fray demo runs the whole engine against committed fixtures: a two-model production stack (fraud + churn, Snowflake → Feast → MLflow → SageMaker) where day-1 contains four silent breaks. You'll see the contract compiled, all four breaks detected and ranked by blast radius, and examples/out/ filled with the baseline, the scan, the handoff brief, and the exact 30 MetadataChangeProposals a live run would publish to DataHub. Exit code is non-zero-safe for CI: 0 when HIGH findings were correctly raised on the demo data.

The same fixtures are replayable into a real instance (below), so offline and live mode are the same data.

Live mode (against a real DataHub)

# 1. DataHub quickstart (per their docs)
pip install acryl-datahub && datahub docker quickstart

# 2. Seed the demo ML stack into DataHub, healthy state
fray ingest-demo --day 0 --server http://localhost:8080

# 3. Freeze the upstream contract for every production model found in the graph
fray snapshot --server http://localhost:8080

# 4. Replay the silent breakage
fray ingest-demo --day 1 --server http://localhost:8080

# 5. Re-scan, write findings back into DataHub, run Claude triage
export ANTHROPIC_API_KEY=...   # optional
fray scan --server http://localhost:8080 --write-back --agent

After step 5, open any broken dataset in the DataHub UI: fray's custom assertions with FAILURE run events and evidence are on the Validation tab, tags are applied, at-risk models carry an institutional-memory link to the run brief, and the brief itself is saved as a DataHub Document. fray snapshot with no model filter discovers every mlModel in the graph via get_urns_by_filter, so on a real deployment it monitors whatever production registers — the demo stack is just seed data.

Works the same against DataHub Cloud: --server https://<tenant>.acryl.io/gms --token <PAT>.

What it uses from DataHub

• Context graph reads — typed aspects via the core SDK (SchemaMetadata, UpstreamLineage, MLModelProperties, MLFeatureProperties, MLModelDeploymentProperties, timeseries DatasetProfile), model discovery via get_urns_by_filter.
• Agent Context Kit (datahub-agent-context) — add_tags (server-side merge), save_document for the handoff brief, get_lineage as a live cross-check of the walked closure. The Claude triage loop's tools mirror ACK's shape so the agent layer is swappable.
• Writeback — emit_mcp of custom AssertionInfo + AssertionRunEvent (with nativeResults evidence), GlobalTags, TagProperties, InstitutionalMemory, Documents. The graph is fray's memory and its output.

Upstream pin bug (and workaround)

datahub-agent-context==1.7.0 pins acryl-datahub==1.6.0.6 but imports datahub.api.entities.agent, which only exists in acryl-datahub>=1.7.0 — the package is broken with its own pin, and any project depending on both hits ResolutionImpossible. Workaround used here: install ACK first, then force acryl-datahub==1.7.0 (pip warns, works), then pip install -e . --no-deps. Reported upstream; see UPSTREAM.md for the issue text.

Architecture

src/fray/
  ports.py      GraphPort protocol + two impls:
                  SnapshotPort — JSON fixtures, collects writebacks in memory (demo, tests, CI)
                  AckPort     — DataHubGraph + Agent Context Kit against live GMS
  contract.py   walk deployment→model→features→datasets→upstream closure; freeze DatasetState per dataset
  detect.py     temporal diff rules: schema.field_removed / type_changed / field_added,
                freshness.stall, volume.drop / spike, column.null_spike / unique_collapse,
                lineage.edge_removed / edge_added, model.features_detached, contract.dataset_detached
  blast.py      reverse lineage indices → per-finding impact set + score; models_at_risk ranking
  writeback.py  assertions + run events, tags, institutional memory, Document brief
  report.py     console report + markdown handoff brief
  agent.py      optional Claude tool-use triage loop (list_findings / get_blast /
                get_dataset_state / submit_triage), injectable transport
  cli.py        fray demo | snapshot | scan | ingest-demo
scripts/        make_fixtures.py (fixtures are built from typed SDK aspects — valid by construction),
                ingest_demo.py (replays fixtures into live DataHub as MCPs)
fixtures/       day0.json (healthy), day1.json (four planted silent breaks)
examples/out/   committed output of `fray demo`
tests/          11 tests: contract walk, every planted break, benign-change handling, dedup,
                blast reach, ranking, MCP validity, e2e, agent loop via fake transport

Offline mode is fully tested in CI with no network. Live mode uses only documented, stable SDK / Agent Context Kit APIs (every call verified against acryl-datahub==1.7.0 + datahub-agent-context==1.7.0 signatures); the ingest → snapshot → break → scan → writeback loop is the same code path the demo exercises through SnapshotPort.

License

Apache-2.0 — see LICENSE.