Skip to content

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

6 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

blastradius

Watch the 64-second demo on YouTube →

Pre-merge impact analysis for schema changes, powered by DataHub column-level lineage.

Someone opens a pull request that drops a column. Review approves it, because the diff looks fine and nobody can hold the warehouse in their head. Three days later an executive dashboard is blank and an ML feature table is full of nulls.

blastradius answers the question review cannot: what does this column actually feed? It walks DataHub's column-level lineage, finds every downstream table, dashboard, job and ML asset that reads the changed column, names the owner of each, and writes the assessment back into DataHub so the finding lives with the data rather than dying in a PR thread.

$ blastradius scan 'urn:li:dataset:(urn:li:dataPlatform:postgres,raw.orders,PROD)' --drop user_id

                                 Blast radius
 severity   asset                    kind        columns    hops   owners
 breaking   Revenue Overview         dashboard   buyer_id   2      unowned
 breaking   analytics.orders_daily   dataset     buyer_id   1      analytics
 breaking   ml.customer_features     dataset     user_id    1      ml-team

 3 breaking, 0 risky, 0 downstream but unaffected

That is real output against a live DataHub, not an illustration.

Follow the top row. The column is user_id upstream, but by the time it reaches the dashboard it is called buyer_id, and the dashboard is owned by nobody. Grepping the repository for user_id finds neither the rename nor the dashboard. Lineage does.

Why this is not just a lineage viewer

DataHub already shows you lineage. This does three things a viewer does not:

  1. Distinguishes breaking from merely downstream. Being downstream of a table is not evidence of reading a column. The tool runs two lineage passes, one on the column and one on the table, and only calls something breaking when there is column-level evidence. Assets with no fine-grained link are reported separately as unproven rather than padding the scary number.
  2. Ranks by who feels it. A dashboard three hops away outranks a staging table one hop away, because one of them is seen by a human.
  3. Reaches the dashboard, which lineage traversal alone cannot. DataHub's fine-grained lineage links warehouse columns to warehouse columns. Dashboards and charts instead declare what they render in inputFields, and searchAcrossLineage does not walk that. So the graph knows orders_daily.buyer_id is affected, and separately knows the dashboard renders orders_daily.buyer_id, but never joins the two. blastradius bridges that last hop, which is what turns "some table changed" into "this named dashboard goes blank".
  4. Writes back. Impacted assets get tagged in DataHub, and the pull request is linked to the dataset, so the next person to open that asset sees it was assessed and why.

Install

python3 -m venv .venv && ./.venv/bin/pip install -e .
export DATAHUB_GMS_URL=http://localhost:8080
export DATAHUB_GMS_TOKEN=       # only needed if your instance requires auth

Try it against a local DataHub

datahub docker quickstart          # DataHub at http://localhost:9002
python examples/seed.py            # small warehouse with real column lineage
blastradius schema 'urn:li:dataset:(urn:li:dataPlatform:postgres,raw.orders,PROD)'
blastradius scan   'urn:li:dataset:(urn:li:dataPlatform:postgres,raw.orders,PROD)' --drop user_id

Use it in CI

blastradius scan "$DATASET_URN" \
  --drop user_id \
  --out impact.md \
  --write-back \
  --pr-url "$PR_URL" \
  --fail-on-breaking

--out writes a markdown report sized for a PR comment. --fail-on-breaking exits non-zero so the merge is blocked. --write-back tags the impacted assets in DataHub and attaches the pull request to the dataset.

Use it from an agent

The CLI answers someone who already knows which column they are touching. An agent reviewing a pull request starts from a diff instead, so the same analysis is exposed over MCP:

pip install -e ".[mcp]"
blastradius-mcp                      # stdio, point any MCP client at it
tool reads writes
list_columns dataset schema
scan_schema_change column lineage, owners
record_impact column lineage, owners tags and PR link in DataHub

Reading and writing are separate tools on purpose. An agent can call scan_schema_change as often as it likes while it works out what the diff touches, and has to reach for record_impact deliberately to change the catalogue. Folding the write into a flag on the read would put the destructive path one forgotten argument away.

record_impact re-runs the scan rather than accepting impacts back from the caller, so nothing can write a claim into DataHub that no lineage walk supported.

The MCP server is an optional extra. The core keeps its four dependencies.

Commands

command what it does
blastradius schema <urn> list the columns DataHub knows for a dataset
blastradius scan <urn> trace what a change breaks downstream

Change flags, repeatable and combinable:

flag meaning severity if a downstream reads it
--drop col column is being removed breaking
--rename old:new column is being renamed breaking
--retype col:old:new column type is changing risky

A retype is risky rather than breaking because the reference still resolves by name; it fails later, in the result, instead of immediately.

How severity is decided

evidence severity
downstream reads the column, and it is dropped or renamed breaking
downstream reads the column, and it is retyped risky
downstream of the table, no column-level link recorded info

That last row matters. Missing fine-grained lineage is common, so an asset landing in info means "not proven", not "safe". The report says so rather than implying the change is clear.

Requirements

Python 3.10+, and a DataHub instance you can reach. Four dependencies: acryl-datahub, typer, rich, and pytest for development.

Tests

./.venv/bin/pip install -e ".[dev]"
./.venv/bin/pytest

Upstream

Building this turned up two things worth sending back to DataHub:

  • datahub#18791, searchAcrossLineage does not join column lineage to dashboard inputFields, which is the gap _bridge_bi_input_fields() exists to close.
  • datahub#18792, a docs change covering why properties needs a per-type alias when a query spans entity types.

License

Apache 2.0.

About

Pre-merge impact analysis for schema changes, using DataHub column-level lineage. Finds which downstream tables, dashboards and ML assets a column change breaks, and writes the verdict back into DataHub.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages