SignalGraph

Produces deterministic evidence for conversion delivery and marketing-journey health from sanitized aggregates, then attaches DataHub owner and lineage context when configured.

Built for DataHub's Agent Hackathon, Agents that do real work category: read DataHub for context, evaluate a deterministic rule, write one explicit action back. See Project status for what is verified today and docs/hackathon-demo.md for the judge path.

The problem

Someone clicks your ad. The click carries a gclid. Days later that person books, and the booking has to travel back to the ad platform as an offline conversion so the platform learns which clicks turned into money.

When that last hop fails, nothing errors.

The booking is still in your database. The dashboard still renders. The ad platform simply never hears about the sale, so it keeps bidding as if those clicks were worthless. There is no alert, because from every individual system's point of view nothing went wrong. Teams usually find out weeks later, by noticing that reported revenue drifted away from real revenue.

Two questions follow, and neither is quick to answer during the incident. How much money is affected? And who owns the thing that broke? The counts live in a warehouse. The ownership lives in a catalog. Nobody is joining them at 9pm.

How it works

sanitized aggregate evidence (operator JSON, or local sanitized CSV rows)
  -> one deterministic rule (conversion delivery, marketing-journey health,
     signal admissibility, or delivery-window pricing)
  -> DataHub readback for owner and lineage (skipped only with --offline)
  -> stable JSON stdout + Markdown evidence file
  -> optional, explicit, idempotent DataHub incident (--apply-incident)

Four independent evaluators share this shape; each is documented in its own section below and in docs/architecture.md. None of them share state, open a connection you did not ask for, or read a clock. DataHub is asked three read-only questions through its official MCP server — search, then get_entities, then get_lineage — never mutated except through the two opt-in write actions below.

Use cases

Situation	Evaluator	Section
An offline conversion may not have reached the ad platform	revenue_signal_integrity	What it tells you
A marketing signal's trustworthiness is unknown before any rule touches it	signal admissibility gate	Deciding whether a signal is usable at all
A whole delivery window needs a price, a likely failure stage, and an owner to route to	conversion_delivery	Pricing the whole delivery window
Click-context, consent, and delivery health across a full marketing journey needs a read	marketing_health	Marketing journey health
Roadmap, not built: the same aggregate-contract shape applied to a ClickHouse web-events warehouse, a BigQuery-hosted CRM, HubSpot, or Typeform	—	docs/roadmap.md

What it tells you

SignalGraph compares how many conversions were eligible at the source against how many the destination actually received, succeeded, and left pending. It sorts that comparison into one of five outcomes, then reads DataHub for the ownership and lineage of the asset involved.

Here is committed synthetic output, from examples/broken-report.md:

## `missing_upload`

- Failure code: `MISSING_DELIVERY`
- Severity: `medium`
- Affected conversions: `1`
- Declared exposure estimate: `120.00 EUR` (not measured loss)
- Next action: Reconcile missing offline conversions and retry destination delivery.

The five outcomes:

Outcome	Severity	What it means
NO_RECENT_UPLOAD	critical	Supplied counts show no received, successful, or pending conversions. This does not prove the upload job is down.
HIGH_MISMATCH	high	Supplied counts show more than half of eligible conversions as unsuccessful.
MISSING_DELIVERY	medium	Supplied counts show some eligible conversions did not arrive or succeed.
PENDING_OVER_TOLERANCE	low	Conversions are sitting in the destination queue past the tolerance you set.
HEALTHY	none	The counts line up.

When DataHub is configured, the same report also carries the destination asset's owner and how many assets sit downstream of it, so the finding arrives with a name attached instead of a dataset URN you then have to look up.

Marketing journey health

The DataHub-first extension checks aggregate health across the path from landing event to mature conversion record: provider click-context capture, storage context, UTM/click-context mismatches, consent state, unauthorized tag firing, context propagation, and destination delivery. It accepts counts only; it never accepts raw gclid, fbclid, cookie values, customer identifiers, or probabilistic attribution.

An upstream data-lake or warehouse job is expected to emit the sanitized contract. SignalGraph evaluates it, then uses DataHub to answer the operational questions: which asset represents the evidence, who owns it, and what is downstream of it. DataHub is context, not the event source of truth.

python3 -m signalgraph.marketing_stub \
  --source-dir demo/stubs/marketing_health_alert \
  --output /tmp/marketing-health.json

python3 -m signalgraph.cli \
  --offline \
  --marketing-health /tmp/marketing-health.json \
  --markdown-out /tmp/marketing-health.md \
  --json-out /tmp/marketing-health.json

Denied consent is reported as context, not automatically treated as a defect. Observed tags firing without required consent are critical. UTM/click-ID mismatches are surfaced as observe until a source-specific policy says they are an incident. Aggregate counts remain explicitly unverified.

What it does not do

This matters more than the feature list, so it is not buried at the bottom.

An upstream producer supplies aggregate evidence. SignalGraph reads a small JSON file emitted by a local fixture or a future data-lake/warehouse adapter. It does not query your warehouse, browser, Google Ads, Meta, or CRM today. Producing trustworthy aggregates from those systems remains outside this repository.

You supply the value assumption. declared_exposure_estimate is conversion_value multiplied by the number of affected conversions, and conversion_value is a field in the file you wrote. It is not measured lost revenue, and nothing looks up a real price.

The evidence is labeled. The finding records counts as operator_supplied, conversion_value as operator_declared_per_conversion, and independent_verification as false. DataHub context is separate metadata readback; it does not verify the conversion counts.

It is read-only by default. The MCP server's mutation, user, and document paths remain switched off. Two independent, opt-in flags are the only writes: --apply-incident may raise one idempotent operational incident on the validated destination asset, and --apply-assertion may upsert one DataHub Custom Assertion and report the current finding as its result. Neither retries, repairs, replays, or changes conversion data. It reports by default; you must opt into either audited action.

There is no model in it. No LLM, no autonomy, no planning, no tool selection. One rule, five branches, the same input always producing the same bytes out.

It refuses rather than guesses. If DataHub returns an ambiguous search result, a malformed entity, an asset with no owner, or a missing lineage edge, SignalGraph exits non-zero. It will not fall back to a cached answer or a fixture and present that as evidence.

The committed examples in examples/ are synthetic offline output, marked datahub_evidence: false. They are proof that the rule behaves, not proof of a live DataHub run.

Try it

Python 3.11 or newer. No dependencies to install, nothing to configure, no DataHub needed:

git clone https://github.com/Atroci/signalgraph.git
cd signalgraph
./demo/run_demo.sh

That runs a healthy fixture and a broken one, and prints the paths to the JSON and Markdown evidence it wrote. The broken one is the MISSING_DELIVERY report shown above.

You can also read the output without running anything, at https://atroci.github.io/signalgraph/.

The file you have to write

One fixture is one comparison. This is the whole input format:

{
  "fixture_id": "missing_upload",
  "source": { "eligible_count": 2 },
  "destination": {
    "received_count": 1,
    "success_count": 1,
    "pending_count": 0
  },
  "conversion_value": "120.00",
  "currency": "EUR",
  "lineage": {
    "source_asset": "urn:li:dataset:(urn:li:dataPlatform:postgres,signalgraph.booking_events,PROD)",
    "queue_asset": "urn:li:dataset:(urn:li:dataPlatform:kafka,signalgraph.qualified_conversion_queue,PROD)",
    "destination_asset": "urn:li:dataset:(urn:li:dataPlatform:postgres,signalgraph.offline_conversions,PROD)"
  }
}

Two eligible conversions, one of which arrived and succeeded. The arithmetic shows one affected conversion and a declared exposure estimate of 120.00 EUR; it does not prove a loss. The three URNs tell SignalGraph which DataHub assets to ask about.

To derive counts from local sanitized rows instead of typing them into the fixture, run the CSV path in docs/demo-script.md.

Adding DataHub context

With DataHub configured, SignalGraph starts the official DataHub MCP Server and asks it three questions, in order: search, then get_entities, then get_lineage. It reads ownership and lineage for the three assets your fixture named, and attaches the destination owner and downstream count to the finding.

You need uv on your PATH, which provides uvx, and a reachable DataHub instance holding the demo assets.

export DATAHUB_GMS_URL=http://localhost:8080
# Optional, only if your instance needs it:
# export DATAHUB_TOKEN='<your-token>'
./demo/run_mcp_demo.sh

The generic script runs one already-seeded fixture. For the complete operator proof, ./demo/run_vps_mcp_demo.sh reseeds two conversion fixtures, one CSV-derived reconciliation fixture, and one marketing-health aggregate fixture on an existing private VPS DataHub, opens an SSH tunnel, and verifies the reports. The full procedure is in docs/datahub.md. Direct GraphQL is still supported as a legacy mode, and it is never used as a fallback when MCP fails. To demonstrate the explicit incident action after validated MCP context:

./demo/run_vps_mcp_demo.sh --apply-incident

To demonstrate the DataHub Custom Assertion action instead — one upsertCustomAssertion plus one reportAssertionResult, run for every finding rather than only a failing one — use the CLI directly, since the VPS runner does not forward this flag yet:

python3 -m signalgraph.cli --mcp --apply-assertion \
  --markdown-out /tmp/evidence.md --json-out /tmp/evidence.json \
  demo/fixtures/missing_upload.json

Both flags are optional, apply only to conversion fixtures, and are not used by the default judge path. Marketing-health reports remain read-only in this slice. See docs/datahub.md for the field-by-field detail.

Deciding whether a signal is usable at all

--signal is a separate, optional gate that answers a different question: before any rule reads a marketing signal, is that signal admissible as evidence?

python3 -m signalgraph.cli \
  --offline \
  --markdown-out /tmp/evidence.md \
  --signal demo/fixtures/marketing_signal_admit.json

It returns admit, quarantine, or deny with a reason code. It denies raw and derived identifiers, personal and health data, identity resolution, CRM synchronisation, attribution modelling, and probabilistic scores. It collects nothing, resolves no identity, keeps no state between runs, and opens no connection. See docs/marketing-signal-contract.md.

Pricing the whole delivery window

The rule above answers one comparison. The commercial question is bigger: over a whole delivery window, how many conversions did not make it, what were they worth, and where did they stop?

signalgraph.conversion_delivery answers that from one sanitized aggregate document:

warehouse / CRM / booking platform
  -> eligible conversions
  -> aligned time window
  -> upload attempts
  -> normalized destination diagnostics
  -> pending / rejected / duplicated / missing
  -> affected count and affected value
  -> likely failure location
  -> DataHub owner
  -> recommendation-only alert / incident / optional retry actions

The committed golden incident, demo/fixtures/conversion_delivery_golden.json, reads like this. Ten conversions were eligible and worth 1000.00 EUR. Nine were attempted, eight were normalized by the destination, four were accepted, and three came back successful in both the CRM and the booking platform. An excerpt of what comes back, alongside a full evidence block and recommended actions:

{
  "affected_count": 7,
  "affected_value": "700.00",
  "currency": "EUR",
  "failure_location": "upload_job",
  "owner_routing": { "label": "growth_operations" },
  "reason_codes": ["duplicated", "missing", "pending", "rejected"],
  "state": "alert"
}

Seven conversions with an affected value of 700.00 EUR are reported, the first place to look is the upload job, and the finding routes to growth_operations. The recommended_actions list contains alert, open incident, and retry suggestions; every one carries executed: false and requires_approval: true. Nothing is sent, opened, or enqueued.

The same boundaries apply here as everywhere else in this repository. It consumes aggregate evidence you produce; it is not a warehouse, CRM, booking platform, GA4, Ads, destination connector, CDP, attribution warehouse, or identity system. Aggregate CRM and booking outcomes are allowed as evidence; rows, references, and identifiers are refused outright. DataHub stays read-only context for lineage, contracts, owners, and diagnostics. There are no live destination adapters and nothing runs on a schedule.

One flag runs it, offline, against the committed golden fixture:

python3 -m signalgraph.cli \
  --offline \
  --conversion-delivery demo/fixtures/conversion_delivery_golden.json \
  --markdown-out /tmp/evidence.md

It is fail-closed. Only a healthy or alert evaluation renders a report; contradictory or excluded evidence exits non-zero with its reason code rather than publishing a partial finding.

Full explanation, the step-to-evidence table, the privacy boundaries, and the MVP-versus-later split are in docs/conversion-delivery-assurance.md.

Project status

SignalGraph was built for the DataHub Agent Hackathon and is a working prototype rather than a product. Two deterministic evidence paths, four synthetic demo assets, no packaging for production use.

The rule engine, the offline path, the MCP adapter, the signal gate, the conversion delivery evaluator, the CSV reconciliation path, the marketing health evaluator, the lake-shaped marketing stub, and the explicit incident action are built and covered by 159 tests. Legacy direct-GraphQL readback passed against DataHub 1.7.0 on 2026-08-05. The fresh MCP path passed against DataHub 1.7.0 on 2026-08-06 through a private SSH tunnel: the two conversion fixtures, CSV-derived reconciliation, and marketing-health fixture returned validated metadata, one owner, one downstream asset, and their expected deterministic findings. The live JSON and Markdown remain outside Git; committed examples remain synthetic offline output. Scope and acceptance gates are in MVP.md; the specifications are in specs/001-datahub-presentation-readiness/spec.md, specs/003-conversion-delivery-assurance/spec.md, and specs/005-marketing-journey-health/spec.md.

Do not commit tokens, live owner data, raw DataHub responses, or live reports to this repository.

Roadmap

Everything above is built, tested, and passed a live DataHub gate. The longer-term idea is bigger than the MVP: a DataHub-grounded signal-detection layer that reads sanitized, aggregate-only signals from wherever a marketing or growth team already has them — ad platforms, email, organic/search, website and backend events, a ClickHouse web-events warehouse, a BigQuery-hosted CRM, HubSpot, Typeform — and attaches DataHub owner and lineage context to whatever it finds.

None of that is built. docs/roadmap.md states the long-term category, the candidate sources, and exactly what has to happen before a candidate stops being a roadmap bullet: its own feature spec, its own sanitized-aggregate contract, tests, and the same live-DataHub verification gate every shipped feature in this repository already passed. The constitution's "Vision and roadmap (not built)" section is the binding version of this boundary.

Checks

python3 -m unittest discover -v
./demo/judge_demo.sh

Reviewers need no credentials and no DataHub for the offline gate. demo/judge_demo.sh checks the suite, syntax, byte-stable offline artifacts, and the CSV-derived path. demo/run_vps_mcp_demo.sh is the separate operator gate for live MCP context and optional incident writeback.

Code map

signalgraph.contracts defines the fixture and finding shapes. signalgraph.rules holds the conversion integrity rule. signalgraph.mcp_context performs read-only MCP normalization. signalgraph.datahub performs the legacy GraphQL readback and the explicit, opt-in incident and Custom Assertion actions. signalgraph.signal_contract gates signal admissibility. signalgraph.conversion_delivery evaluates one delivery window over sanitized aggregates. signalgraph.marketing_health evaluates sanitized marketing journey-health aggregates. docs/architecture.md describes the evidence boundary.

The repository root also carries a conformant Agent Plugins v1.0.0 package (plugin.json plus skills/signalgraph-evidence-check/), so a compatible agent client can discover this CLI as a portable skill. See docs/agent-plugin.md for the conformance checklist; tests/test_agent_plugin.py enforces it.

License

Apache 2.0. See LICENSE.