EDGE — Error Dependency Graph-Guided Multi-Error Attribution

Code, released error-dependency graphs, and prompts for EDGE, evaluated on the TRAIL and MAST benchmarks.

Jun Hou, Priya Pitre, Yi Fang, Xuan Wang. EDGE: Error Dependency Graph-Guided Multi-Error Attribution in Multi-Agent LLM Systems. EMNLP 2026.

This repository ships code, graphs, and prompts only — no benchmark traces, no model prediction outputs, and no model weights. All judge and detector prompts are inline string constants in the scripts listed below (there are no separate prompt files).

Installation

git clone https://github.com/JuneHou/EDGE.git
cd EDGE
pip install -r requirements.txt

Local serving (the *_vllm.py runners) additionally needs vllm and transformers — see the commented lines in requirements.txt. API-backed runs read credentials from the environment (OPENAI_API_KEY, GEMINI_API_KEY, OPENROUTER_API_KEY).

Pipeline map

Each benchmark follows the same three-stage pipeline:

graph/        error-dependency-graph construction (CAPRI: Suppes screen -> AIC/BIC prune -> bootstrap/shuffle -> hierarchy)
intervention/ counterfactual validation (patch -> rerun -> judge A/B -> aggregate Δ per edge)
eval/         detection/attribution (baseline, +CG static-graph, +EDGE two-stage injection) + scorers
graphs/       released graph artifacts (Suppes graph, AIC-pruned graph, intervention-validated effect edges)

TRAIL additionally ships trail/span_level_parser.py and trail/trail_io.py (shared helper modules imported by several eval/ and intervention/ scripts — not separately listed above because they are plumbing, not pipeline stages). MAST additionally ships mast/taxonomy/ (13-category taxonomy definitions + few-shot examples, loaded by all MAST eval scripts at import time) and mast/annotation/ (step-onset annotation pipeline).

Pipeline model IDs

Stage	Model
Patch generation	GPT-4o (both benchmarks)
Judge A (resolved?)	GPT-4o (both benchmarks)
Judge B (effect labeling)	GPT-4o (both benchmarks)
Counterfactual rerun — TRAIL	o3-mini (live replay)
Counterfactual rerun — MAST	GPT-4o (simulated continuation)
MAST step-onset annotation	GPT-4o (two-pass)

Thresholds

Symbol	Value	Meaning
τ (TRAIL, main table)	0.35	+EDGE / +CG correlation threshold (Suppes geomean ∪ causal)
τ (MAST, main table)	0.50	--corr_threshold for +EDGE / +CG on MAST
τ_GI	0.10	--propagation_threshold — min boosted score to trigger Pass 2 (two-stage injection)
Δ-cutoff (intervention validation)	0.15	--threshold in intervention/run_pipeline.py / effect_aggregator.py

Prompts

All judge and detector prompts are inline string constants — no separate prompt files ship in this release. Locations:

Prompt	File
Patch generation	{trail,mast}/intervention/patch_generator.py
Judge A (resolved?)	{trail,mast}/intervention/judge_a_resolved.py
Judge B (effect)	{trail,mast}/intervention/judge_b_effect.py
Baseline detector	trail/eval/run_eval.py, trail/eval/run_eval_vllm.py, mast/eval/run_eval_yesno*.py
+CG (one-pass, in-prompt graph)	trail/eval/run_eval_with_graph*.py, mast/eval/full_run_eval_with_graph*.py
+EDGE (two-pass injection)	trail/eval/run_eval_graph_inject*.py, mast/eval/full_run_eval_graph_inject*.py
MAST step annotation	mast/annotation/prompt.py
MAST taxonomy text injected into all MAST eval prompts	mast/taxonomy/definitions.txt, mast/taxonomy/examples.txt

Layout note — path repairs for this release

The source repositories keep graph/, intervention/ (causal/patch/ in TRAIL, causal_graph/causal_valid/ in MAST), and eval/ in different parts of two separate trees (trail-benchmark/ and MAST/), with a couple of shared helper modules living in a sibling data directory. This release flattens everything per-benchmark into {trail,mast}/{graph,intervention,eval,graphs}/. A small number of repo-relative path constants (import search paths, default graph-file locations, default taxonomy-directory locations) pointed at the old tree layout and would not resolve here; each was adjusted to the minimum extent needed to resolve inside this layout, and every change is marked # path adjusted for release layout at the call site. No prompt text and no pipeline logic was changed. Two absolute-path leaks inside shipped JSON graph artifacts (trail/graphs/suppes_graph.json, trail/graphs/capri_graph.json — params.in_path / params.onsets_path / params.suppes_path) were rewritten to relative paths for the same reason.

Convention: run all TRAIL commands with cwd = trail/, all MAST graph/ and eval/ commands with cwd = mast/, and MAST intervention/ commands with cwd = mast/intervention/ (this mirrors each benchmark's own source convention as closely as possible; see the repro commands below).

The eval/run_threshold_sweep*.sh drivers and run_eval_with_graph_vllm.py originally dispatched to two additional API backends specific to our compute environment. Those branches were removed; only the vllm and litellm backends remain, matching the runners shipped here.

Expected data layout (bring your own, from the public releases)

Download TRAIL and MAST from their public releases and place the derived files at:

trail/data/onsets_gaia_swe_full.jsonl      # TRAIL onset jsonl (present/onset per trace); see graph/preprocess/ for how this is derived from raw traces
trail/data/GAIA/, trail/processed_annotations_gaia/   # for intervention/ (patch generation needs raw traces + annotations)

mast/data/annotation/annotation_ag2_filtered.jsonl    # MAST AG2 annotation + onsets input; see graph/ag2_build_gt.py, graph/ag2_to_onsets.py

Repro commands per stage

TRAIL (cwd = trail/)

# 1. Graph construction (Suppes -> CAPRI AIC/BIC -> bootstrap -> shuffle -> hierarchy)
python graph/run_causal_from_trail_onsets.py \
    --onsets_path data/onsets_gaia_swe_full.jsonl --out_dir graph_outputs

# 2. Intervention / counterfactual validation (patch -> rerun -> judge A/B -> aggregate)
python intervention/run_pipeline.py \
    --trace_dir data/GAIA --annotations_dir processed_annotations_gaia \
    --causal_graph graphs/capri_graph.json \
    --patch_library intervention/patch_library.json \
    --out_dir outputs/interventions \
    --model openai/gpt-4o --rerun_model openai/o3-mini

# 3. Evaluation — baseline / +CG / +EDGE (main-table τ=0.35)
python eval/run_eval.py --model <model> --split GAIA_dedup
python eval/run_eval_with_graph_vllm.py --model <model> --split GAIA_dedup --corr_threshold 0.35
python eval/run_eval_graph_inject_vllm.py --model <model> --split GAIA_dedup --corr_threshold 0.35 --propagation_threshold 0.10

# 4. Score
python eval/calculate_scores.py --results_dir <output_dir>

MAST graph + eval (cwd = mast/)

# 1. Graph construction
python graph/run_causal_pipeline.py --input data/annotation/annotation_ag2_filtered.jsonl

# 2. Evaluation — baseline / +CG / +EDGE (main-table τ=0.50, τ_GI=0.10)
python eval/run_eval_yesno_api.py --model openai/gpt-4o
python eval/full_run_eval_with_graph_api.py --model openai/gpt-4o --corr_threshold 0.50
python eval/full_run_eval_graph_inject_api.py --model openai/gpt-4o --corr_threshold 0.50 --propagation_threshold 0.10

# 3. Score
python eval/calculate_scores_yesno.py --pred_dir <output_dir>

MAST intervention (cwd = mast/intervention/)

python run_pipeline.py \
    --input ../data/annotation/annotation_ag2_filtered.jsonl \
    --causal_graph ../graphs/capri_graph_aic.json \
    --model openai/gpt-4o

Note: the shipped MAST release graph is mast/graphs/capri_graph_aic.json (AIC criterion); pass it explicitly via --causal_graph as shown above — the scripts' own built-in default filename is capri_graph.json (no suffix).

Datasets

• TRAIL: https://huggingface.co/datasets/PatronusAI/TRAIL (see the TRAIL paper/repo for the GAIA/SWE-Bench trace format this pipeline consumes).
• MAST: see the MAST (Multi-Agent System Failure Taxonomy) paper/repo for the AG2 annotation format consumed by graph/ag2_build_gt.py / graph/ag2_to_onsets.py.

What is not included

• Benchmark traces and annotations — download from the public TRAIL and MAST releases and place them as described in Expected data layout above.
• Model predictions, metrics, and result tables — every number reported in the paper is regenerated by running the pipeline.
• Model weights or caches.
• Runner variants tied to our institutional API endpoints.

Note that trail/graph/CAPRI/ and mast/graph/CAPRI/ are intentional per-benchmark copies of the same six-step procedure, kept separate so each benchmark's pipeline runs standalone.

License

Released under the MIT License — see LICENSE. The taxonomy definitions under mast/taxonomy/ and the trail_definition / mast_definition fields in the patch libraries are derived from the TRAIL and MAST benchmark releases and remain subject to their original terms.

Citation

Machine-readable metadata is in CITATION.cff.

@inproceedings{hou2026edge,
  title     = {{EDGE}: Error Dependency Graph-Guided Multi-Error Attribution in Multi-Agent {LLM} Systems},
  author    = {Hou, Jun and Pitre, Priya and Fang, Yi and Wang, Xuan},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year      = {2026}
}