Stemma: Induced Decision Regions Reveal LLM Provenance

Keyu Zhang, Vadim Safronov, and Andrew Martin

This repository contains the official implementation of Stemma.

Stemma is a black-box fingerprinting method for LLM provenance testing. It maps open-ended outputs into a finite decision space instantiated by multiple-choice options and measures whether a suspect model inherits the source's induced decision regions. Probes are selected for stability, robustness, and specificity.

Overview of the Stemma workflow.

Method

1. Calibration — pick a prompt template per model for reliable label extraction.
2. Construction — select probes whose source decisions are stable across option permutations, supported by large logit margins, and specific to the source, favouring non-gold decisions rarely reproduced by unrelated background models.
3. Verification — query each suspect using the selected source probes and estimate the inheritance of the source's decision regions.

Orchestration: stemma/method/pipeline.py.

Install

Requires Python ≥ 3.10, a CUDA toolkit visible at install time (for gptqmodel), and a GPU suitable for the models in the benchmark config (we used an NVIDIA L40S, 48GB, in the paper).

bash scripts/install.sh

Authentication

Models and datasets are loaded from Hugging Face. Set a read token before running:

export HF_TOKEN=hf_...

Create a token at https://huggingface.co/settings/tokens and accept access terms for any gated models in the config.

Quick start

# Default: stemma_base + benchmark_main
bash scripts/run.sh

# Effectiveness benchmark
BENCHMARK_CONFIG=configs/benchmark_main.yaml bash scripts/run.sh

# Robustness benchmark (system prompts, RAG, sampling, …)
BENCHMARK_CONFIG=configs/benchmark_deploy.yaml bash scripts/run.sh

# Direct invocation
python -m stemma run \
  --config configs/stemma_base.yaml \
  --benchmark-config configs/benchmark_main.yaml \
  --output-dir outputs

Useful flags: --models a,b,c restricts sources/suspects; --seed N overrides the config seed.

Ablations and alternate settings live under configs/ (selection/, size/, dataset/, background/).

Results

Each run writes <output-dir>/metrics.csv: a source × suspect matrix of estimated decision-region inheritance scores in [0, 1]. Paper tables report detection metrics (AUC, pAUC@5%FPR, TPR@1%FPR, d′) over that matrix.

source_model,suspect_a,suspect_b,...
source_1,0.812345,0.104200,...
source_2,0.093100,0.774900,...

Headline numbers from the paper (full runs need substantial GPU time):

Setting	Pairs	AUC	TPR @ 1% FPR	Matrix
Effectiveness (56 checkpoints)	770	0.967	87.8%	results/main/metrics.csv
Robustness (91 instances)	1,260	0.995	93.5%	results/deploy/metrics.csv

Shipped matrices live under results/ (main, deploy, raw, plus ablations such as bg_*, random_*, dataset short names, and size_*).

Reproduce detection metrics from a shipped matrix:

python -m analysis.metrics_eval --metrics results/main/metrics.csv
python -m analysis.metrics_eval --metrics results/deploy/metrics.csv
# or: python -m analysis.metrics_eval --metrics-dir results/

Citation

@misc{zhang2026stemma,
  title         = {Stemma: Induced Decision Regions Reveal {LLM} Provenance},
  author        = {Zhang, Keyu and Safronov, Vadim and Martin, Andrew},
  year          = {2026},
  eprint        = {2607.25880},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CR},
  url           = {https://arxiv.org/abs/2607.25880}
}

License

MIT. See LICENSE.