dfbench — pluggable speech-deepfake detection benchmark

Evaluate audio deepfake / spoofing detectors on shuohann/df_eval and report EER / Accuracy / F1 / AUC per detector per config.

• Streams the dataset from the HF Hub — stratified 50/50 sampling, no local copy, no protocol.csv, decodes FLAC via soundfile (no torchcodec).
• Add a detector by dropping one file into dfbench/detectors/ — HF Hub, a cloned GitHub repo, or your own code. Templates provided.
• Every detector sees the same clips (loaded once) → results are comparable.
• Ships 10 built-in SOTA detectors (architectures vendored in dfbench/model/).

Layout

dfbench/
  registry.py            # REGISTRY + @register / register_factory
  base.py                # Detector ABC + score convention + pad util
  data.py                # HF df_eval streaming / stratified loader
  metrics.py             # EER / AUC / Accuracy / F1
  run.py                 # CLI runner -> reports/results.csv + .md
  detectors/             # ACTIVE detectors (auto-discovered)
    builtin.py             #   the 10 built-in models
  templates/             # reference templates (NOT auto-loaded — copy to detectors/)
    example_energy.py    #   custom / from-scratch model
    hf_model.py          #   HuggingFace audio classifier
    repo_model.py        #   model whose code lives in a downloaded repo
  model/          # vendored model architectures (self-contained, hardcoded configs)
scripts/
  setup_env.sh                    # build the conda env (WSL / RTX 50-series)
  fetch_and_upload_checkpoints.py # download checkpoints, mirror to HF
ENVIRONMENT.md           # reproducible-env notes + every gotcha we hit

Setup

bash scripts/setup_env.sh          # conda env 'df_eval' (see ENVIRONMENT.md)
conda activate df_eval

Checkpoints download automatically. If a model's weight isn't found locally, dfbench pulls it from the HF repo shuohann/df_eval_ck into the HF cache — no manual step needed.

• DFBENCH_CKPT_DIR — optional local dir checked first (weights named by model prefix: aasist*.pth, rawnet_2*.pth, …).
• DFBENCH_HF_REPO — override the fallback repo (default shuohann/df_eval_ck).
• scripts/sync_checkpoints_to_hf.py mirrors a local checkpoints/ up to the repo.

Run

python -m dfbench.run --list                                  # registered detectors
python -m dfbench.run --detectors aasist --configs asvspoof_2019 --per-class 50   # quick
python -m dfbench.run --detectors all --per-class 500         # full 5 configs x 1000

Flags: --configs (subset of asvspoof_2019 / fake_or_real / in_the_wild / librisevoc / sonar), --per-class (N bonafide + N spoof), --batch-size, --buffer-size (streaming shuffle; smaller = faster/less download), --device. Output → reports/results.csv and reports/results.md.

Run on a SLURM cluster (offline GPU node)

The GPU node usually has no internet, so prefetch on the login node (online), then submit the offline job. scripts/eval.sbatch is a ready template.

# 1) LOGIN node (has internet): download samples + all model assets into caches
export DFBENCH_CACHE=$PWD/caches HF_HOME=$PWD/caches/hf DFBENCH_CKPT_DIR=$PWD/caches/checkpoints
python -m dfbench.run --prefetch --data-dir $DFBENCH_CACHE/samples \
    --detectors aasist rawnet_2 rawgat_st wavlm_ecapa hubert_ecapa wav2vec2_ecapa xlsr_sls tcm_add \
    --per-class 500
# 2) submit the offline job (reads only the caches; HF_HUB_OFFLINE=1 inside)
mkdir -p sbatch_logs && sbatch scripts/eval.sbatch

--prefetch materializes the sampled audio to --data-dir (local FLAC) and downloads every weight / HF backbone / XLS-R front-end into the caches. The offline run then loads data from disk and models from cache — no network.

Score convention (read once)

Detector.score_batch returns cm_score where higher = bonafide (genuine), lower = spoof (fake) — same as ASVspoof. metrics.py treats bonafide as the positive class, so EER/AUC line up with the leaderboard. If your model's positive class is "fake", return the negated score.

Add a detector

Three copy-paste paths. Drop a file in dfbench/detectors/, fill it in, and uncomment its @register line — auto-discovery does the rest (--list shows it).

You have…	Start from	What to edit
a HuggingFace audio classifier	templates/hf_model.py	HF_REPO, the genuine-class index
a cloned GitHub / HF-snapshot repo (bundled code + weights)	templates/repo_model.py	REPO_DIR, the import + build call, output column
your own architecture / weights	templates/example_energy.py	load() + score_batch()

Minimal contract:

from ..base import Detector
from ..registry import register

@register("my_model")                # name used in --detectors
class MyModel(Detector):
    sample_rate = 16000              # wavs arrive resampled to this
    fix_length = 64600               # pad/trim to N samples; None = variable
    def load(self, device="cuda"):
        self.net = ...               # build + load weights, .to(device).eval()
    def score_batch(self, wavs):     # list[np.float32 @ sample_rate]
        return cm_scores             # np.ndarray, higher = bonafide

Heavy imports (torch/transformers) stay inside load() so an unfinished copy can never break the registry. A detector whose checkpoint/deps are missing is skipped (logged); the run continues with the rest.

Included models

aasist, rawnet_2, rawgat_st, wavlm_ecapa, hubert_ecapa, wav2vec2_ecapa, tcm_add, xlsr_sls, wav2vec2_aasist, nes2net_x. The 4 SSL models (xlsr_sls, tcm_add, wav2vec2_aasist, nes2net_x) also need fairseq; the shared xlsr2_300m.pt front-end auto-downloads on first use. wav2vec2_aasist and nes2net_x have no public weights (not in the repo), so they are skipped until you supply them.

Reproducibility

Environment build, pinned versions, and every dependency gotcha (Blackwell GPU → torch cu128, fairseq compiler, numpy/scipy conflict, datasets→torchcodec) are in ENVIRONMENT.md.