detector metrics never load: analyze/detector_metrics.py reads detectors_eval, shipped dir is detectors-eval

[immu4989]

Steps to reproduce

garak/analyze/detector_metrics.py loads the shipped detector metrics from a directory that does not exist. The file lives in garak/data/detectors-eval/ (hyphen), but DetectorMetrics._load_metrics looks in detectors_eval (underscore):

# garak/analyze/detector_metrics.py:22
metrics_file = (
    Path(data_path) / "detectors_eval" / "detector_metrics_summary.json"
)

garak/_plugins.py:46 reads the same file from the correct hyphenated path, so the repo spells it both ways.

No scan or config needed, this runs in a couple of seconds against a source checkout:

from garak.analyze.detector_metrics import get_detector_metrics
from garak._plugins import PluginCache
import os

m = get_detector_metrics()
print("metrics_loaded:", m.metrics_loaded)
print("dan.DAN Se/Sp :", m.get_detector_se_sp("dan.DAN"))

print("_plugins path exists:", os.path.exists(PluginCache._detector_metrics_filename))
print("_plugins dan.DAN    :", PluginCache._get_detector_metrics()["dan.DAN"]["metrics"])

Output on a clean checkout of main:

metrics_loaded: False
dan.DAN Se/Sp : (1.0, 1.0)
_plugins path exists: True
_plugins dan.DAN    : {'hit_f1': 0.888888888888889, 'hit_sensitivity': 1.0, 'hit_specificity': 0.75}

Expected behavior

DetectorMetrics loads detector_metrics_summary.json and returns the measured sensitivity and specificity, so dan.DAN reports Se 1.0 and Sp 0.75.

Current behavior

The open fails with FileNotFoundError, which is caught and logged at debug level, metrics_loaded stays False, and get_detector_se_sp returns the (1.0, 1.0) fallback for every detector. The same values come back for a detector name that does not exist, so there is no visible difference between "no metrics" and "perfect detector".

The consumer is ci_calculator.calculate_ci_from_report, which pulls Se/Sp at line 163 and passes them into the interval computation. Confidence intervals produced through analyze/rebuild_cis.py are therefore computed as if every detector were perfect, rather than using the shipped measurements. Since the failure is only logged at debug level, nothing surfaces during a normal run.

The existing tests in tests/analyze/test_detector_metrics.py set metrics_loaded = True by hand or assert graceful degradation when it is False, so none of them exercise loading the shipped file from its real path and the suite stays green.

garak version

0b51f87a (source checkout of main)

Additional Information

1. macOS (Darwin 25.5.0), also path independent since it is a string mismatch
2. Python 3.12
3. Direct repository checkout with git, editable install
4. No scan required, the snippet above reproduces it directly
5. No special config

Happy to send a PR: correcting the path in detector_metrics.py to match the shipped directory, plus a test asserting the real file loads and that dan.DAN returns its measured Se/Sp rather than the fallback, so a future rename cannot silently disable it again.

[immu4989]

Correcting my own diagnosis above: the directory name is only half of this, and fixing the name alone does not resolve it.

garak.data.path is a LocalDataPath, which resolves a resource across the data dir and then the package dir on each /. Wrapping it in Path(...) discards that behaviour and pins the lookup to _config.transient.data_dir / "data", where nothing ships. So both of these still fail:

Path(data_path) / "detectors_eval" / "detector_metrics_summary.json"   # as shipped
Path(data_path) / "detectors-eval" / "detector_metrics_summary.json"   # name corrected only

Dropping the Path() wrapper is what reaches the bundled copy:

data_path / "detectors-eval" / "detector_metrics_summary.json"
# -> <package>/garak/data/detectors-eval/detector_metrics_summary.json

One consequence for the fix: LocalDataPath raises GarakException when a resource is genuinely absent, rather than yielding a path that later raises FileNotFoundError. The current handler catches FileNotFoundError, JSONDecodeError and OSError, so GarakException needs handling too to preserve the graceful fallback on installs without the data file.