Skip to content

v0.5.0 — GroundTruthTarget

Choose a tag to compare

View all tags
@jascal jascal released this 18 May 05:09
v0.5.0
3a81174
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Install

Since sae-forge is not yet on PyPI, install from this tag:

pip install git+https://github.com/jascal/sae-forge.git@v0.5.0

Headline

add-gt-alignment-target — third built-in FaithfulnessTarget, motivated by jascal/sm-sae's production GroundTruthAlignment scorer. Family defaults (KL for LM hosts, cosine for whisper) are byte-identical to v0.4.0; GT-alignment is opt-in only via ForgePipeline(faithfulness=GroundTruthTarget(labels=L)).

The minor-version bump (vs. a patch) reflects the new scipy>=1.10 runtime dependency — technically a breaking change for callers with strict pins, even though the default surface is additive.

Added

saeforge.eval.targets.GroundTruthTarget (also re-exported as saeforge.eval.GroundTruthTarget). Scores forged residual-stream activations against an (N, M) binary label matrix via per-feature × per-label AUC — the right gate when your eval fixture carries known per-sample categories (synthetic mixtures, BERT-probe-derived datasets, concept-bottleneck suites).

  • Supported pool strategies: "mean" / "max" / "last".
  • Default hidden_extractor covers the six bundled LM-shape families (gpt2 / llama / gemma2 / qwen2 / qwen3 / qwen3_moe) via duck typing; Whisper / exotic forges supply their own.
  • AUC uses scipy.stats.rankdata(method="average") for bit-equal parity with sklearn.metrics.roc_auc_score on tie-heavy fixtures (no sklearn dep).

Demo: examples/forge_with_gt_alignment.py (mixture-of-gaussians, ~20s on CPU).

Quick example

import numpy as np
from saeforge import ForgePipeline
from saeforge.eval import GroundTruthTarget

labels = np.eye(3, dtype=np.float32)[cluster_ids]  # (N, 3) one-hot

pipeline = ForgePipeline(
    basis=basis,
    projector=projector,
    faithfulness=GroundTruthTarget(labels=labels, pool="mean"),
)
result = pipeline.run(output_dir)
print(result.faithfulness, result.faithfulness_target_name)

PRs in this release

  • #51 — openspec proposal
  • #52 — implementation (GroundTruthTarget + scipy AUC + tests + docs)
  • #53 — openspec archive; promote spec onto canonical faithfulness-target/spec.md
  • #54 — version bump + CHANGELOG promotion

What is NOT in this release

  • Additional scorers (Pearson, Spearman, monosemanticity, probe accuracy).
  • Multi-label-hierarchy comparison.
  • Whisper-encoder default hidden_extractor support.
  • requires_host=False protocol opt-out.
  • Family default routing for GT-alignment (still opt-in only).
  • CLI flag (--faithfulness-target gt_alignment). Label matrices have no reasonable CLI representation.
Assets 2
Loading