results: add verbalizer-axis eval for domain-aware-contrastive Gemma-4-E2B NLA - #22
Conversation
…a-4-E2B NLA minimal_pair_discrimination 0.491 (~chance, vs v0.1 ~0.531), doc_retrieval +0.033, prior_deviation +0.037. Domain-aware contrastive training lifts likelihood-based domain routing but not generation-based content discrimination. Honest negative on the generation axes.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds a new JSON file containing evaluation results for the google/gemma-4-E2B model with a domain-aware contrastive NLA variant. The reviewer suggested incorporating specific routing accuracy and within-domain content metrics mentioned in the PR description into the JSON's note field to ensure complete documentation.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| "prior_deviation": {"raw": 0.037, "null": 0.000, "margin": 0.037, "n": 60} | ||
| }, | ||
| "baseline_v0_1_minimal_pair": 0.531, | ||
| "note": "Domain-aware contrastive training improves LIKELIHOOD-based domain discrimination but the GENERATION-based minimal-pair AUC stays at chance (~0.49, marginally below the v0.1 baseline ~0.531). I.e. the verbalizer routes domain but its generations do not carry distinguishing within-domain content. Reported as an honest negative on the generation axes." |
There was a problem hiding this comment.
The pull request description highlights Routing accuracy 0.676 vs within-domain content 0.456, but these specific numbers are not captured in the JSON results. To ensure complete documentation and data permanence, please consider integrating these metrics into the note field or as separate structured fields.
| "note": "Domain-aware contrastive training improves LIKELIHOOD-based domain discrimination but the GENERATION-based minimal-pair AUC stays at chance (~0.49, marginally below the v0.1 baseline ~0.531). I.e. the verbalizer routes domain but its generations do not carry distinguishing within-domain content. Reported as an honest negative on the generation axes." | |
| "note": "Domain-aware contrastive training improves LIKELIHOOD-based domain discrimination (routing accuracy: 0.676) but the GENERATION-based minimal-pair AUC stays at chance (~0.49, marginally below the v0.1 baseline ~0.531, within-domain content: 0.456). I.e. the verbalizer routes domain but its generations do not carry distinguishing within-domain content. Reported as an honest negative on the generation axes." |
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request adds a JSON file containing evaluation results for the google/gemma-4-E2B model using a domain-aware contrastive NLA variant. The reviewer identified a discrepancy between the metrics mentioned in the pull request description and those recorded in the JSON file, requesting clarification or an update to ensure consistency.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| "verbalizer_axes": { | ||
| "minimal_pair_discrimination": {"raw": 0.491, "null": 0.505, "margin": -0.014, "n": 40, "chance": 0.5}, | ||
| "doc_retrieval": {"raw": 0.050, "null": 0.017, "margin": 0.033, "n": 60}, | ||
| "prior_deviation": {"raw": 0.037, "null": 0.000, "margin": 0.037, "n": 60} | ||
| }, | ||
| "baseline_v0_1_minimal_pair": 0.531, | ||
| "note": "Domain-aware contrastive training improves LIKELIHOOD-based domain discrimination but the GENERATION-based minimal-pair AUC stays at chance (~0.49, marginally below the v0.1 baseline ~0.531). I.e. the verbalizer routes domain but its generations do not carry distinguishing within-domain content. Reported as an honest negative on the generation axes." |
There was a problem hiding this comment.
There is a discrepancy between the metrics reported in the PR description and the values recorded in this JSON file. The PR description states: Routing accuracy of 0.676 and within-domain content of 0.456. However, these numbers do not appear in this results file (e.g., minimal_pair_discrimination raw is 0.491). Please update the JSON file to include these metrics or clarify the discrepancy.
Promote the prior frozen benchmark to v1 and this work to a formal v2.0.0 release, reconciling versioning, docs, and counts across the repo. v1 content and its published results are carried forward unchanged. Version & release infra - nla_eval/__init__.py -> 2.0.0; CITATION.cff version 2.0.0 + abstract (128 plans / 14 families). - New CHANGELOG.md (v1 + v2 entries) following the freeze-on-release policy. - VERSIONING.md rewritten around plain v1/v2 generation labels (keeps the freeze policy; maps the old v0.1->v1, v0.2->v2; clarifies that a checkpoint's own "v0.1" version names the NLA artifact, not the benchmark generation). - docs/ROADMAP_v0.2.md -> docs/ROADMAP_v2.md; reframed so the CTF family is "shipped in v2.0.0" and the multi-domain expansion + H1-H12 harness fixes are the tracked v2.x roadmap (not claimed in the tagged release). Catalog & naming - Family N (CTF Red/Blue) promoted from "v0.2 additive" to a first-class v2 family. Counts reconciled to 128 plans / 14 families (v2); families A-M (P001-P118) remain the frozen v1 catalog. Updated INDEX, plans/README, EVALUATIONS, CTF_RED_BLUE, ctf.py, the demo, and the hero-figure prompt. README & results - README simplified into a single-screen front page; v1 results table kept visible and labeled frozen. - RESULTS.md: v1 findings labeled frozen; new "v2 additions" section (CTF control-tier demo + the domain-aware honest-negative), with the control-vs-primary integrity caveat stated. - New results/README.md provenance map (which artifact belongs to which generation; v1 never overwritten). Honesty: the tagged v2.0.0 ships the new adversarial Red/Blue CTF family plus release maturation; it does not fabricate domain-coverage or fix-dependent numbers it has not produced. Note: PR #21 (versioning scaffold) and PR #22 (domain-aware result) are already merged to master, so this branch already contains them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MMuBSoDsAPwNSvev9nmvVr
1 participant
Summary
Test plan
🤖 Generated with Claude Code