Skip to content

results: add verbalizer-axis eval for domain-aware-contrastive Gemma-4-E2B NLA #22

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
SolshineCode merged 1 commit into from
Jul 27, 2026
Merged

results: add verbalizer-axis eval for domain-aware-contrastive Gemma-4-E2B NLA #22

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions results/local_gemma_e2b/domainaware_nla_verbalizer_axes.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
{
"model": "google/gemma-4-E2B",
"nla_variant": "domain-aware contrastive (InfoNCE over hard-negative activations) NLA, rank-8 LoRA",
"injection": "residual L23 last-token activation injected at the concept token, L2-normalized to sqrt(1536)",
"eval": "de-circularized held-out set, 5 academic domains (sciq/PubMedQA/hendrycks_math/lex_glue/imdb)",
"verbalizer_axes": {
"minimal_pair_discrimination": {"raw": 0.491, "null": 0.505, "margin": -0.014, "n": 40, "chance": 0.5},
"doc_retrieval": {"raw": 0.050, "null": 0.017, "margin": 0.033, "n": 60},
"prior_deviation": {"raw": 0.037, "null": 0.000, "margin": 0.037, "n": 60}
},
"baseline_v0_1_minimal_pair": 0.531,
"note": "Domain-aware contrastive training improves LIKELIHOOD-based domain discrimination but the GENERATION-based minimal-pair AUC stays at chance (~0.49, marginally below the v0.1 baseline ~0.531). I.e. the verbalizer routes domain but its generations do not carry distinguishing within-domain content. Reported as an honest negative on the generation axes."

@gemini-code-assist gemini-code-assist Bot Jun 19, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The pull request description highlights Routing accuracy 0.676 vs within-domain content 0.456, but these specific numbers are not captured in the JSON results. To ensure complete documentation and data permanence, please consider integrating these metrics into the note field or as separate structured fields.

Suggested change
"note": "Domain-aware contrastive training improves LIKELIHOOD-based domain discrimination but the GENERATION-based minimal-pair AUC stays at chance (~0.49, marginally below the v0.1 baseline ~0.531). I.e. the verbalizer routes domain but its generations do not carry distinguishing within-domain content. Reported as an honest negative on the generation axes."
"note": "Domain-aware contrastive training improves LIKELIHOOD-based domain discrimination (routing accuracy: 0.676) but the GENERATION-based minimal-pair AUC stays at chance (~0.49, marginally below the v0.1 baseline ~0.531, within-domain content: 0.456). I.e. the verbalizer routes domain but its generations do not carry distinguishing within-domain content. Reported as an honest negative on the generation axes."

Comment on lines +6 to +12

@gemini-code-assist gemini-code-assist Bot Jun 19, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

There is a discrepancy between the metrics reported in the PR description and the values recorded in this JSON file. The PR description states: Routing accuracy of 0.676 and within-domain content of 0.456. However, these numbers do not appear in this results file (e.g., minimal_pair_discrimination raw is 0.491). Please update the JSON file to include these metrics or clarify the discrepancy.

}