feat(query): make the hybrid semantic weight configurable (default unchanged) - #181
Merged
Merged
Conversation
…changed)
The 1.5x multiplier on the semantic arm's RRF contribution was a
hardcoded constant with a large, undocumented effect: with the standard
RRF constant K=60, it means semantic ranks 1 through 31 all outscore a
keyword-only rank-1 hit, so lexical matches the vector arm missed are
close to invisible. Exposed as HYBRID_SEMANTIC_WEIGHT so the balance can
be measured per deployment rather than assumed.
The default is deliberately NOT changed. A train/test experiment says
the current value is fine and that the apparent alternatives are noise:
sweep on a stratified 42-question sample (7 per category), validated
on a disjoint 42-question sample selected the same way
weight train R@5 test R@5 train R@1 test R@1
0.75 95.2% 88.1% 76.2% 76.2%
1.50 92.9% 88.1% 69.0% 81.0%
On train, 0.75 looked clearly better and would have been adopted by any
single-sample tuning. It did not replicate: R@5 identical on held-out
data and R@1 4.8pp worse. Every metric flips direction between the two
samples, which is the signature of sampling noise rather than signal —
at n=42 a single question moves a metric 2.4pp, and the observed gaps
are one to two questions.
Pre-registered rule (declared before the sweep ran): adopt only if the
winner beats the 1.5 baseline on the held-out sample at R@5 without
regressing R@1 by more than 2pp. It failed both conditions, so the
weight stays at 1.5.
Resolving this properly needs the full 500-question run; the knob is
what makes that measurement possible without a code change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xCqQo49d3CEbn6oEvb3Ru
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
The
1.5×multiplier on the semantic arm's RRF contribution was a hardcoded constant with a large, undocumented effect: with RRFK=60, it means semantic ranks 1–31 all outscore a keyword-only rank-1 hit — lexical matches the vector arm missed are close to invisible. NowHYBRID_SEMANTIC_WEIGHT, documented in the config reference.The default is deliberately unchanged, and that is the finding.
The experiment, and why the answer is "leave it alone"
Pre-registered before running: sweep on a stratified 42-question sample (7 per category), then validate the winner on a disjoint 42-question sample selected the same way. Adopt only if it beats the 1.5 baseline on held-out R@5 without regressing R@1 by more than 2pp.
On train, 0.75 looked clearly better across every k — the monotonic trend was tidy enough to be convincing, and any single-sample tuning would have adopted it. It did not replicate. On held-out data R@5 is identical and R@1 is 4.8pp worse.
Every metric flips direction between the two samples. That's the signature of sampling noise, not signal: at n=42 one question moves a metric 2.4pp, and the observed gaps are one to two questions.
So the pre-registered rule fails on both conditions, and the weight stays at 1.5.
Why this is still worth merging
bge-small, and a stronger embedder would likely shift it.Gates: type-check, lint, integration 24, http 43, retrieval-determinism 3, causal-graph 39 — all green.
🤖 Generated with Claude Code
https://claude.ai/code/session_011xCqQo49d3CEbn6oEvb3Ru