Three measurements in this repo contradict the material. We kept all three. #27
Replies: 2 comments 2 replies
The reranker finding is the one I would put first rather than third, because it is the one that contradicts the strongest prior. "Add a reranker" is close to universal advice, and the measurement here shows it making retrieval worse at every k on this corpus. What makes it a good finding is not that reranking is bad — it is that the proxy cross-encoder is fitted on features that the first stage has already optimised for, so it reorders without adding information. The transferable version: a reranker only helps if it can see something the first stage could not. A reranker over the same signals as the retriever is an expensive identity function, and occasionally worse than one. |
|
That is a better statement of it than the one in the post, and I am going to lift it into the README's finding. The measurement that would change the conclusion is a genuine cross-encoder doing full attention over the query-document pair — that does see something the bi-encoder cannot. #15 is open for exactly this and is a good second issue for someone. |
What makes this page unusual is keeping the findings that contradict the teaching material rather than quietly re-running until they agreed. Would suggest one addition: for each finding, a line saying what would change the conclusion. "This is true on this corpus with this encoder" is much stronger with "and here is the experiment that would overturn it" attached — it is the difference between a result and a claim. |
|
Agreed and adopting it. Adding a "what would change this" line to all three findings, and to the ADR template — |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Three measurements in these notebooks contradict what the source deck's decision matrices
predict. Each one could have been engineered away. We kept all three, and this post explains
why, because it is the single most important thing about how this material is taught.
The findings
1 · Equal-weight RRF does not beat BM25 alone here. Weighted fusion at α=0.2 does. The
mechanism: RRF gives both legs the same vote, and our offline dense leg is a fifty-year-old
method that is genuinely weaker on this corpus. Fusing a strong retriever with a weak one at
equal weight moves you toward the weak one.
2 · Comparison-question starvation does not reproduce. The matrix predicts one entity
dominating top-k while the other is starved. Our corpus is balanced by construction — every
company has the same number of quarters — so the prevalence ratio between two compared entities
is ≈1 and nothing starves.
3 · No retrieval-score threshold separates answerable from unanswerable (best F1 0.38). Four
signals were tried; all sit near chance. Null questions name real entities in the corpus's own
vocabulary while real questions paraphrase — so the unanswerable ones are lexically closer to
the corpus.
Why we did not fix them
Because a decision matrix names a mechanism you should go and test, and the test is allowed
to come back negative.
A curriculum where every matrix row confirms teaches something corrosive: that these tables are
facts to recite, and that when your measurement disagrees with the slide you should change your
measurement. That is exactly backwards, and it is the habit that produces a team that spends a
sprint fixing starvation it does not have.
Each finding in the README carries a "when the expected result returns" column, because the
matrix is not wrong — it names a real mechanism under conditions our corpus does not meet.
What this means for you
When you run an exercise and your result contradicts what you expected, that is a finding,
not a mistake. Post it in Show & Tell. Then work out the mechanism, and state the condition
under which the expected result would return.
That sequence — measure, contradict, explain, bound — is most of what separates a senior
engineer from a competent one in this field.
Full write-up: ADR-0007.
All reactions