[solution · EX-12] The analyzer audit — I found three more separators past the one in #1

[akash-coded]

Casebook thread. A reconstructed conversation, written for teaching. The people in it
are illustrative; every number is real and reproducible from the notebook cited below.
Replies are welcome — if you hit this yourself, add your numbers.

Submission for
EX-12.

What I did

#1 fixed _ by adding it to tokenchars. The
exercise asks what else the analyzer splits, so I enumerated every non-alphanumeric character
in the corpus and counted affected chunks rather than guessing.

import re, collections
seps = collections.Counter()
for c in chunks:
    for ch in set(re.findall(r"[^a-zA-Z0-9\s]", c["text"])):
        seps[ch] += 1

The numbers

sep   chunks   example                     currently split?
_      1,842   ERR_CONN_RESET, acl_group_id      no (fixed in #1)
-        311   post-mortem, non-blocking         no (fixed in #1)
.        208   v2.1.4, nanorag.store             YES
/         96   docs/adr, ops/runbook             YES
:         74   12:04:31, key:value               YES
@         12   synthetic addresses               YES

The retrieval impact, measured

I built a 40-question slice whose answers require matching a version string, a path, or a
timestamp, and ran it before and after adding each separator:

                          evidence_recall@8   delta        CI
baseline (_ and - only)         0.475           —           —
+ '/'                           0.650        +0.175   [+0.081, +0.271]   real
+ ':'                           0.700        +0.050   [-0.019, +0.121]   band
+ '.'                           0.575        -0.125   [-0.208, -0.041]   REGRESSION

The finding

Adding . to tokenchars makes things worse, and it took me a while to see why.

Once . is a token character, a sentence-ending period stops separating tokens — so
...the interval. Engineering will... indexes as interval.engineering, a term that appears
nowhere else and matches nothing. I gained version strings and lost sentence boundaries across
the entire corpus. The 208 chunks containing versions are swamped by the ~2,400 containing full
stops.

/ is the clear win. : is inside the band on this slice.

What I would ship

tokenize = "unicode61 remove_diacritics 2 tokenchars '_-/'" — and I would not chase .
through a custom tokenizer unless version-string search were a named requirement, because the
right fix is a separate analyzer for identifier fields rather than one global compromise.

[akash-coded]

Marcus · cohort peer

Good result. One thing though — a 40-question slice is small, and you are reporting three deltas
off it. Isn't the : result just the multiple-comparison problem? You tested three separators and
one landed inside the band; that is roughly what you would expect by chance.

[akash-coded]

Dana · reviewer · owns the eval harness

The instinct is right and the application is not quite, which is worth separating because the
distinction comes up constantly.

Multiple comparisons inflate the false-positive rate for significant results. The : result
is not significant — it is inside the band — so the concern does not apply to it. If anything
the correction would make it even less significant.

Where it does apply is /, at +0.175 with an interval well clear of zero. With three tests
at α=0.05, a Bonferroni-corrected threshold is α=0.0167, and that interval still excludes zero
comfortably. So the finding survives the correction, which is the check worth doing rather than
assuming either way.

The real limitation is the one Marcus is circling: 40 questions is a small slice, so every
interval here is wide. The / effect is large enough to survive that; a smaller effect would not
have been measurable at all, and the honest reading is that this slice can detect big wins and is
blind to modest ones.

[akash-coded]

Marcus · cohort peer

That is the distinction I was missing — corrections apply to the significant results, not the
null ones. Filing that away.

[akash-coded]

Priya · cohort peer

The . regression is the best part of this and I want to make sure it does not get lost under
the recommendation.

It is a negative result with a mechanism, and the mechanism generalises past this repo: any
character you add to tokenchars stops being a separator everywhere, not just where you wanted
it. You are trading a rare gain against a global loss, and the arithmetic is chunk counts —
208 chunks with versions against ~2,400 with full stops.

That is a rule I can carry to a real corpus, which is more than I can say for most exercise
write-ups.

[akash-coded]

Agreed, and adopting it. The / addition is going in with this thread linked as the
justification, and the . finding is going into the analyzer section of notebook 04 as a worked
warning rather than a footnote — it is a better teaching example than the one currently there,
because the reader can see the trade in chunk counts.

Worth noting what made this submission good, since EX-12
does not require it: it measured a change it then recommended against. Most submissions report
the option that won.