feat(vocab): stopword deviation analysis — audit the words we DROP (task #13) - #260
Merged
Conversation
…ask #13) The stoplist is itself ungoverned vocabulary: a word that is filler in one domain (set, class, state, required) is a real term in another. tools/stopword_analysis.py audits the dropped words across domains using TWO signals — cross-domain deviation (concentration) AND compositional density (repeated-collocation rate) — because frequency alone can't tell a domain term from a stylistic quirk (e.g. 'and' is concentrated in chatty prose but forms no repeated collocations). Surfaces term-candidate / stylistic / noise; term-candidates are un-stoplist proposals (a remediation signal like the currency loop's candidates and the agreement test's drift). validate-stopword-analysis teeth: domain terms surfaced; a stylistically-concentrated word NOT promoted (the trap frequency alone falls into); a uniform word is noise. stopword-analysis-live audits the shipped STOP over specs/*.md (currently clean: 0 candidates / 38 noise / 9 insufficient). Compositional density is the bigram floor of the k-gram TF-IDF/LSA differential (3..7) to follow.
There was a problem hiding this comment.
Pull request overview
Adds a new “stopword deviation analysis” toolchain to audit the dropped stoplist words across domains, turning the stoplist into a measurable governance surface (task #13) rather than a silent universal assumption.
Changes:
- Introduces
tools/stopword_analysis.pyto score stoplisted words using cross-domain concentration plus compositional density and emitterm-candidatevs non-candidate outputs. - Adds CI “teeth” via
tools/validate_stopword_analysis.pyand wires it intomake validateasvalidate-stopword-analysis; also adds an on-demandstopword-analysis-livetarget. - Adds fixtures and documentation (
specs/stopword-analysis.md) plus changelog entry for v0.1.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/stopword_analysis.py | Implements the deviation + compositional-density analysis and CLI entrypoint. |
| tools/validate_stopword_analysis.py | Adds fixture-based CI assertions to ensure the two-signal discriminator behaves as intended. |
| specs/stopword-analysis.md | Documents the v0.1 method, signals, and Make targets. |
| Makefile | Adds validate-stopword-analysis to validate and adds stopword-analysis-live. |
| fixtures/stopword-analysis/stoplist.json | Defines a small fixture stoplist to exercise the classifier. |
| fixtures/stopword-analysis/domains/narrative.json | Narrative-domain fixture corpus to trigger stylistic concentration (e.g., “and”). |
| fixtures/stopword-analysis/domains/formal.json | Formal-domain fixture corpus to trigger term-like collocations (“empty set”, “state machine”, etc.). |
| CHANGELOG.md | Records the addition of stopword deviation analysis v0.1 and its validation targets. |
Comment on lines
+71
to
+85
| total_adj = sum(pairs.values()) | ||
| repeated_adj = sum(c for c in pairs.values() if c >= 2) | ||
| density = round(repeated_adj / total_adj, 3) if total_adj else 0.0 | ||
|
|
||
| row = {"word": w, "concentration": round(concentration, 3), "candidateDomain": cand, | ||
| "compositionalDensity": density, "occurrences": total_occ, | ||
| "perDomainFreq": {d: round(relfreq[d], 5) for d in domains}, | ||
| "topPartners": sorted(pairs, key=pairs.get, reverse=True)[:5]} | ||
|
|
||
| if concentration < CONCENTRATION_INTERESTING: | ||
| row["verdict"] = "noise" # uniform across domains — a true stopword everywhere | ||
| noise.append(row) | ||
| elif density >= DENSITY_TERMLIKE: | ||
| row["verdict"] = "term-candidate" # concentrated AND recurs in fixed collocations | ||
| row["proposal"] = f"un-stoplist '{w}' in domain '{cand}' (candidate domain term)" |
Comment on lines
+42
to
+49
| def analyze(domains: dict[str, str], stoplist: set[str]) -> dict: | ||
| tokens = {d: raw_tokenize(text) for d, text in domains.items()} | ||
| totals = {d: len(t) or 1 for d, t in tokens.items()} | ||
|
|
||
| interesting, noise, insufficient = [], [], [] | ||
| for w in sorted(stoplist): | ||
| occ = {d: t.count(w) for d, t in tokens.items()} | ||
| total_occ = sum(occ.values()) |
Comment on lines
+93
to
+96
| return {"domains": list(domains), "stoplistSize": len(stoplist), | ||
| "termCandidates": interesting, | ||
| "noiseCount": len(noise), "insufficientDataCount": len(insufficient), | ||
| "noiseWords": [r["word"] for r in noise]} |
Comment on lines
+41
to
+46
| # 2. Each term-candidate is CONCENTRATED and points at the domain that uses it as a term. | ||
| if all(c["concentration"] >= S.CONCENTRATION_INTERESTING and c["candidateDomain"] == "formal" | ||
| for c in r["termCandidates"]): | ||
| CHECKS["candidates:concentrated-in-right-domain"] = True | ||
| else: | ||
| FAILURES.append("a term-candidate was not concentrated in the formal domain") |
- compare UNROUNDED compositional density to the threshold (rounding could flip a boundary verdict); round only for reporting. - precompute a Counter per domain instead of t.count(w) per (word, domain) — O(tokens + words*domains). - expose the three verdicts distinctly (stylisticWords/uniformNoiseWords + counts); keep the flattened noiseWords (their union) for existing consumers. - validator: require >=1 term-candidate before the all(...) concentration check (all([]) is vacuously True and would mask an empty result).
…n edit The previous commit accidentally removed the repeated_adj assignment while switching to the unrounded-density comparison, causing a NameError. Restored; all teeth + make validate green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The stoplist is ungoverned vocabulary too
The vocab-currency loop drops a fixed stoplist so domain terms surface. But that list is a hard-coded governance assumption: a word that's filler in one domain (
set,class,state,required,value) is a real term in another (math, OOP, state machines, config). Dropping it universally erases signal where it matters. So we audit the dropped words the same way we audit the kept ones.Two signals — because frequency alone lies:
concentration) — is the word's relative-frequency mass concentrated in a subset of domains?and) glues arbitrary unique content.Verdicts:
term-candidate(concentrated + compositional → a domain term hiding in the stoplist → propose un-stoplisting there),stylistic(concentrated by style only),noise(uniform). Term-candidates are a remediation signal, like the currency loop's candidate terms and the agreement test's drift.Teeth (
make validate-stopword-analysis)set/class/state/required) surfaced as candidates in the right domainand) is NOT wrongly promoted — frequency deviation alone would; compositional density saves itthe) is noisemake stopword-analysis-liveaudits the real shippedSTOPoverspecs/*.md— currently clean (0 candidates / 38 noise / 9 insufficient): no domain terms hiding in it for this corpus, but the mechanism will catch them across other domains/repos.Next (as discussed): the k-gram TF-IDF/LSA differential over orders 3..7 — compositional density here is its bigram floor; the differential confirms a candidate by showing its signal grows with n-gram order (participates in domain-specific higher-order collocations) rather than staying diffuse.