Skip to content

feat(enrichment): add non-inclusive terminology analyzer - #3268

Merged
JSONbored merged 1 commit into
JSONbored:mainfrom
jaso0n0818:feat/terminology-analyzer-2031
Jul 5, 2026
Merged

feat(enrichment): add non-inclusive terminology analyzer#3268
JSONbored merged 1 commit into
JSONbored:mainfrom
jaso0n0818:feat/terminology-analyzer-2031

Conversation

@jaso0n0818

Copy link
Copy Markdown
Contributor

Closes #2031

What

A new local REES analyzer, terminology, that flags non-inclusive terms a PR adds in identifiers or comments (whitelist/blacklist, master/slave) and suggests the neutral replacement — a config-driven house-style signal. Pure compute over added lines, no network.

Detection (stateless, precision-first)

  • Token-based, not substring: each added line is tokenized on camelCase + acronym boundaries, snake_case, and any non-alphanumeric run, then each lowercased token is matched against a bounded term→suggestion table. A non-inclusive term is flagged wherever it appears as a whole token — including as a camelCase/snake_case compound component, which is the point of scanning identifiers: slaveNodes[slave, nodes], masterNodes[master, nodes], blacklistIDs[blacklist, ids], and master_key are all flagged. What is never flagged is a different word that merely contains the letters: masterclass, postmaster, mastermind, slavery, and enslaved each tokenize to a single distinct token, so there is no substring false positive (the issue's explicit precision requirement).
  • URLs are blanked before tokenizing, so a link path segment (http://host/master/x) cannot trip a finding.
  • The term→suggestion table is a bounded in-file policy so a self-host operator can read the whole thing: whitelist(+ed/+ing/+s) → allowlist…, blacklist(+ed/+ing/+s) → denylist…, master → main or primary, slave(+s) → replica or secondary. Inflections we want are listed explicitly rather than fuzzy-stemmed, so an unlisted form like mastered/slavery — a genuinely different word — is out of scope.
  • Added lines only, line-cited via hunk headers, with the shared \ No newline line-counter fix. Terms de-duplicated per line; findings capped (maxFindings: 25) per file and globally.

Registration

Registered as a local descriptor (category quality, cost local, requires ["files"]) with an inline render(), following the looseRange descriptor shape. All wiring updated: types.ts (TerminologyFinding + terminology? key), render.ts, analyzer-registry.test.ts, root src/review/enrichment-analyzer-names.ts, root test/unit/enrichment-wire.test.ts, and the generated analyzer-metadata.json / rees-analyzers.ts / .env.example via node scripts/generate-analyzer-metadata.mjs.

Tests

review-enrichment/test/terminology.test.ts (14 tests) covers: tokenization across camelCase/snake_case/non-alphanumeric with URL blanking, each term flagged with its suggestion, camelCase/acronym plural compounds intentionally matched on their non-inclusive component (slaveNodes→slave, masterNodes→master, blacklistIDs→blacklist — the documented-and-tested behavior the gate asked to pin down), token-based no-substring-false-positive cases (masterclass/postmaster/mastermind/slavery/enslaved), per-line de-duplication, URL suppression, case-insensitive matching, added-line scanning with exact locations, a single line with two distinct terms emitting two findings and filling a maxFindings:1 budget with just the first, added-lines-only with line-number accuracy across mixed hunks, the per-file cap + maxFindings: 0, the entrypoint's global cap across files, the no-files case, and the rendered brief section. Analyzer metadata is regenerated and committed.

@jaso0n0818
jaso0n0818 requested a review from JSONbored as a code owner July 5, 2026 00:04
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@loopover-orb loopover-orb Bot added the gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. label Jul 5, 2026
@loopover-orb

loopover-orb Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Important

🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪

🔍 Gittensory is reviewing…

AI analysis is in progress. This comment will update when the review is complete.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed · 🟪 Reviewing

@codecov

codecov Bot commented Jul 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.12%. Comparing base (295c213) to head (e658a12).
⚠️ Report is 6 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #3268   +/-   ##
=======================================
  Coverage   94.12%   94.12%           
=======================================
  Files         276      276           
  Lines       30202    30202           
  Branches    11011    11011           
=======================================
  Hits        28429    28429           
  Misses       1127     1127           
  Partials      646      646           
Files with missing lines Coverage Δ
src/review/enrichment-analyzer-names.ts 100.00% <ø> (ø)
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@JSONbored
JSONbored merged commit 2b3d439 into JSONbored:main Jul 5, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. manual-review Gittensor contributor context

Development

Successfully merging this pull request may close these issues.

feat(enrichment): non-inclusive / banned-terminology analyzer

2 participants