feat: enhance documentation tools with code example extraction and re…

[dkalinovInfra]

Description

Adds a new get_example MCP tool that returns only the runnable code examples for a component doc, with no prose. It resolves a component (plus optional sub-feature topic) to a single doc, extracts the fenced code blocks, groups them per example, and labels each with its section heading. An optional language filter narrows the response to one fence language.

Motivation: most Ignite UI questions are "show me how to use X in code". get_doc answers those by returning the entire document, which is dominated by content the model does not need for a code answer. Measured across all 1,230 docs in the shipped DB, get_example returns 36% fewer tokens than get_doc on average, and 59% fewer when a language is passed.

Along the way this PR extracts the doc-name resolution that get_doc had inline into a shared resolveDoc(), so both tools resolve names identically, and fixes two accuracy problems that the shared path exposed:

1. Angular grid-variant docs resolved to the wrong doc. Angular keys these docs with a compact prefix (treegrid-filtering), while the user-facing component name is hyphenated (tree-grid). Composing component + topic produced names no doc uses, which fell through to the search fallback and landed on a plausible-but-wrong doc — tree-grid-editing served treegrid-batch-editing. A new applyCompactGridPrefix() rewrites the prefix before the fallback runs. Measured over all 80 Angular tree/hierarchical/pivot-grid topic docs: 13 wrong (16%) → 0 wrong (0%).

2. Silent wrong-doc substitution. When the search fallback serves a doc other than the one requested, the response now says so instead of reading as an exact hit. resolveDoc() returns a fuzzy flag set only on the search path, so deterministic resolutions (direct hit, alias, prefix rewrite) stay silent and produce no noise.

Related Issue

Closes

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update
• Refactoring / code cleanup
• Build / CI configuration change

"Breaking change" is ticked under the template's literal wording — existing functionality changes — not because any client API breaks. See Behavior changes to existing tools below; both changes are additive to the response, and no tool signature, name, or schema is removed or altered.

Affected Packages

• igniteui-cli (packages/cli)
• @igniteui/cli-core (packages/core)
• @igniteui/angular-templates (packages/igx-templates)
• @igniteui/angular-schematics (packages/ng-schematics)
• @igniteui/mcp-server (packages/igniteui-mcp)

Behavior changes to existing tools

Two shipped tools change behavior. Neither is a schema change, but both are visible to callers:

get_doc gains the search fallback. It previously returned isError when a name did not map mechanically; via the shared resolveDoc() it now runs a full-text search as a last resort and can serve the closest related doc. Those responses are prefixed with an explicit notice:

Note: no doc named `hierarchical-grid-paging` exists — showing the closest match,
`hierarchical-grid-overview`. The content below may cover a different feature than
requested; use list_components or search_docs to see other options.

search_docs drops natural-language stopwords. FTS4 uses implicit AND, so leaving how/do/i in "how do I enable row editing" forced those words to appear in a doc and collapsed recall to near zero. Stopwords are now stripped, falling back to the full term list when every term is a stopword. and/or/but are deliberately kept as ordinary terms.

Checklist

• I have tested my changes locally (npm run test)
• I have built the project successfully (npm run build)
• I have run the linter (npm run lint)
• I have added/updated tests as needed
• My changes do not introduce new warnings or errors

Exactly what was run, since two of these need qualification:

Command	Result
npm test in packages/igniteui-mcp/igniteui-doc-mcp	268 passed (9 files), 0 failures
npm run build (root)	Clean — tsc, config-schema, and build:mcp all succeed
npm run lint (root)	0 findings in any changed file

• The root npm run test was not run to completion: its pretest hook runs npm run lint, which currently fails on ~11,984 parse errors originating entirely from the documentation git submodules (webcomponents/igniteui-webcomponents, etc.) being picked up by the root ESLint config. That is pre-existing on master and unrelated to this PR. The monorepo Jasmine suite also has 11 known pre-existing ng-schematics "migration-0X not found" failures.
• Lint output was filtered for the changed paths (igniteui-doc-mcp/src, igniteui-doc-mcp/scripts) and returns zero matches.

Additional Context

Measured token savings

scripts/benchmark-tool-tokens.ts (added in this PR) counts tokens with js-tiktoken / o200k_base on the exact string each tool places in content[0].text, so the numbers carry no model or tool-call overhead:

framework	docs	get_doc	get_example	vs doc	get_example + language	vs doc	no-example docs
angular	374	4,373	3,016	−31%	1,622	−63%	11 (3%)
react	287	3,579	2,359	−34%	1,623	−55%	11 (4%)
webcomponents	299	2,636	1,268	−52%	897	−66%	11 (4%)
blazor	270	3,535	2,289	−35%	1,729	−51%	9 (3%)
ALL	1,230	3,581	2,278	−36%	1,469	−59%	42 (3%)

Corpus totals over the 1,188 docs that contain examples: 4,277,559 → 2,800,996 tokens (−35%), so the 3% of docs with no examples are not skewing the mean.

The language filter is doing most of the work, which follows from the pipeline: LLM compression already strips most prose, so a compressed doc is largely code blocks. Dropping prose alone cannot save much; dropping the other language variants of every sample can. The tool and parameter descriptions were written to push the model toward always passing language when the target language is known.

Run it with:

npx tsx scripts/benchmark-tool-tokens.ts
npx tsx scripts/benchmark-tool-tokens.ts --framework angular --language typescript --csv dist/bench.csv

Sample output

Code examples from `treegrid-filtering` (angular) (html only):

## Angular Tree Grid Filtering Example

```html
<igx-tree-grid #treeGrid [data]="data" [allowFiltering]="true" ...
```

Known gaps, deliberately not addressed here

• No output cap. 2% of get_example calls still return >8k tokens; the worst is angular/types-stacked-chart (19,849 → 17,001, only 14% saved) because those chart docs are almost entirely code already. Mitigated by pushing language in the descriptions, which is a softer guarantee than a cap. A follow-up could cap at ~6–8k tokens with a "N more examples — narrow with topic/language" footer.
• Redundant grouping logic. extractCodeExamples() groups consecutive blocks into examples, but mergeExamplesByHeading() then collapses every adjacent same-heading example, so the two mechanisms cancel — verified across real docs to produce byte-identical output either way. It is dead complexity rather than a bug, so it is left for a cleanup PR.

Files in this PR

File	Change
src/index.ts	Register get_example; route get_doc through resolveDoc(); emit the substitution notice
src/tools/doc-tools.ts	resolveDoc(), applyCompactGridPrefix(), formatSubstitutionNotice(), extractCodeExamples(), mergeExamplesByHeading(), formatCodeExamples(), canonicalLang(); stopwords in sanitizeSearchDocsQuery(); normalizeDocName() now trims and folds spaces/underscores
src/tools/constants.ts	get_example tool description
src/__tests__/tools/doc-tools.test.ts	+468 lines of tests
scripts/benchmark-tool-tokens.ts	New token benchmark (not covered by tsconfig.json's include: ["src/**/*"], consistent with the other scripts/ files; typechecked separately)

Note for whoever opens the PR: the working branch also carries unrelated modifications that are not staged and must not be included — submodule pointer updates, both igniteui-docs.db files, and several untracked scratch files.

[coveralls]

coverage: 87.816%. remained the same — dkalinov/mcp-examples into master

[Copilot]

Pull request overview

This PR enhances the Ignite UI docs MCP server (@igniteui/mcp-server) by adding a new get_example tool that returns code-only snippets (optionally language-filtered) and by refactoring doc-name resolution into a shared resolveDoc() path used by both get_doc and get_example. It also improves search recall by stripping common natural-language stopwords before FTS4 matching, and adds a benchmark script to quantify token savings.

Changes:

• Add get_example MCP tool that extracts and formats fenced code blocks (grouped/labeled by section headings), with optional fence-language filtering.
• Refactor doc resolution into resolveDoc() with Angular-specific compact grid-prefix rewriting and a guarded FTS search fallback (with explicit substitution notice when fuzzy).
• Improve search_docs recall by removing natural-language stopwords from multi-term queries; expand unit tests and add a token benchmarking script.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
packages/igniteui-mcp/igniteui-doc-mcp/src/tools/doc-tools.ts	Adds stopword-aware query sanitization, shared resolveDoc() logic, Angular compact grid-prefix rewrite, and code example extraction/formatting utilities.
packages/igniteui-mcp/igniteui-doc-mcp/src/tools/constants.ts	Adds the get_example tool description and usage guidance (including language filter recommendation).
packages/igniteui-mcp/igniteui-doc-mcp/src/index.ts	Registers get_example, routes get_doc through resolveDoc(), and emits a substitution notice for fuzzy resolutions.
packages/igniteui-mcp/igniteui-doc-mcp/src/tests/tools/doc-tools.test.ts	Adds extensive unit tests covering stopwords, doc resolution behavior, and code extraction/formatting helpers.
packages/igniteui-mcp/igniteui-doc-mcp/scripts/benchmark-tool-tokens.ts	Adds a script to benchmark token counts for get_doc vs get_example across the corpus.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.