Skip to content

feat(v0.19): make --spi pay off — framework metadata + retrieval boost for Hono/Fastify/tRPC/Prisma#129

Merged
mohanagy merged 1 commit into
mainfrom
feat/v0.19-spi-payoff
May 11, 2026
Merged

feat(v0.19): make --spi pay off — framework metadata + retrieval boost for Hono/Fastify/tRPC/Prisma#129
mohanagy merged 1 commit into
mainfrom
feat/v0.19-spi-payoff

Conversation

@mohanagy
Copy link
Copy Markdown
Owner

@mohanagy mohanagy commented May 11, 2026

Summary

Closes the gap between "SPI substrate exists" and "users see token savings." Two complementary pieces.

1. Confirmed: framework metadata flows end-to-end with --spi

My earlier claim that the graph.json serializer was dropping framework_role / route_path was wrong — it was a bad test assertion (whitespace formatting mismatch). Tightened the v0.18 test to assert what's actually produced:

{
  "label": "listUsers()",
  "framework": "express",
  "framework_role": "express_route",
  "node_kind": "route",
  "route_path": "/users",
  ...
}

The serializer in src/pipeline/export.ts already spreads all node attributes via ...attributes. No serializer change needed.

2. Retrieval boost for the v0.17 substrates

frameworkBoostForNode() in src/runtime/retrieve.ts previously only handled express / redux / reactRouter / nest / next. v0.17 added 4 more substrates (Hono / Fastify / tRPC / Prisma) but retrieval ignored their framework_role values.

Now wired:

  • New profile slots: hono / fastify / trpc / prisma
  • New intents: pluginIntent / procedureIntent / queryIntent / mutationIntent / subscriptionIntent / modelIntent
  • Boost rules for all 11 new framework_role values (hono_route, hono_middleware, hono_app, fastify_route, fastify_plugin, fastify_app, trpc_procedure_query/mutation/subscription, trpc_router, prisma_client)
  • Compatibility filter: activeFrameworksForProfile now registers hono / fastify / trpc / prisma so framework-compatible filtering recognises them
  • Express disambiguation: Express boost is suppressed when Hono or Fastify are explicit in the question (prevents misrouting "show me hono routes" to Express handlers)

Token / benchmark impact

Framework-shaped questions ("show me hono routes", "what tRPC mutations exist", "find the Prisma client") now favour the structurally-correct nodes via framework_role boost. Concretely:

  • Fewer plain function/class matches push the substrate-correct nodes off the budget cutoff
  • The user-visible context_pack response surfaces the right nodes inside budget instead of bumping it
  • Pure code-comprehension queries are unaffected (no boost fires)

Test plan

  • npm run typecheck clean
  • npm run build clean
  • npm run test:run — 1817/1817 pass (110 files)
  • New tests cover Hono/Fastify/tRPC/Prisma boost paths + negative case (unrelated question gets no boost)
  • v0.18 test tightened to actually assert framework metadata in graph.json

Summary by CodeRabbit

  • New Features

    • Extended framework detection and support for Hono, Fastify, tRPC, and Prisma alongside existing frameworks.
    • Improved retrieval ranking and context suggestions based on detected framework patterns and roles.
  • Tests

    • Added comprehensive framework-aware retrieval validation tests to ensure accurate context ranking across multiple framework substrates.

Review Change Stack

…t for Hono/Fastify/tRPC/Prisma

Two complementary pieces that turn the v0.14-v0.18 substrate work into actual end-user value:

## 1. Confirmed: framework metadata flows end-to-end with --spi

Tightened tests/unit/generate-spi-flag.test.ts to assert that graph.json nodes produced via --spi carry the framework_role / route_path / node_kind / framework fields. The serializer (toJson in src/pipeline/export.ts) already spreads all node attributes via ...attributes — my earlier claim that 'fields are dropped' was wrong (the test assertion was checking for the wrong whitespace formatting). Verified by debug test that the express handler node in graph.json contains framework:'express', framework_role:'express_route', node_kind:'route', route_path:'/users'.

## 2. Retrieval boost for the v0.17 substrates (Hono / Fastify / tRPC / Prisma)

The existing frameworkBoostForNode() in src/runtime/retrieve.ts only handled express/redux/reactRouter/nest/next. v0.17 added 4 more substrates but the retrieval engine ignored their framework_role values, so questions like 'show me hono routes' got no boost over plain label matching.

Now wired:

- New profile slots: hono / fastify / trpc / prisma

- New intents: pluginIntent / procedureIntent / queryIntent / mutationIntent / subscriptionIntent / modelIntent

- Boost rules:

  * hono_route: 4 with routeIntent, 1.5 otherwise

  * hono_middleware: 2.5 with middlewareIntent, 1 otherwise

  * fastify_route: 4 with routeIntent, 1.5 otherwise

  * fastify_plugin: 2.5 with pluginIntent or middlewareIntent

  * trpc_procedure_query: 3.5 with queryIntent or procedureIntent, 1.5 otherwise

  * trpc_procedure_mutation: 3.5 with mutationIntent or procedureIntent, 1.5 otherwise

  * trpc_procedure_subscription: 3.5 with subscriptionIntent or procedureIntent, 1.5 otherwise

  * trpc_router: 2 with routeIntent or procedureIntent, 0.75 otherwise

  * prisma_client: 3 with modelIntent, 1.5 otherwise

- activeFrameworksForProfile registers the new framework names so the framework-compatible filter recognises hono/fastify/trpc/prisma

- Bonus: express boost is now suppressed when hono/fastify are explicit (prevents misrouting Hono questions to Express)

## What this means for tokens / benchmarks

Same number of nodes per retrieval, but framework-shaped questions ('show me hono routes', 'what tRPC mutations exist', 'find the Prisma client') now favour the structurally-correct nodes via framework_role boost. Concretely: fewer plain function/class matches push the substrate-correct nodes off the budget cutoff, which lets the user-visible context_pack response surface the right nodes inside the budget instead of bumping it.

Whether this moves the benchmark numbers depends on the benchmark queries — for framework-shaped queries it should; for pure code-comprehension queries it shouldn't (no boost fires).

## Tests

- 5 new tests in tests/unit/retrieve-framework-boost-v0-19.test.ts: Hono/Fastify/tRPC/Prisma boost paths + a negative case (unrelated question gets no boost)

- Updated the v0.18 test to actually assert framework_role + route_path flow into graph.json

- 1817/1817 pass, typecheck + build clean
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 11, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 683ab116-2aa5-44ea-a5fa-7336cd41180a

📥 Commits

Reviewing files that changed from the base of the PR and between 03c84b8 and cf59af9.

📒 Files selected for processing (3)
  • src/runtime/retrieve.ts
  • tests/unit/generate-spi-flag.test.ts
  • tests/unit/retrieve-framework-boost-v0-19.test.ts

📝 Walkthrough

Walkthrough

This PR extends the framework-aware retrieval system to recognize and boost framework-specific nodes for Hono, Fastify, tRPC, and Prisma. The runtime retrieval module gains new profile types, intent signals, framework selection logic with precedence rules, and dedicated node boost scoring. Tests validate SPI attribute propagation and framework-specific retrieval behavior.

Changes

Framework-Aware Retrieval Boost Extension

Layer / File(s) Summary
Framework Question Profile Type Extensions
src/runtime/retrieve.ts
FrameworkQuestionProfile interface extends with boolean fields for Hono, Fastify, tRPC, and Prisma framework shaping; adds new intent flags for plugin/procedure/query/mutation/subscription/model.
Framework Intent Detection and Active Frameworks
src/runtime/retrieve.ts
buildFrameworkQuestionProfile adds explicit-question detection and intent signal expansion for the four new frameworks; activeFrameworksForProfile includes them in the active set when profile flags are true.
Framework Selection with Precedence Rules
src/runtime/retrieve.ts
Framework selection logic treats Hono/Fastify as higher-priority signals that suppress Express; expanded frameworkShaped gating covers all four new substrates.
Framework-Specific Node Boost Rules
src/runtime/retrieve.ts
frameworkBoostForNode adds dedicated boost rules mapping framework roles (Hono route/middleware/app, Fastify route/plugin/app, tRPC procedure query/mutation/subscription/router, Prisma client/model_access) to score adjustments.
SPI Attribute Propagation Validation
tests/unit/generate-spi-flag.test.ts
Express/SPI test now parses graph.json and validates that framework, framework_role, node_kind, and route_path attributes are correctly serialized on nodes.
Framework Boost Test Suite Infrastructure
tests/unit/retrieve-framework-boost-v0-19.test.ts
Establishes v0.19 framework-boost test suite with sandboxed file creation, graph generation/loading, and per-test setup/teardown scaffolding.
Framework-Specific Boost Test Cases
tests/unit/retrieve-framework-boost-v0-19.test.ts
End-to-end tests for Hono routes, Fastify plugins, tRPC mutations/queries, and Prisma client usage verify positive framework_boost scoring; control test ensures non-framework questions do not trigger high boosts.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

  • mohanagy/graphify-ts#127: Introduces SPI pipeline that projects framework_role and framework metadata consumed by this PR's retrieval boost logic.
  • mohanagy/graphify-ts#126: Release PR packaging and documenting the same framework substrate implementations (Hono, Fastify, tRPC, Prisma) added in this PR.
  • mohanagy/graphify-ts#128: Related SPI integration work that ships framework metadata into graph.json, enabling the framework/framework_role assertions in this PR's tests.

Poem

🐰 Four frameworks now boost with cheer,
Hono, Fastify, tRPC here!
Prisma joins the retrieval dance,
With profile types and role-based stance.
Intent signals bloom and scores advance! 🌟

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately summarizes the main change: extending framework metadata and retrieval boost support to four new substrates (Hono, Fastify, tRPC, Prisma) as part of making the --spi flag more effective.
Description check ✅ Passed The description is well-structured with clear sections covering both pieces of work, token impact, and test plan. It includes a summary, implementation details, testing verification, and checklist items completed.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/v0.19-spi-payoff

Comment @coderabbitai help to get the list of available commands and usage tips.

@mohanagy mohanagy merged commit b0bf431 into main May 11, 2026
7 checks passed
mohanagy added a commit that referenced this pull request May 11, 2026
…ethod / slice_name / procedure_name (#137)

* feat(#133): metadata-aware framework boost in retrieve — uses route_path / http_method / slice_name / procedure_name / mount_path

Generalises framework-aware retrieval from 'role-only' boost to 'role + metadata' boost. PR #129 added boost rules per framework_role. This PR uses the structural metadata each role carries to add targeted substring-match boosts.

## Substrate metadata audit + plumbing fix

Discovered while writing tests: Express was tagging route_path but NOT http_method on framework_metadata. The Hono detector already did this; Express was behind. Patched framework-express.ts emitRouteForCall to co-attach http_method alongside route_path when the path is statically known. Dynamic-path contract (no metadata) preserved.

## New boost rules in frameworkBoostForNode

When the question contains a substring matching the node's metadata, add a targeted boost on top of the existing role-based boost:

- route_path match (>=3 chars, substring): +3 boost

- http_method match (case-insensitive): +1.5 boost

- mount_path match (>=3 chars, substring): +1.5 boost

- slice_name match (>=2 chars, substring): +2 boost

- procedure_name match (>=2 chars, substring): +2 boost

- router_name match (>=2 chars, substring): +1 boost

## Why these substrings + thresholds

Substrings, not exact match: users phrase queries informally ('the GET /users/:id route', 'how does cancelOrder work', 'auth slice'). Substring match catches these without requiring the question to syntactically own the path.

Minimum-length guards prevent noise: a 1-char route_path like '/' would match every English sentence. 3-char minimum on paths + 2-char minimum on names is the threshold I found empirically; it ignores '/' and 't' / 'a' style names while keeping real ones.

## Plumbing

- FrameworkNodeMetadata interface added (route_path | http_method | mount_path | slice_name | procedure_name | router_name)

- frameworkMetadataFromAttributes(attributes) extracts the fields from a graph node attribute bag

- frameworkBoostForNode signature extended: now takes (profile, nodeKind, role, metadata, questionLower)

- Both call sites updated (scoredNodeFromGraphEntry path + seedCandidates path)

## Tests

- 5 new tests in tests/unit/retrieve-metadata-boost-133.test.ts: route_path match, http_method match (POST vs GET differentiation), slice_name match (auth vs counter), procedure_name match (cancelOrder vs getUser), negative case (unrelated query gets no metadata boost)

- 3 existing Express test fixtures updated to assert http_method alongside route_path on framework_metadata

- 1822/1822 tests pass, typecheck + build clean

## Benchmark re-run

Re-ran the #130 fixture with #133 enabled. spi total: 946 tokens (unchanged). legacy total: 1160 (varies ~10% run-to-run on this small fixture). The substantive improvement is in CORRECTNESS — the unit tests prove the boost differentiates correctly between nodes that previously tied (POST vs GET routes on /users, auth vs counter slices, cancelOrder vs getUser procedures).

* fix(#133): metadata-only seeding + word-boundary http_method match (CodeRabbit)

Two valid findings from CodeRabbit on PR #137:

1) MAJOR — metadata-only matches couldn't enter the seed set. Code only pushed seeds when score.total > 0, meaning a handler named 'h()' tagged with route_path: '/orders/:id' was invisible to lexical retrieval even when the question literally contained that path. Fix: compute framework boost BEFORE the gate, and allow seeding when either lexical score OR metadata boost is positive. Metadata-only seeds get evidence tier >= 1.

2) HTTP-verb substring match was too greedy — 'GET' would match 'budget', 'forget', 'target' etc. Fix: switched from .includes(verb) to a word-boundary regex \\b${verb}\\b.

Tests:

- New regression: handler 'h()' with no label overlap is now seeded when route_path matches the question. Pre-fix this node was invisible.

- New regression: 'budget' in the question does NOT trigger the http_method GET boost; literal 'GET' does.

- 1824/1824 pass.

* fix(#133): don't double-count frameworkBoost in async retrieval + escape regex metacharacters (CodeRabbit)

Two valid follow-ups from CodeRabbit on PR #137:

1) MAJOR — frameworkBoost was double-counted in the async retrieval path. lexicalScoresById holds match_score values from retrieveContext, which already bake in frameworkBoost. The downstream merge then added candidate.frameworkBoost AGAIN, inflating lexical winners by an extra +5-8 points. Fix: gate frameworkBoost addition on lexicalScore === 0, so it only contributes to semantic-only candidates. Lexical winners keep their already-boosted score; semantic-only candidates with metadata matches still get the boost.

2) Defensive regex escape on http_method. Even though HTTP methods are letters-only in practice, escaping metacharacters before constructing the RegExp prevents future regression if metadata.http_method ever leaks user-controlled or otherwise unexpected content.

1824/1824 tests still pass.
mohanagy added a commit that referenced this pull request May 11, 2026
… v0.19+v0.20) (#139)

Combined release covering both v0.19-spi-payoff (#129/#130/#133) and v0.20-context-compiler (#131/#132/#134) milestones.
mohanagy added a commit that referenced this pull request May 11, 2026
…'what's new' callout + test recipe (#140)

* docs(readme): v0.20 refresh — framework substrate audit + npm video fallback + 'what's new' callout + test recipe

Honest answers to all 4 user concerns:

1) README was stale for v0.18-v0.20. Updated:

- Top-of-file 'What's new in v0.20' callout with the measured -26%/-32% deltas and a runnable 'Try it' recipe

- Line 46 framework list — now includes Hono / Fastify / tRPC / Prisma (was missing all 4)

- Line 243 'Honest disclosure' framework list — same correction

- Roadmap: added the missed v0.19/v0.20 items (#129 framework-aware boost, #131 value-per-token, #132 signature mode, #133 metadata match, #134 readiness criteria, plus the #130 benchmark receipts). Added #135 (task-conditioned slicing v1) and #84 (Python/Go deeper passes) to Planned.

2) Test recipe — added inline to the 'What's new' block:

  graphify-ts generate . --spi

  jq '.nodes[] | select(.framework_role)' graphify-out/graph.json

3) npm video fallback — added a shields.io 'Watch the 30-second demo' button above the bare video URL. npm renders the button; GitHub renders both the button and the inline video. The button links back to the GitHub README anchor where the video plays.

4) Light cleanup only. Larger structural reorg (Why/What/Core concept overlap, section ordering) intentionally NOT in this PR — too risky to bundle with the v0.20 content fixes. Saved for a follow-up.

* docs(readme): restructure — 40% shorter, no roadmap, consolidated sections

Replaces the previous 20+ section sprawl with 12 focused sections per user feedback ('it looks crowded and messy').

## New structure (12 sections, down from 20+)

1. Title + pitch + badges

2. Demo (video + npm shields fallback)

3. Quickstart

4. What graphify-ts is (consolidates the old Why + What it does + Core concept + What's it for sections)

5. Measured results (one benchmark table — dropped the ASCII turns walkthrough that was longer than the benchmark itself)

6. Works with your AI tools

7. MCP tools

8. Common commands

9. Trust + limitations (merged old What stays local + Honest disclosure)

10. Documentation & receipts (merged old Public proof + Documentation; CHANGELOG link replaces the in-README Roadmap per user request)

11. Contributors (preserved verbatim with auto-update workflow note)

12. Credit & license

## What was removed (intentionally)

- Roadmap section — user explicitly opted out. The CHANGELOG link in Documentation covers this.

- 'See it work' ASCII turns walkthrough — verbose and redundant with the benchmark table.

- 'Context-plane surfaces' section — its content overlapped with MCP tools + Common commands; not a distinct concept.

- Per-use-case prose ('Our AI bill', 'Code review takes hours', 'Can't ship to cloud') — collapsed into 3 bullets in What graphify-ts is. The product positioning is now a few sentences, not 3 sub-headings.

## What was preserved

- All benchmark numbers + receipts links

- Full agent install table

- Full MCP core/full tool surface

- Common commands reference

- Contributors section with auto-update marker

- Credit + license + Safi Shamsi attribution

* docs(readme): restore phrases pinned by existing tests (no behavioral change)

After the v0.20 README restructure, 6 README-pinning tests failed. Restored the exact phrases each test expects:

- 'context plane' + 'context compiler' in the lede (used by package-metadata.test.ts and why-graphify-doc.test.ts to assert product positioning)

- '## License' as its own H2 (split from 'Credit & license' into 'Credit' + 'License')

- '| **Latency**' bold cell formatting in the benchmark table

- 'These six MCP tools' + 'The full surface is 25 tools' phrasing

- Full enumeration of full-profile MCP tools (the previous shortening to '...and more' broke the test that pins specific tool names like 'get_neighbors')

No structural / length impact — the README is still 12 sections and ~219 lines. These are wording-level restorations to satisfy semantic-content tests.

1833/1833 pass.
mohanagy added a commit that referenced this pull request May 11, 2026
* docs(readme): v0.20 refresh — framework substrate audit + npm video fallback + 'what's new' callout + test recipe

Honest answers to all 4 user concerns:

1) README was stale for v0.18-v0.20. Updated:

- Top-of-file 'What's new in v0.20' callout with the measured -26%/-32% deltas and a runnable 'Try it' recipe

- Line 46 framework list — now includes Hono / Fastify / tRPC / Prisma (was missing all 4)

- Line 243 'Honest disclosure' framework list — same correction

- Roadmap: added the missed v0.19/v0.20 items (#129 framework-aware boost, #131 value-per-token, #132 signature mode, #133 metadata match, #134 readiness criteria, plus the #130 benchmark receipts). Added #135 (task-conditioned slicing v1) and #84 (Python/Go deeper passes) to Planned.

2) Test recipe — added inline to the 'What's new' block:

  graphify-ts generate . --spi

  jq '.nodes[] | select(.framework_role)' graphify-out/graph.json

3) npm video fallback — added a shields.io 'Watch the 30-second demo' button above the bare video URL. npm renders the button; GitHub renders both the button and the inline video. The button links back to the GitHub README anchor where the video plays.

4) Light cleanup only. Larger structural reorg (Why/What/Core concept overlap, section ordering) intentionally NOT in this PR — too risky to bundle with the v0.20 content fixes. Saved for a follow-up.

* docs(readme): restructure — 40% shorter, no roadmap, consolidated sections

Replaces the previous 20+ section sprawl with 12 focused sections per user feedback ('it looks crowded and messy').

## New structure (12 sections, down from 20+)

1. Title + pitch + badges

2. Demo (video + npm shields fallback)

3. Quickstart

4. What graphify-ts is (consolidates the old Why + What it does + Core concept + What's it for sections)

5. Measured results (one benchmark table — dropped the ASCII turns walkthrough that was longer than the benchmark itself)

6. Works with your AI tools

7. MCP tools

8. Common commands

9. Trust + limitations (merged old What stays local + Honest disclosure)

10. Documentation & receipts (merged old Public proof + Documentation; CHANGELOG link replaces the in-README Roadmap per user request)

11. Contributors (preserved verbatim with auto-update workflow note)

12. Credit & license

## What was removed (intentionally)

- Roadmap section — user explicitly opted out. The CHANGELOG link in Documentation covers this.

- 'See it work' ASCII turns walkthrough — verbose and redundant with the benchmark table.

- 'Context-plane surfaces' section — its content overlapped with MCP tools + Common commands; not a distinct concept.

- Per-use-case prose ('Our AI bill', 'Code review takes hours', 'Can't ship to cloud') — collapsed into 3 bullets in What graphify-ts is. The product positioning is now a few sentences, not 3 sub-headings.

## What was preserved

- All benchmark numbers + receipts links

- Full agent install table

- Full MCP core/full tool surface

- Common commands reference

- Contributors section with auto-update marker

- Credit + license + Safi Shamsi attribution

* docs(readme): restore phrases pinned by existing tests (no behavioral change)

After the v0.20 README restructure, 6 README-pinning tests failed. Restored the exact phrases each test expects:

- 'context plane' + 'context compiler' in the lede (used by package-metadata.test.ts and why-graphify-doc.test.ts to assert product positioning)

- '## License' as its own H2 (split from 'Credit & license' into 'Credit' + 'License')

- '| **Latency**' bold cell formatting in the benchmark table

- 'These six MCP tools' + 'The full surface is 25 tools' phrasing

- Full enumeration of full-profile MCP tools (the previous shortening to '...and more' broke the test that pins specific tool names like 'get_neighbors')

No structural / length impact — the README is still 12 sections and ~219 lines. These are wording-level restorations to satisfy semantic-content tests.

1833/1833 pass.

* Implement context compiler payoff

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix PR comments and CI

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix remaining review comments

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant