Add frontmatter transformers#58
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (13)
📜 Recent review details🧰 Additional context used📓 Path-based instructions (4)**/*.{ts,tsx}📄 CodeRabbit inference engine (AGENTS.md)
Files:
**/*.{js,ts,jsx,tsx}📄 CodeRabbit inference engine (AGENTS.md)
Files:
**/*.{test,spec}.{js,ts,jsx,tsx}📄 CodeRabbit inference engine (AGENTS.md)
Files:
**/index.{js,ts,jsx,tsx}📄 CodeRabbit inference engine (AGENTS.md)
Files:
🔇 Additional comments (42)
📝 WalkthroughSummary by CodeRabbitRelease Notes
WalkthroughThis pull request implements a comprehensive typed transformer lifecycle hook system enabling extensibility across Leadtype's documentation pipeline. The change spans from foundational types and execution utilities through integration into MDX conversion, source loading, search indexing, and LLM artifact generation, with full TypeScript generics support for custom frontmatter schemas. ChangesTransformer Lifecycle Hooks
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
ESLint skipped: no ESLint configuration detected in root package.json. To enable, add Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: dc1a5b1ead
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![pullfrog[bot]](https://avatars.githubusercontent.com/in/1768019?s=60&v=4){kind=small}
There was a problem hiding this comment.
Note
Mergeable as-is. The transformer pipeline lands cleanly with consistent typing across convertMdxFile, createDocsSource, search, and LLM artifacts. A few minor rough edges flagged below as informational — none block merge.
TL;DR — Custom frontmatter becomes a first-class typed surface and a DocsTransformer lifecycle is threaded through conversion, source loading, search indexing, LLM artifacts, and CLI generation. Adds the contract module, regression coverage, and reference docs.
Key changes
- Add
DocsTransformercontract and runner — newtransformers.tsdefines hook enum, error wrapper, sync/async runners, andvalidateFrontmatter. - Generic frontmatter in
convertMdxFile— pipeline runsbeforeParse→ frontmatter resolution →afterFrontmatter(re-parses MDX when content changes) →afterMdxAst→afterFlattenMarkdown, with schema validation after every data-touching hook. - Generic
createDocsSource—DocsPageMeta,DocsPage, and the source factory carryTFrontmatter;readPageMetahonorsbeforeParse/afterFrontmatter, andbuildSearchIndexforwards transformers. - Search hooks —
createDocsSearchIndexacceptstransformers, appliesbeforeSearchIndexover the document list andbeforeSearchChunkper chunk, recomputing chunk length from the post-hook text. - LLM + agent artifacts —
beforeLlmsTxt(root and docs-scoped),beforeLlmsFull, andbeforeAgentsMdartifacts run before write;DocsFrontmatterSchemais re-exported from the transformers module. - CLI wiring —
runGenerateCommandreadsfrontmatterSchema+transformersfrom the loaded config and threads them to convert plus every LLM / search / agent generator across default and i18n locales;needsGroupInferencenow also fires when explicit path filters strip nav. - Docs + tests — new
docs/reference/frontmatter-transformers.mdx, updates tosource.mdx, and tests for convert (success + error wrapping), source (typed frontmatter), search (index + chunk), and llm (root llms.txt).
Summary ｜ 17 files ｜ 1 commit ｜ base: main ← KayleeWilliams/46
Pipeline ordering and validation cadence
Before:
convertMdxFilereturned{ data: Record<string, unknown>, ... }with no hook points and no schema validation.
After: Generic onTFrontmatter, validates after each data-mutating hook, and re-serializes frontmatter viastringifyFrontmatterwhenever a schema or transformers are present.
Hooks run in array order; an undefined return preserves the current value; throws are wrapped with transformer name + hook + file path in DocsTransformerError. When afterFrontmatter mutates content, the MDX is re-parsed and re-processed before afterMdxAst sees it.
packages/leadtype/src/transformers.ts · packages/leadtype/src/convert/convert.ts
Source: typed metadata, double-evaluation note
Before:
DocsPageMetaexposedgroups, nofrontmatter;DocsPagehadfrontmatter: Record<string, unknown>.
After: Both carryfrontmatter: TFrontmatter;listPagesandloadPageroute throughvalidateFrontmatter.
Worth noting (informational, no fix needed): beforeParse and afterFrontmatter execute once inside readPageMeta (driving listPages / buildSearchIndex) and again inside convertMdxFile when loadPage is invoked. Pure transformers are idempotent so behavior is correct, but expensive hooks pay the cost twice per page.
packages/leadtype/src/source/index.ts
Search hooks and synchronous contract
Before: Chunk metadata was assembled and pushed directly; index construction had no extension points.
After:beforeSearchIndexandbeforeSearchChunkinterpose;runTransformersSyncthrows if a hook returns a Promise so the synchronous indexing path stays safe.
Chunk length is recomputed from the post-hook text, matching the docs guarantee in frontmatter-transformers.mdx. The hook type signature is MaybePromise<...>; the sync-only requirement is enforced at runtime rather than at compile time, which is reasonable given the shared DocsTransformer shape.
packages/leadtype/src/search/search.ts
LLM, agents, and CLI threading
Before: Artifacts were written directly; CLI passed no schema or transformer wiring through to artifact generators.
After: Each artifact runs throughbeforeLlmsTxt/beforeLlmsFull/beforeAgentsMdbefore write;runGenerateCommandforwardsfrontmatterSchema+transformersto convert and every generator across the default and i18n locales.
The needsGroupInference widening (firing when explicit path filters reduce a config to nav-only) is an orthogonal fix called out in the inline comment; behavior is documented and bounded to filtered runs.
packages/leadtype/src/llm/llm.ts · packages/leadtype/src/cli/generate.ts
Claude Opus ｜ 𝕏
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/reference/frontmatter-transformers.mdx`:
- Around line 35-37: Replace the OS-dependent file path check using
page.filePath with a route-relative field: update the apiArea assignment that
currently uses page.filePath.includes("/reference/") to use page.urlPath (or
page.relativePath) for deterministic matching — e.g. check
page.urlPath.startsWith("/reference") or
page.relativePath.startsWith("reference") to decide between "reference" and
"guides", keeping the existing apiArea logic and any surrounding transform in
place.
In `@docs/reference/source.mdx`:
- Around line 42-43: Update the docs to narrow the scope of the `transformers`
description: in the table row referencing `transformers` / `DocsTransformer[]`
(and the `createDocsSource` context), remove or replace "agent artifacts" and
explicitly state that transformers provide lifecycle hooks for frontmatter,
markdown, and search processing only; ensure references to `createDocsSource`
and the `DocsTransformer` concept remain so readers can locate the API and its
relevant hook types.
In `@packages/leadtype/src/cli/generate.ts`:
- Around line 857-865: The config validation is dropping frontmatterSchema and
transformers so config-driven runs lose schema/transformer wiring; update
validateDocsConfig() to preserve and return loaded.config.frontmatterSchema and
loaded.config.transformers (or merge them into the validated config object) so
that the values used in generate.ts (references: frontmatterSchema,
transformers, loaded.config, and validateDocsConfig()) are threaded through the
validation step and available to the caller.
In `@packages/leadtype/src/convert/index.ts`:
- Around line 1-9: Remove the aggregated re-export block in this index.ts and
instead export each symbol from its original module file: export DocsAstPage,
DocsFrontmatterPage, DocsMarkdownPage, DocsRawPage, DocsTransformContext, and
DocsTransformer from the module that defines the types (e.g.,
"../transformers/types"), and export DocsTransformerError from the module that
defines the error (e.g., "../transformers/error"); do not re-export everything
via the barrel "../transformers" index. Ensure each export uses the explicit
source module path and that the index.ts no longer expands the barrel
entrypoint.
In `@packages/leadtype/src/llm/llm.ts`:
- Around line 1650-1655: The hook context always passes a hard-coded
relativePath "docs/llms.txt", which is incorrect for localized outputs; update
the call to runTransformers (the call site using config.transformers,
"beforeLlmsTxt", input, { stage: "llm", relativePath: "docs/llms.txt", locale },
...) to compute a locale-aware relativePath (e.g. include locale when present so
non-default locales use "docs/<locale>/llms.txt" while default keeps
"docs/llms.txt") so transformer error contexts reflect the actual output path.
In `@packages/leadtype/src/search/index.ts`:
- Around line 1-6: Remove the new re-exports (DocsSearchChunkInput,
DocsTransformContext, DocsTransformer, DocsTransformerError) from the search
barrel export in index.ts and instead import these symbols directly from the
module that defines them (the transformers module) at call sites; locate the
export block that lists type DocsSearchChunkInput, type DocsTransformContext,
type DocsTransformer, and DocsTransformerError and delete those lines so the
barrel no longer surfaces those transformer symbols, then update any files that
currently import them from the search index to import from the transformers
module instead.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: ef2ff294-c6d7-410d-bb77-bb79d0f57e86
📒 Files selected for processing (17)
docs/reference/frontmatter-transformers.mdxdocs/reference/source.mdxpackages/leadtype/src/cli/generate.tspackages/leadtype/src/convert/convert.test.tspackages/leadtype/src/convert/convert.tspackages/leadtype/src/convert/index.tspackages/leadtype/src/index.tspackages/leadtype/src/llm/index.tspackages/leadtype/src/llm/llm.test.tspackages/leadtype/src/llm/llm.tspackages/leadtype/src/search/index.tspackages/leadtype/src/search/node.tspackages/leadtype/src/search/search.test.tspackages/leadtype/src/search/search.tspackages/leadtype/src/source/index.tspackages/leadtype/src/source/source.test.tspackages/leadtype/src/transformers.ts
📜 Review details
🧰 Additional context used
📓 Path-based instructions (4)
**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.{ts,tsx}: Use explicit types for function parameters and return values when they enhance clarity
Preferunknownoveranywhen the type is genuinely unknown
Use const assertions (as const) for immutable values and literal types
Leverage TypeScript's type narrowing instead of type assertions
Files:
packages/leadtype/src/convert/index.tspackages/leadtype/src/search/index.tspackages/leadtype/src/search/search.test.tspackages/leadtype/src/llm/index.tspackages/leadtype/src/llm/llm.test.tspackages/leadtype/src/convert/convert.test.tspackages/leadtype/src/index.tspackages/leadtype/src/source/source.test.tspackages/leadtype/src/search/node.tspackages/leadtype/src/search/search.tspackages/leadtype/src/cli/generate.tspackages/leadtype/src/transformers.tspackages/leadtype/src/convert/convert.tspackages/leadtype/src/llm/llm.tspackages/leadtype/src/source/index.ts
**/*.{js,ts,jsx,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.{js,ts,jsx,tsx}: Use meaningful variable names instead of magic numbers - extract constants with descriptive names
Use arrow functions for callbacks and short functions
Preferfor...ofloops over.forEach()and indexedforloops
Use optional chaining (?.) and nullish coalescing (??) for safer property access
Prefer template literals over string concatenation
Use destructuring for object and array assignments
Useconstby default,letonly when reassignment is needed, nevervar
Alwaysawaitpromises in async functions - don't forget to use the return value
Useasync/awaitsyntax instead of promise chains for better readability
Handle errors appropriately in async code with try-catch blocks
Don't use async functions as Promise executors
Removeconsole.log,debugger, andalertstatements from production code
ThrowErrorobjects with descriptive messages, not strings or other values
Usetry-catchblocks meaningfully - don't catch errors just to rethrow them
Prefer early returns over nested conditionals for error cases
Extract complex conditions into well-named boolean variables
Use early returns to reduce nesting
Prefer simple conditionals over nested ternary operators
Don't useeval()or assign directly todocument.cookie
Avoid spread syntax in accumulators within loops
Use top-level regex literals instead of creating them in loops
Prefer specific imports over namespace imports
Use descriptive names for functions, variables, and types for meaningful naming
Add comments for complex logic, but prefer self-documenting code
Files:
packages/leadtype/src/convert/index.tspackages/leadtype/src/search/index.tspackages/leadtype/src/search/search.test.tspackages/leadtype/src/llm/index.tspackages/leadtype/src/llm/llm.test.tspackages/leadtype/src/convert/convert.test.tspackages/leadtype/src/index.tspackages/leadtype/src/source/source.test.tspackages/leadtype/src/search/node.tspackages/leadtype/src/search/search.tspackages/leadtype/src/cli/generate.tspackages/leadtype/src/transformers.tspackages/leadtype/src/convert/convert.tspackages/leadtype/src/llm/llm.tspackages/leadtype/src/source/index.ts
**/index.{js,ts,jsx,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Avoid barrel files (index files that re-export everything)
Files:
packages/leadtype/src/convert/index.tspackages/leadtype/src/search/index.tspackages/leadtype/src/llm/index.tspackages/leadtype/src/index.tspackages/leadtype/src/source/index.ts
**/*.{test,spec}.{js,ts,jsx,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.{test,spec}.{js,ts,jsx,tsx}: Write assertions insideit()ortest()blocks
Avoid done callbacks in async tests - use async/await instead
Don't use.onlyor.skipin committed code
Keep test suites reasonably flat - avoid excessivedescribenesting
Files:
packages/leadtype/src/search/search.test.tspackages/leadtype/src/llm/llm.test.tspackages/leadtype/src/convert/convert.test.tspackages/leadtype/src/source/source.test.ts
🔇 Additional comments (54)
packages/leadtype/src/transformers.ts (10)
1-21: LGTM!
23-55: LGTM!
57-94: LGTM!
96-114: LGTM!
116-166: LGTM!
168-185: LGTM!
187-211: LGTM!
213-245: LGTM!
247-294: LGTM!
296-317: LGTM!packages/leadtype/src/convert/convert.ts (12)
241-248: LGTM!
272-277: LGTM!
354-365: LGTM!
378-401: LGTM!
414-425: LGTM!
437-478: LGTM!
480-505: LGTM!
507-543: LGTM!
549-570: LGTM!
598-624: LGTM!
685-691: LGTM!
747-756: LGTM!packages/leadtype/src/convert/convert.test.ts (4)
5-5: LGTM!
104-134: LGTM!
136-158: LGTM!
160-177: LGTM!packages/leadtype/src/source/index.ts (12)
63-71: LGTM!
73-98: LGTM!
100-110: LGTM!
112-153: LGTM!
155-179: LGTM!
221-306: LGTM!
414-418: LGTM!
443-445: LGTM!
462-479: LGTM!
487-508: LGTM!Also applies to: 512-518
535-537: LGTM!Also applies to: 539-575
597-640: LGTM!packages/leadtype/src/source/source.test.ts (2)
4-4: LGTM!
79-112: LGTM!packages/leadtype/src/search/search.ts (5)
3-8: LGTM!
120-120: LGTM!
210-210: LGTM!
855-864: LGTM!
915-966: LGTM!packages/leadtype/src/search/search.test.ts (1)
136-166: LGTM!packages/leadtype/src/search/node.ts (3)
24-24: LGTM!Also applies to: 50-50
140-140: LGTM!
310-313: LGTM!packages/leadtype/src/llm/llm.ts (1)
36-41: LGTM!Also applies to: 175-175, 226-233, 265-267, 292-292, 306-306, 320-320, 1920-1920
packages/leadtype/src/llm/llm.test.ts (1)
73-108: LGTM!packages/leadtype/src/cli/generate.ts (1)
25-25: LGTM!Also applies to: 49-49, 175-180, 1424-1435, 1477-1478, 1491-1605
packages/leadtype/src/index.ts (1)
53-66: Duplicate concern.Same barrel-file expansion concern already raised for
packages/leadtype/src/convert/index.ts.packages/leadtype/src/llm/index.ts (1)
17-24: Duplicate concern.Same barrel-file expansion concern already raised for
packages/leadtype/src/convert/index.ts.
There was a problem hiding this comment.
No new issues. Reviewed the following changes:
- Exposed
leadtype/transformersas a first-class subpath inpackage.json,rollup.config.ts, and the package-surface test, and pruned the transformer re-exports from theconvert,search, andllmbarrels so the types are sourced from one entry point. - Threaded
frontmatterSchemaandtransformersthroughvalidateDocsConfig, so config-driven runs no longer drop them before reaching the generators. - Extracted
prepareMdxConversioninconvert.tsand exportedresolveMdxFrontmatter; routedreadPageMetathrough it so synthesized frontmatter now flows through the same schema/transformer pipeline as conversion (covered by the newvalidates synthesized frontmatter through custom source schemastest). - Computed a locale-aware
docsLlmsRelativePathfor thebeforeLlmsTxthook so non-default locales reportdocs/<locale>/llms.txtin the transformer context. - Updated the
frontmatter-transformers.mdxexample to deriveapiAreafromcontext.relativePath(OS-independent) and narrowed thetransformersdescription insource.mdxto frontmatter/markdown/search.
Claude Opus ｜ 𝕏
1 participant
Summary
Closes #46.
Validation
bun x ultracite checkbun run --filter leadtype check-typesbun run --filter leadtype buildbun run --filter leadtype test