Skip to content

feat(landing): publish agent markdown bundles#274

Merged
lentil32 merged 4 commits into
mainfrom
feat/landing-agent-markdown-bundles
Jun 4, 2026
Merged

feat(landing): publish agent markdown bundles#274
lentil32 merged 4 commits into
mainfrom
feat/landing-agent-markdown-bundles

Conversation

@lentil32
Copy link
Copy Markdown
Contributor

@lentil32 lentil32 commented Jun 4, 2026

One-Line Summary

Publish agent-readable OneQuery Markdown bundles so agents can ingest focused or full public site context without crawling page chrome.

User-Facing Changes

  • Adds /llms.txt as an index for OneQuery agent-readable Markdown exports.
  • Adds /llms-small.txt for focused product, setup, source API, and safety context.
  • Adds /llms-full.txt for broader marketing, connector, comparison, blog, and documentation context.
  • Keeps landing page code and terminal snippets on the Tokyo Night Shiki theme through explicit theme objects.

Why This Changed

The existing agent Markdown integration could expose individual content and HTML-derived page sidecars, but it did not produce canonical bundle files for LLM ingestion. That made agent context depend on discovering many separate URLs and could include page chrome instead of the primary page content.

The landing home route also needs a curated Markdown source for bundle generation because the Cloudflare build currently emits a shell-only HTML asset for that route.

How It Changed

  • Adds exportAgentMarkdownBundles to collect generated Markdown route files, filter pages by document set, apply promote/demote ordering, validate output URLs, and emit bundle documents plus an optional index.
  • Extends the agentMarkdown Astro integration with bundle options, dev placeholder routes, build-time bundle export logging, and dev server dependency pre-optimization for YAML.
  • Configures the landing site to generate full and small OneQuery bundle documents, plus compare content Markdown routes.
  • Adds a curated prerendered index.md route for the landing home page so bundle output has stable product context.
  • Updates HTML-to-Markdown conversion to prefer <main> content over page chrome and drop unreadable links.
  • Exports the new bundle option types from @onequery/astro-agent-markdown.

Bug Fixes

  • When HTML-derived Markdown was generated from full pages, header and footer content could be selected ahead of the main article content; conversion now prefers <main>.
  • During dev SSR, YAML could be discovered mid-request by workerd through the generated Markdown middleware; the integration now pre-optimizes it in dev.
  • Astro component-level <Code /> usage did not inherit the global Markdown Shiki config, so home page snippets now pass the imported Tokyo Night theme object directly.

Performance Notes (Optional)

Bundle generation runs once at build completion and reuses the Markdown sidecar files already emitted for public routes. The generated bundle routes in dev are lightweight placeholders instead of rebuilding full bundle content per request.

Extra Context / Decisions (Optional)

The small bundle is intentionally focused on setup, connector, source API, and safety pages for shorter agent contexts. The full bundle keeps broader public pages for retrieval and long-context ingestion.

Verification

  • rtk mise exec -- bun run format
  • rtk mise exec -- bunx turbo test --json --filter=@onequery/astro-agent-markdown --filter=@onequery/landing
  • rtk mise exec -- bunx turbo typecheck --json --filter=@onequery/astro-agent-markdown --filter=@onequery/landing
  • rtk mise exec -- bunx turbo build --json --filter=@onequery/landing
  • Pre-push hook: turbo lint with type-aware lint completed with 0 warnings and 0 errors.

Bundle tests added in packages/astro-agent-markdown/src/bundle.test.ts cover route path normalization, generated bundle/index output, page ordering/excludes, and missing-match warnings. HTML conversion coverage now includes preferring main content over page chrome.

Video / Screenshot (Optional)

  • N/A

@cloudflare-workers-and-pages
Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
onequery-landing eb86959 Commit Preview URL

Branch Preview URL
Jun 04 2026, 07:35 AM

@lentil32 lentil32 merged commit f393700 into main Jun 4, 2026
9 checks passed
@lentil32 lentil32 deleted the feat/landing-agent-markdown-bundles branch June 4, 2026 07:48
lentil32 added a commit that referenced this pull request Jun 4, 2026
## One-Line Summary
Prevent Astro dev from hitting a Vite SSR optimizer reload crash after
the agent Markdown middleware loads YAML.

## User-Facing Changes
- Keeps `yaml` out of the server-side Vite dep optimizer for
`@onequery/astro-agent-markdown` dev middleware.
- Avoids the dev-server failure where Vite reports a missing
`.vite/deps_ssr/chunk-*.js` file after optimized dependencies reload.

## Why This Changed
PR #274 added build-time agent Markdown bundles and attempted to
pre-optimize `yaml` during dev SSR. In the Cloudflare dev runner, that
can cause Vite to rewrite the SSR optimized dependency cache during a
full reload while workerd is still resolving the previous chunk path.

The result is a local dev crash like:

`The file does not exist at ".../node_modules/.vite/deps_ssr/chunk-*.js"
which is in the optimize deps directory.`

## How It Changed
- Switches the integration's dev-only Vite environment plugin from
`optimizeDeps.include` to `optimizeDeps.exclude` for `yaml`.
- Keeps the exclusion scoped to non-client environments, so client
dependency optimization remains unchanged.
- Renames the plugin from a pre-optimization-specific name to a more
general dev dependency optimizer name.

## Bug Fixes
- Under `astro dev` with the Cloudflare adapter, loading the agent
Markdown middleware could trigger a full reload after `yaml`
optimization and leave the worker resolving a stale `deps_ssr` chunk.
This change keeps `yaml` as source in server environments to avoid that
optimizer invalidation path.

## Extra Context / Decisions (Optional)
The generated agent Markdown bundle behavior from PR #274 is unchanged.
This only adjusts dev-server dependency handling.

## Verification
- `rtk git merge origin/main`
- `rtk mise exec -- bun run format`
- `rtk mise exec -- bunx turbo test --json
--filter=@onequery/astro-agent-markdown`
- `rtk mise exec -- bunx turbo typecheck --json
--filter=@onequery/astro-agent-markdown`
- Pre-push hook: `turbo lint` with type-aware lint completed with 0
warnings and 0 errors.

Dev-server smoke validation was not completed in this pass after the
workflow was redirected to merge/push/PR.

## Video / Screenshot (Optional)
- N/A
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant