feat: refactor rules generation to ai-context with skill output

[stefanjudis]

Affected Components

• CLI
• Create CLI
• Test
• Docs
• Examples
• Other

Notes for the Reviewer

This PR changes how Checkly AI context files (aka rules.md) are generated to align with upcoming skill conventions in the AI ecosystem. The primary concern is to provide a SKILL.md for MaC and be compatible with Vercel's new skills tooling:

npx skills add checkly/checkly-cli
npx skills add checkly/checkly-cli --skill monitoring

For that to work, the repo needs to include a skills directory in the root, which contains all the provided markdown skills (skills/monitoring/SKILL.md). For that, I repurposed the script of the checkly.rules.md and now create two different context files.

/package/cli/dist/ui-context/skills is then copied over to the root directory via prepare:ai-context to avoid that the cli package rights files outside its scope. The new file needs to be reviewed and commited (otherwise a new GH action fails).

---

Generally, I changed the naming convention from rules to ai-context because the script now generates our "rules" (rules.md) and a Checkly "monitoring" skill (skills/monitoring/SKILL.md). The content is more or less the same, though. I've chosen the monitoring skill name to be flexible in creating more skills (e.g. a playwright-best-practices skill) in the future. We'll see how this plays out.

Additionally, I extracted the configuration code from the actual script into its own file so that there's a single config file.

Content-wise, I expect multiple iterations on the new "context" files, but I still made some adjustments on the way:

• checkly.config code is now part of the generated context files. To do so, the code config now supports inline code samples bc checkly.config and pwCheckSuites aren't part of the exported files used for the existing code generation.

Resolves Slack conversation and MAR-504

@sorccu

And I discussed the best approach and landed on a GH workflow that generates the skills files and checks if there are uncommitted changes. This way the team needs to review and commit changes manually and we also have a sanity check in each PR.

To check later

We need to make sure that the configuration in the marketing site still works and provides the generated file from GitHub.

"source": "/docs/ai/checkly.rules.md",
"destination": "https://github.com/checkly/checkly-cli/releases/latest/download/checkly.rules.md"

[Copilot]

Pull request overview

Refactors how Checkly AI context (“rules.md”) content is generated so it can also produce a Vercel-compatible SKILL.md under a root-level skills/ directory.

Changes:

• Move rules/context sources to src/ai-context/ and update the rules command to read from dist/ai-context/checkly.rules.md.
• Replace the old compile-rules.ts pipeline with compile-ai-context.ts, generating both dist/ai-context/checkly.rules.md (without frontmatter) and skills/monitoring/SKILL.md (with frontmatter).
• Add a centralized EXAMPLE_CONFIGS configuration and new fixtures for the generation flow.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
packages/cli/src/commands/rules.ts	Updates base rules path to new dist/ai-context output location.
packages/cli/src/ai-context/context.ts	Adds centralized example/placeholder config (including inline code samples + doc references).
packages/cli/src/ai-context/context.fixtures.json	Adds fixtures used to generate example code via import plan.
packages/cli/src/ai-context/checkly.context.template.md	Introduces skill-style frontmatter and adjusts template to rely on injected examples.
packages/cli/scripts/compile-rules.ts	Removes the old rules compiler script.
packages/cli/scripts/compile-ai-context.ts	Adds new compiler that outputs both SKILL.md and stripped checkly.rules.md.
packages/cli/package.json	Renames generation script and wires it into prepare.

Comments suppressed due to low confidence (3)

packages/cli/src/ai-context/checkly.context.template.md:108

• There’s still a hard-coded “Reference [the docs…]” link here, while other sections moved references into the generated example blocks. For consistency (and to avoid duplicated/competing references), consider removing this link and relying on the per-example reference injected by the compiler.
  packages/cli/src/ai-context/checkly.context.template.md:132
• Grammar: “All alert are described…” should be “All alerts are described…”. Since this template content is distributed to users/AI tooling, it’s worth fixing the wording.
  packages/cli/src/ai-context/checkly.context.template.md:79
• This line reads awkwardly/incorrectly (“if your tasked”). Please fix the grammar (e.g., “Use pwProjects if you’re tasked with reusing a Playwright project.”) since this text is shipped in generated AI context.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

⚠️ AI context is out of date

The skills/ directory doesn't match the generated output from packages/cli.

Please run the following locally and commit the changes:

npm run prepare

[sorccu]

I forgot that we also needed to change the workflows: #1217