feat: generate rules files for LLM coding [sc-00] #1096
Conversation
WalkthroughThis update introduces a new CLI command class to generate a rules file with embedded example code snippets, adds a comprehensive markdown template string detailing Checkly CLI usage and guidelines with placeholders for examples, and includes a JSON fixture file containing sample resource definitions for monitoring checks, alert channels, and subscriptions. Changes
📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
⏰ Context from checks skipped due to timeout of 90000ms (2)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
packages/cli/src/commands/rules.ts(1 hunks)packages/cli/src/rules/checkly.rules.ts(1 hunks)packages/cli/src/rules/rules.fixtures.json(1 hunks)
🧰 Additional context used
🧠 Learnings (1)
packages/cli/src/rules/rules.fixtures.json (1)
Learnt from: sorccu
PR: checkly/checkly-cli#1082
File: packages/create-cli/src/loader/ts-node.ts:0-0
Timestamp: 2025-06-10T15:36:26.431Z
Learning: In the checkly-cli create-cli package, only single instances of UninitializedTSNodeFileLoaderState and UninitializedJitiFileLoaderState are created, which eliminates race condition concerns in their current usage pattern.
🧬 Code Graph Analysis (1)
packages/cli/src/commands/rules.ts (1)
packages/cli/src/rules/checkly.rules.ts (1)
checklyRulesTemplate(1-95)
🪛 Gitleaks (8.26.0)
packages/cli/src/rules/rules.fixtures.json
588-588: Discovered a Slack Webhook, which could lead to unauthorized message posting and data leakage in Slack channels.
(slack-webhook-url)
🪛 Checkov (3.2.334)
packages/cli/src/rules/rules.fixtures.json
[MEDIUM] 588-589: Slack Token
(CKV_SECRET_14)
⏰ Context from checks skipped due to timeout of 90000ms (2)
- GitHub Check: test - windows-latest
- GitHub Check: test - ubuntu-latest
🔇 Additional comments (10)
packages/cli/src/rules/checkly.rules.ts (2)
1-95: Well-structured template with comprehensive documentation.The template provides excellent coverage of Checkly CLI usage patterns and constructs. The markdown formatting is clean and the placeholder syntax is consistent across different check types.
7-7: Consider escaping backticks in template string.The template contains unescaped backticks in the documentation which might cause issues when processed. However, I see that the Rules command handles this with
unescapeBackticks()method, so this is correctly addressed.packages/cli/src/commands/rules.ts (8)
1-3: Clean imports and proper structure.The imports are correctly structured and the command extends the appropriate base class.
8-23: Well-implemented main command logic with proper error handling.The
runmethod properly processes the template through the pipeline of unescaping backticks and replacing placeholders. Error handling covers the template processing step appropriately.
25-27: Simple and effective backtick unescaping.The regex replacement correctly handles escaped backticks in the template string.
29-43: Robust placeholder replacement logic.The method systematically replaces all placeholders with their corresponding examples. The mapping approach is clean and maintainable.
45-78: Comprehensive browser check example with realistic Playwright code.The example demonstrates proper usage of BrowserCheck construct with realistic configuration and includes a separate Playwright test file example. The environment variable usage follows the documented handlebar syntax.
80-128: Excellent multi-step check example with practical user journey.The example shows a realistic user journey test with multiple steps, proper environment variable usage, and demonstrates both the construct definition and the accompanying Playwright test code.
130-149: Accurate TCP check example with proper assertion usage.The example correctly demonstrates TcpCheck usage with TcpAssertionBuilder for response time assertions, following the documented patterns.
151-166: Clear heartbeat check example with appropriate configuration.The example shows proper HeartbeatCheck usage with realistic timing configuration for monitoring cron jobs.
| "name": "tim's test channel", | ||
| "method": "POST", | ||
| "headers": [], | ||
| "template": "chat_id=1352727080&parse_mode=HTML&text=<b>{{ALERT_TITLE}}</b> at {{STARTED_AT}} in {{RUN_LOCATION}} ({{RESPONSE_TIME}}ms)\nTags: {{#each TAGS}} <i><b>{{this}}</b></i> {{#unless @last}},{{/unless}} {{/each}}\n<a href=\"{{RESULT_LINK}}\">View check result</a>\n", |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Replace chat ID with placeholder.
The Telegram template contains a specific chat ID that should be parameterized.
Apply this diff:
- "template": "chat_id=1352727080&parse_mode=HTML&text=<b>{{ALERT_TITLE}}</b> at {{STARTED_AT}} in {{RUN_LOCATION}} ({{RESPONSE_TIME}}ms)\nTags: {{#each TAGS}} <i><b>{{this}}</b></i> {{#unless @last}},{{/unless}} {{/each}}\n<a href=\"{{RESULT_LINK}}\">View check result</a>\n",
+ "template": "chat_id={{TELEGRAM_CHAT_ID}}&parse_mode=HTML&text=<b>{{ALERT_TITLE}}</b> at {{STARTED_AT}} in {{RUN_LOCATION}} ({{RESPONSE_TIME}}ms)\nTags: {{#each TAGS}} <i><b>{{this}}</b></i> {{#unless @last}},{{/unless}} {{/each}}\n<a href=\"{{RESULT_LINK}}\">View check result</a>\n",📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| "template": "chat_id=1352727080&parse_mode=HTML&text=<b>{{ALERT_TITLE}}</b> at {{STARTED_AT}} in {{RUN_LOCATION}} ({{RESPONSE_TIME}}ms)\nTags: {{#each TAGS}} <i><b>{{this}}</b></i> {{#unless @last}},{{/unless}} {{/each}}\n<a href=\"{{RESULT_LINK}}\">View check result</a>\n", | |
| "template": "chat_id={{TELEGRAM_CHAT_ID}}&parse_mode=HTML&text=<b>{{ALERT_TITLE}}</b> at {{STARTED_AT}} in {{RUN_LOCATION}} ({{RESPONSE_TIME}}ms)\nTags: {{#each TAGS}} <i><b>{{this}}</b></i> {{#unless @last}},{{/unless}} {{/each}}\n<a href=\"{{RESULT_LINK}}\">View check result</a>\n", |
🤖 Prompt for AI Agents
In packages/cli/src/rules/rules.fixtures.json at line 518, the Telegram template
contains a hardcoded chat ID. Replace the specific chat ID value with a
placeholder variable to allow parameterization. Update the template string so
that the chat ID is represented by a variable like {{CHAT_ID}} instead of the
fixed number.
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (2)
packages/cli/src/rules/rules.fixtures.json (2)
514-521: Leaking real Telegram bot token & chat ID – replace with placeholdersThe fixture still exposes what looks like a production Telegram bot token and a hard-coded chat ID. These are secrets that allow anyone with this file to post messages on your behalf. Replace both with clear placeholders.
- "url": "https://api.telegram.org/bot5692603774:AAGDbMPNOttAngeWneFnQJIz1xwbQo4UMx0/sendMessage", + "url": "https://api.telegram.org/bot{{TELEGRAM_BOT_TOKEN}}/sendMessage", @@ - "template": "chat_id=1352727080&parse_mode=HTML&text=<b>{{ALERT_TITLE}}</b> at {{STARTED_AT}} in {{RUN_LOCATION}} ({{RESPONSE_TIME}}ms)\nTags: {{#each TAGS}} <i><b>{{this}}</b></i> {{#unless @last}},{{/unless}} {{/each}}\n<a href=\"{{RESULT_LINK}}\">View check result</a>\n", + "template": "chat_id={{TELEGRAM_CHAT_ID}}&parse_mode=HTML&text=<b>{{ALERT_TITLE}}</b> at {{STARTED_AT}} in {{RUN_LOCATION}} ({{RESPONSE_TIME}}ms)\nTags: {{#each TAGS}} <i><b>{{this}}</b></i> {{#unless @last}},{{/unless}} {{/each}}\n<a href=\"{{RESULT_LINK}}\">View check result</a>\n",
588-595: Slack webhook URL exposes live token – must be sanitizedA Slack Incoming-Webhook URL (including the token) is committed in plain text. This is a credential that can immediately be abused. Swap it for a placeholder before merging.
- "url": "https://hooks.slack.com/services/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", + "url": "https://hooks.slack.com/services/{{SLACK_WEBHOOK_URL}}",
🧹 Nitpick comments (1)
packages/cli/src/rules/rules.fixtures.json (1)
1-626: Automate secrets scrubbing for future fixture generationTwo credential leaks re-appeared after previous reviews, indicating the sanitisation step is still manual and error-prone. Consider extending the
npx checkly rulesgenerator with a post-processing script (e.g. usingjqor a lightweight Node transform) that automatically:
- Replaces URL tokens matching common webhook/token patterns.
- Parameterises any numeric IDs that are not required for rule semantics.
- Flags any keys that still contain plausible secrets (regex + deny-list).
Automating this eliminates the risk of credential exposure going forward and removes review overhead.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
packages/cli/src/rules/rules.fixtures.json(1 hunks)
🧰 Additional context used
🧠 Learnings (1)
packages/cli/src/rules/rules.fixtures.json (1)
Learnt from: sorccu
PR: checkly/checkly-cli#1082
File: packages/create-cli/src/loader/ts-node.ts:0-0
Timestamp: 2025-06-10T15:36:26.431Z
Learning: In the checkly-cli create-cli package, only single instances of UninitializedTSNodeFileLoaderState and UninitializedJitiFileLoaderState are created, which eliminates race condition concerns in their current usage pattern.
🪛 Gitleaks (8.26.0)
packages/cli/src/rules/rules.fixtures.json
588-588: Discovered a Slack Webhook, which could lead to unauthorized message posting and data leakage in Slack channels.
(slack-webhook-url)
🪛 Checkov (3.2.334)
packages/cli/src/rules/rules.fixtures.json
[MEDIUM] 588-589: Slack Token
(CKV_SECRET_14)
⏰ Context from checks skipped due to timeout of 90000ms (3)
- GitHub Check: test - ubuntu-latest
- GitHub Check: test - windows-latest
- GitHub Check: lint
🔇 Additional comments (1)
packages/cli/src/rules/rules.fixtures.json (1)
490-492: Phone number correctly sanitized – nice catch
The number is now a placeholder; thanks for addressing the earlier feedback.
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
aluedeke
force-pushed
the
tnolet/generate-rules-file-sc00
branch
from
b170d73 to
33a9269
Compare
aluedeke
force-pushed
the
tnolet/generate-rules-file-sc00
branch
from
33a9269 to
4a39bd6
Compare
Affected Components
Goal
rulesfile for LLM code assistant tools with canonical examples of all relevant constructs.npx checkly rulescommand.Hints
npx checkly import plan --debug-import-plan --previewcommand see feat(internal): utility to generate code for an import plan loaded from a file [sc-24645] #1095) by pointing it to a representative prod, staging or dev account.importlogic to the placeholders in thecheckly.rules.tsmarkdown file.After this is done, we can change the rewrite
https://www.checklyhq.com/docs/ai/checkly.rules.mdand point it to this file and update the docs.