feat(slack): Slack mrkdwn output contract in <output> prompt section
#212
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| import { describe } from "vitest"; | ||
| import { mention, rubric, slackEval } from "../helpers"; | ||
|
|
||
| describe("Slack mrkdwn hygiene", () => { | ||
| slackEval( | ||
| "uses single-asterisk bold, single-tilde strike, and Slack link syntax", | ||
| { | ||
| events: [ | ||
| mention( | ||
| "In one short Slack reply, bold the word 'ready', strike through the word 'draft', and link the label 'docs' to https://docs.slack.dev/ .", | ||
| ), | ||
| ], | ||
| overrides: { | ||
| reply_timeout_ms: 120_000, | ||
| }, | ||
| requireSandboxReady: false, | ||
| taskTimeout: 150_000, | ||
| timeout: 210_000, | ||
| criteria: rubric({ | ||
| contract: | ||
| "Emphasis and link syntax follow Slack `mrkdwn`: single-asterisk bold, single-tilde strike, and `<url|label>` links. CommonMark/GFM equivalents are forbidden.", | ||
| pass: [ | ||
| "assistant_posts contains a single reply that addresses the bold, strike, and link asks.", | ||
| "Bold uses `*ready*` (single asterisks).", | ||
| "Strike uses `~draft~` (single tildes).", | ||
| "The docs link appears as `<https://docs.slack.dev/|docs>` or the bare URL.", | ||
| ], | ||
| fail: [ | ||
| "Do not emit `**ready**` (CommonMark bold).", | ||
| "Do not emit `~~draft~~` (CommonMark strike).", | ||
| "Do not emit `[docs](https://docs.slack.dev/)` (CommonMark link).", | ||
| ], | ||
| }), | ||
| }, | ||
| ); | ||
|
|
||
| slackEval("uses bold section labels instead of markdown headings", { | ||
| events: [ | ||
| mention( | ||
| "Give me a two-section Slack reply with short headings 'Summary' and 'Next steps', each with one sentence under it.", | ||
| ), | ||
| ], | ||
| overrides: { | ||
| reply_timeout_ms: 120_000, | ||
| }, | ||
| requireSandboxReady: false, | ||
| taskTimeout: 150_000, | ||
| timeout: 210_000, | ||
| criteria: rubric({ | ||
| contract: | ||
| "Section structure uses a bold label on its own line. Markdown heading syntax is forbidden because Slack does not render it.", | ||
| pass: [ | ||
| "assistant_posts contains a single reply with two sections.", | ||
| "Each section label appears as `*Summary*` and `*Next steps*` on their own lines (bold labels), followed by a sentence.", | ||
| ], | ||
| fail: [ | ||
| "Do not emit `# Summary`, `## Summary`, `### Summary`, or any other markdown heading syntax.", | ||
| "Do not emit `**Summary**` (CommonMark bold).", | ||
| ], | ||
| }), | ||
| }); | ||
| }); | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| /** | ||
| * Slack Block Kit types used by the outbound reply boundary. This is a local | ||
| * subset of the Slack API surface — just the fields the repository actually | ||
| * emits when wrapping final `mrkdwn` replies in section/context envelopes. | ||
| */ | ||
|
|
||
| export interface SlackMrkdwnText { | ||
| text: string; | ||
| type: "mrkdwn"; | ||
| } | ||
|
|
||
| export interface SlackPlainText { | ||
| emoji?: boolean; | ||
| text: string; | ||
| type: "plain_text"; | ||
| } | ||
|
|
||
| export interface SlackHeaderBlock { | ||
| text: SlackPlainText; | ||
| type: "header"; | ||
| } | ||
|
|
||
| export interface SlackSectionBlock { | ||
| fields?: SlackMrkdwnText[]; | ||
| text?: SlackMrkdwnText; | ||
| type: "section"; | ||
| } | ||
|
|
||
| export interface SlackDividerBlock { | ||
| type: "divider"; | ||
| } | ||
|
|
||
| export interface SlackContextBlock { | ||
| elements: SlackMrkdwnText[]; | ||
| type: "context"; | ||
| } | ||
|
|
||
| export interface SlackLinkButtonElement { | ||
| text: SlackPlainText; | ||
| type: "button"; | ||
| url: string; | ||
| } | ||
|
|
||
| export interface SlackActionsBlock { | ||
| elements: SlackLinkButtonElement[]; | ||
| type: "actions"; | ||
| } | ||
|
|
||
| export type SlackMessageBlock = | ||
| | SlackActionsBlock | ||
| | SlackContextBlock | ||
| | SlackDividerBlock | ||
| | SlackHeaderBlock | ||
| | SlackSectionBlock; | ||
|
|
||
| /** Escape user-provided text for safe inclusion in Slack mrkdwn fields. */ | ||
| export function escapeSlackMrkdwnText(text: string): string { | ||
| return text | ||
| .replaceAll("&", "&") | ||
| .replaceAll("<", "<") | ||
| .replaceAll(">", ">"); | ||
| } | ||
|
There was a problem hiding this comment. Unused exported escape function duplicates existing oneLow Severity
Additional Locations (1)  Reviewed by Cursor Bugbot for commit 8b7cfc4. Configure here. |
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Missing pipe-table eval scenario described in PR
Medium Severity
The PR description states there are three eval scenarios, but only two are present. The missing first scenario — "Give me a short comparison table…" that validates GFM pipe-table syntax is not emitted — is absent. This is the primary regression the PR aims to fix ("the real regression we kept hitting in dev (e.g. GFM pipe-tables in comparison replies)"), and the Review & Testing Checklist instructs reviewers to "Run the three new evals," yet only two exist.
Reviewed by Cursor Bugbot for commit 8b7cfc4. Configure here.