Skip to content

Move detection caution callout to top of agent failure issue/comment bodies #27896

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pelikhan merged 6 commits into from
Apr 22, 2026
Merged

Move detection caution callout to top of agent failure issue/comment bodies #27896

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
7120f36
Plan agent failure caution callout placement
Copilot Apr 22, 2026
571ec3b
Move detection caution alert to top of agent failure bodies
Copilot Apr 22, 2026
b2e49b6
Revert unrelated workflow lock permission changes
Copilot Apr 22, 2026
ded0466
Plan addressing PR review comments
Copilot Apr 22, 2026
b0cdfd2
Add coverage for caution placement in handle_agent_failure main flow
Copilot Apr 22, 2026
bea4c5d
Revert unrelated workflow lock file permission diffs
Copilot Apr 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 10 additions & 3 deletions actions/setup/js/handle_agent_failure.cjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

const { getErrorMessage } = require("./error_helpers.cjs");
const { sanitizeContent } = require("./sanitize_content.cjs");
const { getFooterAgentFailureIssueMessage, getFooterAgentFailureCommentMessage, generateXMLMarker } = require("./messages.cjs");
const { getDetectionCautionAlert, getFooterAgentFailureIssueMessage, getFooterAgentFailureCommentMessage, generateXMLMarker } = require("./messages.cjs");
const { renderTemplate, renderTemplateFromFile } = require("./messages_core.cjs");
const { getCurrentBranch } = require("./get_current_branch.cjs");
const { createExpirationLine, generateFooterWithExpiration } = require("./ephemerals.cjs");
Expand Down Expand Up @@ -1374,8 +1374,12 @@ async function main() {
};
const footer = getFooterAgentFailureCommentMessage(ctx);

// Prepend detection caution alert (when present) so it appears first in the comment body
const detectionCaution = getDetectionCautionAlert(workflowName, runUrl);
const fullCommentBodyRaw = detectionCaution ? `${detectionCaution}\n\n${commentBody}\n\n${footer}` : `${commentBody}\n\n${footer}`;

// Combine comment body with footer
const fullCommentBody = sanitizeContent(commentBody + "\n\n" + footer, { maxLength: 65000 });
const fullCommentBody = sanitizeContent(fullCommentBodyRaw, { maxLength: 65000 });

Comment on lines +1377 to 1383
Copy link

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new detection caution callout placement for agent-failure comments isn’t covered by tests in this PR. messages.test.cjs now asserts the footer does not contain the callout, but there’s no corresponding assertion that handle_agent_failure actually prepends the callout to the comment body (and that it appears before the rendered template text). Add a focused test in handle_agent_failure.test.cjs that sets GH_AW_DETECTION_CONCLUSION=warning and verifies the created comment body begins with the caution block and only includes it once.

This issue also appears on line 1546 of the same file.

Copilot uses AI. Check for mistakes.
await github.rest.issues.createComment({
owner,
Expand Down Expand Up @@ -1539,8 +1543,11 @@ async function main() {
suffix: `\n\n${generateXMLMarker(workflowName, runUrl)}`,
});

// Prepend detection caution alert (when present) so it appears first in the issue body
const detectionCaution = getDetectionCautionAlert(workflowName, runUrl);

// Combine issue body with footer
const bodyLines = [issueBodyContent, "", footerWithExpires];
const bodyLines = detectionCaution ? [detectionCaution, "", issueBodyContent, "", footerWithExpires] : [issueBodyContent, "", footerWithExpires];
const issueBody = bodyLines.join("\n");

const newIssue = await github.rest.issues.create({
Expand Down
125 changes: 124 additions & 1 deletion actions/setup/js/handle_agent_failure.test.cjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import { createRequire } from "module";
const require = createRequire(import.meta.url);

describe("handle_agent_failure", () => {
let main;
let buildCodePushFailureContext;
let buildPushRepoMemoryFailureContext;
let getActionFailureIssueExpiresHours;
Expand All @@ -25,7 +26,7 @@ describe("handle_agent_failure", () => {

// Reset module registry so each test gets a fresh require
vi.resetModules();
({ buildCodePushFailureContext, buildPushRepoMemoryFailureContext, getActionFailureIssueExpiresHours } = require("./handle_agent_failure.cjs"));
({ main, buildCodePushFailureContext, buildPushRepoMemoryFailureContext, getActionFailureIssueExpiresHours } = require("./handle_agent_failure.cjs"));
});

afterEach(() => {
Expand Down Expand Up @@ -54,6 +55,128 @@ describe("handle_agent_failure", () => {
});
});

describe("detection caution placement in main()", () => {
const fs = require("fs");
const path = require("path");
const os = require("os");

/** @type {string} */
let tmpDir;
/** @type {string} */
let promptsDir;

beforeEach(() => {
tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "aw-handle-agent-failure-"));
promptsDir = path.join(tmpDir, "gh-aw", "prompts");
fs.mkdirSync(promptsDir, { recursive: true });

// Minimal templates used by main()
fs.writeFileSync(path.join(promptsDir, "agent_failure_comment.md"), "COMMENT TEMPLATE CONTENT");
fs.writeFileSync(path.join(promptsDir, "agent_failure_issue.md"), "ISSUE TEMPLATE CONTENT");

process.env.RUNNER_TEMP = tmpDir;
process.env.GH_AW_WORKFLOW_NAME = "Test Workflow";
process.env.GH_AW_WORKFLOW_ID = "test-workflow";
process.env.GH_AW_RUN_URL = "https://github.com/owner/repo/actions/runs/123456";
process.env.GH_AW_AGENT_CONCLUSION = "failure";
process.env.GH_AW_DETECTION_CONCLUSION = "warning";
process.env.GH_AW_DETECTION_REASON = "threat_detected";
});

afterEach(() => {
delete process.env.RUNNER_TEMP;
delete process.env.GH_AW_WORKFLOW_NAME;
delete process.env.GH_AW_WORKFLOW_ID;
delete process.env.GH_AW_RUN_URL;
delete process.env.GH_AW_AGENT_CONCLUSION;
delete process.env.GH_AW_DETECTION_CONCLUSION;
delete process.env.GH_AW_DETECTION_REASON;

if (tmpDir && fs.existsSync(tmpDir)) {
fs.rmSync(tmpDir, { recursive: true, force: true });
}
});

it("prepends caution callout to existing-issue comment body and includes it only once", async () => {
/** @type {string} */
let capturedCommentBody = "";

global.github = {
rest: {
search: {
issuesAndPullRequests: vi.fn(async ({ q }) => {
if (q.includes("is:pr")) {
return { data: { total_count: 0, items: [] } };
}
return {
data: {
total_count: 1,
items: [{ number: 42, html_url: "https://github.com/owner/repo/issues/42" }],
},
};
}),
},
issues: {
createComment: vi.fn(async ({ body }) => {
capturedCommentBody = body;
return { data: { id: 1001 } };
}),
},
pulls: {
get: vi.fn(),
},
},
graphql: vi.fn(),
};

await main();

expect(capturedCommentBody).toBeTruthy();
expect(capturedCommentBody.startsWith("> [!CAUTION]")).toBe(true);
expect(capturedCommentBody.indexOf("> [!CAUTION]")).toBeLessThan(capturedCommentBody.indexOf("COMMENT TEMPLATE CONTENT"));
expect((capturedCommentBody.match(/> \[!CAUTION\]/g) || []).length).toBe(1);
expect(capturedCommentBody).toContain("> Generated from [Test Workflow]");
});

it("prepends caution callout to new issue body and includes it only once", async () => {
/** @type {string} */
let capturedIssueBody = "";

global.github = {
rest: {
search: {
issuesAndPullRequests: vi.fn(async ({ q }) => {
if (q.includes("is:pr")) {
return { data: { total_count: 0, items: [] } };
}
return { data: { total_count: 0, items: [] } };
}),
},
issues: {
create: vi.fn(async ({ body }) => {
capturedIssueBody = body;
return {
data: { number: 101, html_url: "https://github.com/owner/repo/issues/101", node_id: "I_123" },
};
}),
},
pulls: {
get: vi.fn(),
},
},
graphql: vi.fn(),
};

await main();

expect(capturedIssueBody).toBeTruthy();
expect(capturedIssueBody.startsWith("> [!CAUTION]")).toBe(true);
expect(capturedIssueBody.indexOf("> [!CAUTION]")).toBeLessThan(capturedIssueBody.indexOf("ISSUE TEMPLATE CONTENT"));
expect((capturedIssueBody.match(/> \[!CAUTION\]/g) || []).length).toBe(1);
expect(capturedIssueBody).toContain("> Generated from [Test Workflow]");
});
});

describe("buildCodePushFailureContext", () => {
it("returns empty string when no errors", () => {
expect(buildCodePushFailureContext("")).toBe("");
Expand Down
10 changes: 4 additions & 6 deletions actions/setup/js/messages.test.cjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -921,7 +921,7 @@ describe("messages.cjs", () => {
expect(result).toContain("> Generated by [Test Workflow]");
});

it("should include caution alert in getFooterAgentFailureIssueMessage when detection is warning", async () => {
it("should not include caution alert in getFooterAgentFailureIssueMessage when detection is warning", async () => {
process.env.GH_AW_DETECTION_CONCLUSION = "warning";
process.env.GH_AW_DETECTION_REASON = "threat_detected";

Expand All @@ -932,12 +932,11 @@ describe("messages.cjs", () => {
runUrl: "https://github.com/test/repo/actions/runs/123",
});

expect(result).toContain("> [!CAUTION]");
expect(result).toContain("Security scanning requires review");
expect(result).not.toContain("> [!CAUTION]");
expect(result).toContain("> Generated from [Test Workflow]");
});

it("should include caution alert in getFooterAgentFailureCommentMessage when detection is warning", async () => {
it("should not include caution alert in getFooterAgentFailureCommentMessage when detection is warning", async () => {
process.env.GH_AW_DETECTION_CONCLUSION = "warning";
process.env.GH_AW_DETECTION_REASON = "parse_error";

Expand All @@ -948,8 +947,7 @@ describe("messages.cjs", () => {
runUrl: "https://github.com/test/repo/actions/runs/123",
});

expect(result).toContain("> [!CAUTION]");
expect(result).toContain("could not be parsed");
expect(result).not.toContain("> [!CAUTION]");
expect(result).toContain("> Generated from [Test Workflow]");
});
});
Expand Down
12 changes: 0 additions & 12 deletions actions/setup/js/messages_footer.cjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -250,12 +250,6 @@ function getFooterAgentFailureIssueMessage(ctx) {
footer = renderTemplate(defaultFooter, templateContext);
}

// Prepend detection caution alert if detection job found a potential issue
const detectionCaution = getDetectionCautionAlert(ctx.workflowName, ctx.runUrl);
if (detectionCaution) {
footer = detectionCaution + "\n\n" + footer;
}

return footer;
}

Expand Down Expand Up @@ -297,12 +291,6 @@ function getFooterAgentFailureCommentMessage(ctx) {
footer = renderTemplate(defaultFooter, templateContext);
}

// Prepend detection caution alert if detection job found a potential issue
const detectionCaution = getDetectionCautionAlert(ctx.workflowName, ctx.runUrl);
if (detectionCaution) {
footer = detectionCaution + "\n\n" + footer;
}

return footer;
}

Expand Down