Conversation
Merged
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Update detection job to include rendered prompt in step summary
detection job
Oct 2, 2025
pelikhan
approved these changes
Oct 2, 2025
pelikhan
added a commit
that referenced
this pull request
Oct 2, 2025
* Implement streamlined context-aware threat detection for GitHub Agentic Workflows safe-outputs (#1077) * Initial plan * Add threat detection infrastructure - config, parsing, and job building Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * Add comprehensive tests and schema validation for threat detection Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * Fix safe-output job dependencies to correctly reference agent outputs Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * Remove engine option from threat-detection, use top-level engine instead Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * Implement agentic engine execution for threat detection Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * Refactor threat detection to use embedded JS parsing and prompt files Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * Update threat detection to use single-line JSON format and remove custom prompt support Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * Add workflow source context to threat detection prompt Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * Replace bash setup step with JavaScript using actions/github-script Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * Update JavaScript setup to use @actions/core package and add .cjs files to TypeScript config Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * Merge main and simplify threat detection implementation Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * Implement threat detection workflows across multiple test scenarios - Added a new detection job to `test-copilot-missing-tool.lock.yml` to analyze agent output for security threats, including prompt injection, secret leaks, and malicious patches. - Introduced a detection job in `test-copilot-patch-size-exceeded.lock.yml` to validate max-patch-size limits and ensure security analysis of agent outputs. - Enhanced `test-copilot-playwright-screenshots.lock.yml` with a detection job to assess potential security threats in the context of documentation builds and accessibility analysis. - Updated `test-copilot-safe-jobs.lock.yml` to include a detection job for analyzing security threats in safe job executions. - Each detection job includes steps for downloading agent output, setting up threat analysis, executing GitHub Copilot CLI, and parsing results for security verdicts. * Refactor workflow to create issues directly from agent outputs and update documentation for new functionality * simpler codegen * Changes before error encountered Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * Refactor code formatting and remove unnecessary blank lines in workflow files * Add constants for activation and detection job names and update references * Fix safe-output job dependencies to include both agent and detection jobs Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * Refactor discussion and issue creation scripts for improved readability and error handling - Simplified the parsing and validation of agent output in create_discussion.js and create_issue.js. - Enhanced error messages for better debugging. - Streamlined the process of creating discussions and issues, including handling labels and categories. - Added checks for repository settings to ensure discussions and issues can be created. - Improved summary output for staged mode previews. * Refactor buildSafeJobs function to remove agentJobName parameter and use constants.AgentJobName directly * Reorder safe-jobs build logic to improve readability and maintainability * Remove agent parameter from buildSafeJobs calls in tests for consistency --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> Co-authored-by: Peli de Halleux <pelikhan@users.noreply.github.com> * [WIP] Address feedback: format, lint, recompile, and fix tests for detection job (#1160) * Initial plan * Fix test to use correct variable name $COPILOT_CLI_INSTRUCTION Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * [WIP] Address feedback on "detection job" by removing dev.invalid.yml (#1161) * Initial plan * Delete dev.invalid.yml file Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * Remove stop-time checks and unused tools from dev workflow files * Update AI agent output formatting in workflow files and remove deprecated detection template * Update threat detection setup to use 'outputs.output' instead of 'outputs.text' in workflow files * detection job (#1164) * [WIP] Address feedback on using .patch files in detection job implementation (#1165) * Update log commands in GitHub Agentic Workflows instructions to reflect new naming conventions * Refactor log capture steps to improve consistency and clarity in workflow logs * [WIP] Address tests based on feedback for detection job PR (#1168) * Detection job with OWASP GenAI agentic threat checks (#1170) * detection job (#1171) * Threat detection job with isolated execution environment (#1172) * [WIP] Address feedback on formatting, linting, and recompiling for detection job (#1174) --------- Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
post processing job to run threat detection on safe outputs
Changes Made
core.summary.*APIs, making it easy to review what prompt was sent to the threat detection engine without downloading log filesThe step summary displays:
This enhancement improves transparency and debugging by surfacing the exact prompt used for threat analysis directly in the GitHub Actions UI.
💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.