diff --git a/scratchpad/layout.md b/scratchpad/layout.md index 420b59f6a7c..ee5149c1eb5 100644 --- a/scratchpad/layout.md +++ b/scratchpad/layout.md @@ -1,18 +1,18 @@ # GitHub Actions Workflow Layout Specification > Auto-generated specification documenting patterns used in compiled `.lock.yml` files. -> Last updated: 2026-04-06 +> Last updated: 2026-04-13 ## Overview This document catalogs all file paths, folder names, artifact names, and other patterns used across our compiled GitHub Actions workflows (`.lock.yml` files). It serves as a comprehensive reference for developers working with the gh-aw codebase. **Statistics:** -- **Lock files analyzed**: 181 +- **Lock files analyzed**: 187 - **Unique GitHub Actions**: 25 -- **Artifact patterns**: 25 +- **Artifact patterns**: 27 - **Job name patterns**: 30 -- **File path references**: 43 +- **File path references**: 47 ## GitHub Actions @@ -21,26 +21,26 @@ Common GitHub Actions used across compiled workflows: | Action | Version (SHA) | Description | Context | |--------|---------------|-------------|---------| | `actions/checkout` | `de0fac2e...` | Checks out repository code | Used in almost all workflows for accessing repo content | -| `actions/upload-artifact` | `bbbca2dd...` | Uploads build artifacts | Used for agent outputs, patches, prompts, logs, and safe-output data | +| `actions/upload-artifact` | `043fb46d...` | Uploads build artifacts | Used for agent outputs, patches, prompts, logs, and safe-output data | | `actions/download-artifact` | `3e5f45b2...` | Downloads artifacts from previous jobs | Used in safe-output jobs and conclusion jobs | | `actions/setup-node` | `53b83947...` | Sets up Node.js environment | Used in workflows requiring npm/node | | `actions/setup-python` | `a309ff8b...` | Sets up Python environment | Used for Python-based workflows and scripts | | `actions/setup-go` | `4a360112...` | Sets up Go environment | Used for Go-based builds and tests | | `actions/setup-java` | `be666c2f...` | Sets up Java environment | Used for Java-based workflows | | `actions/setup-dotnet` | `c2fa09f4...` | Sets up .NET environment | Used for .NET-based workflows | -| `actions/github-script` | `ed597411...` | Runs GitHub API scripts | Used for GitHub API interactions and workflow logic | +| `actions/github-script` | `373c709c...` | Runs GitHub API scripts | Used for GitHub API interactions and workflow logic | | `actions/cache` | `66822842...` | Caches dependencies | Used for caching npm, pip, go modules | | `actions/cache/restore` | `66822842...` | Restores cached dependencies | Explicit cache restore action | | `actions/cache/save` | `66822842...` | Saves dependencies to cache | Explicit cache save action | | `docker/setup-buildx-action` | `4d04d5d9...` | Sets up Docker Buildx | Used for multi-platform Docker builds | -| `docker/build-push-action` | `d08e5c35...` | Builds and pushes Docker images | Used in release workflows | +| `docker/build-push-action` | `bcafcacb...` | Builds and pushes Docker images | Used in release workflows | | `docker/login-action` | `4907a6dd...` | Logs in to Docker registry | Used before pushing Docker images | | `docker/metadata-action` | `030e8812...` | Extracts Docker metadata | Used for tagging Docker images | | `astral-sh/setup-uv` | `cec20831...`, `eac588ad...` | Sets up uv package manager | Used for Python package management | | `anchore/sbom-action` | `e22c3899...` | Generates SBOM | Used for security and compliance | | `super-linter/super-linter` | `9e863354...` | Runs super-linter | Used for code quality checks | | `github/codeql-action/upload-sarif` | `0e9f5595...` | Uploads SARIF to GitHub Code Scanning | Used for security scanning results from threat detection | -| `github/stale-repos` | `25946246...` | Manages stale repositories | Used for repository maintenance | +| `github/stale-repos` | `5f2e18fc...` | Manages stale repositories | Used for repository maintenance | | `microsoft/apm-action` | `a190b0b1...` | Collects APM bundle data | Used for performance monitoring and APM artifact creation | | `./actions/setup` | N/A (local) | Custom setup action | Copies JavaScript and shell scripts to `/tmp/gh-aw/actions` | | `actions-ecosystem/action-add-labels` | `c96b68fe...` | Adds labels to issues/PRs | Used in safe-output jobs for label management | @@ -57,8 +57,9 @@ Artifacts uploaded/downloaded between workflow jobs: | `agent-output` | Agent job (legacy) | Safe-output jobs | Legacy name for agent output; superseded by unified `agent` artifact | | `detection` | Detection job | Conclusion job | Detection analysis output (threat logs, analysis results) | | `safe-output` | Safe-output jobs | Conclusion job | Safe output results (PR creation, commenting, etc.) | -| `safe-output-items` | Safe-output jobs | Conclusion job | Individual safe output item payloads (JSONL format) | +| `safe-outputs-items` | Safe-output jobs | Conclusion job | Individual safe output item payloads (JSONL format) | | `safe-outputs-assets` | Safe-output jobs | Conclusion job | Assets generated by safe-output tools | +| `safe-outputs-upload-artifacts` | Safe-output jobs | N/A | Staged upload artifacts from safe-output handlers | | `cache-memory` | Agent job, update_cache_memory job | Next workflow run | Persistent data across workflow runs | | `cache-memory-focus-areas` | Agent job | Next workflow run | Focused cache data for specific areas | | `cache-memory-repo-audits` | Agent job | Next workflow run | Repository audit cache data | @@ -159,6 +160,13 @@ Common file paths referenced in workflow files: | `${{ runner.temp }}/gh-aw/safe-jobs/` | Directory | Safe job definitions | Safe output job definitions and handlers | | `${{ runner.temp }}/gh-aw/gh-aw` | File | gh-aw binary | The gh-aw CLI binary copied for use in safe-output jobs | | `${{ runner.temp }}/gh-aw/safeoutputs/` | Directory | Safe outputs runtime | Safe outputs MCP server, config, and output files | +| `/tmp/gh-aw/otel.jsonl` | File | OpenTelemetry spans log | OTLP span mirror written by `send_otlp_span.cjs` | +| `/tmp/gh-aw/github_rate_limits.jsonl` | File | GitHub rate limits log | Per-call rate-limit headers captured from GitHub API calls | +| `/tmp/gh-aw/temporary-id-map.json` | File | Temporary ID map | Maps temporary IDs (e.g., `aw_abc123`) to GitHub resource references | +| `/tmp/gh-aw/apm-bundle` | Directory | APM bundle | Application Performance Monitoring bundle data | +| `/tmp/gh-aw/proxy-logs/` | Directory | Proxy logs | Logs from network proxy (AWF/DIFC proxy) | +| `/tmp/gh-aw/threat-detection/detection.log` | File | Detection log | Threat detection analysis log | +| `/tmp/gh-aw/sarif/` | Directory | SARIF download path | Where `upload_code_scanning_sarif` job downloads the SARIF artifact | | `${{ env.GH_AW_AGENT_OUTPUT }}` | Environment var | Agent output path | Dynamic path to agent output file | | `${{ env.GH_AW_SAFE_OUTPUTS }}` | Environment var | Safe outputs path | Dynamic path to safe outputs directory | @@ -183,11 +191,20 @@ const AgentOutputArtifactName = "agent-output" const AgentArtifactName = "agent" const DetectionArtifactName = "detection" const ActivationArtifactName = "activation" -const APMArtifactName = "apm" -const SafeOutputItemsArtifactName = "safe-output-items" +const SafeOutputItemsArtifactName = "safe-outputs-items" +const SarifArtifactName = "code-scanning-sarif" +const FirewallAuditArtifactName = "firewall-audit-logs" // Legacy +const LegacyDetectionArtifactName = "threat-detection.log" // Legacy const AgentOutputFilename = "agent_output.json" const SafeOutputsFilename = "safeoutputs.jsonl" +const TokenUsageFilename = "agent_usage.json" +const GithubRateLimitsFilename = "github_rate_limits.jsonl" +const OtelJsonlFilename = "otel.jsonl" +const TemporaryIdMapFilename = "temporary-id-map.json" +const SarifFileName = "code-scanning-alert.sarif" +const SarifArtifactDownloadPath = "/tmp/gh-aw/sarif/" const ArtifactPrefixOutputName = "artifact_prefix" +const SafeOutputsUploadArtifactStagingArtifactName = "safe-outputs-upload-artifacts" ```` ### Job Names @@ -205,9 +222,14 @@ const CheckStopTimeStepID StepID = "check_stop_time" const CheckSkipIfMatchStepID StepID = "check_skip_if_match" const CheckSkipIfNoMatchStepID StepID = "check_skip_if_no_match" const CheckCommandPositionStepID StepID = "check_command_position" +const RemoveTriggerLabelStepID StepID = "remove_trigger_label" +const GetTriggerLabelStepID StepID = "get_trigger_label" const CheckRateLimitStepID StepID = "check_rate_limit" const CheckSkipRolesStepID StepID = "check_skip_roles" const CheckSkipBotsStepID StepID = "check_skip_bots" +const CheckSkipIfCheckFailingStepID StepID = "check_skip_if_check_failing" +const PreActivationAppTokenStepID StepID = "pre-activation-app-token" +const ParseMCPGatewayStepID StepID = "parse-mcp-gateway" ```` ### Directory Paths @@ -215,7 +237,9 @@ const CheckSkipBotsStepID StepID = "check_skip_bots" const GhAwRootDir = "${{ runner.temp }}/gh-aw" const GhAwRootDirShell = "${RUNNER_TEMP}/gh-aw" const AWFProxyLogsDir = "/tmp/gh-aw/sandbox/firewall/logs" +const AWFAuditDir = "/tmp/gh-aw/sandbox/firewall/audit" const DefaultMCPGatewayPayloadDir = "/tmp/gh-aw/mcp-payloads" +const SarifArtifactDownloadPath = "/tmp/gh-aw/sarif/" ```` ### MCP Server IDs @@ -226,12 +250,15 @@ const AgenticWorkflowsMCPServerID MCPServerID = "agenticworkflows" ```` ### Default Versions ````go +const DefaultCopilotVersion Version = "1.0.21" +const DefaultClaudeCodeVersion Version = "2.1.98" +const DefaultCodexVersion Version = "0.118.0" +const DefaultGeminiVersion Version = "0.37.1" const DefaultGitHubMCPServerVersion Version = "v0.32.0" -const DefaultFirewallVersion Version = "v0.25.13" -const DefaultMCPGatewayVersion Version = "v0.2.14" +const DefaultFirewallVersion Version = "v0.25.18" +const DefaultMCPGatewayVersion Version = "v0.2.17" const DefaultPlaywrightMCPVersion Version = "0.0.70" -const DefaultQmdVersion Version = "2.0.1" -const DefaultGitHubScriptVersion Version = "v8" +const DefaultGitHubScriptVersion Version = "v9" const DefaultPlaywrightBrowserVersion Version = "v1.59.1" const DefaultMCPSDKVersion Version = "1.24.0" const DefaultBunVersion Version = "1.1" @@ -244,6 +271,10 @@ const DefaultJavaVersion Version = "21" const DefaultElixirVersion Version = "1.17" const DefaultHaskellVersion Version = "9.10" const DefaultDenoVersion Version = "2.x" +// Minimum version guards +const AWFExcludeEnvMinVersion Version = "v0.25.3" +const AWFCliProxyMinVersion Version = "v0.25.17" +const CopilotNoAskUserMinVersion Version = "1.0.19" ```` ### Container Images @@ -294,7 +325,9 @@ MCPScriptsFeatureFlag FeatureFlag = "mcp-scripts" MCPGatewayFeatureFlag FeatureFlag = "mcp-gateway" DisableXPIAPromptFeatureFlag FeatureFlag = "disable-xpia-prompt" CopilotRequestsFeatureFlag FeatureFlag = "copilot-requests" -DIFCProxyFeatureFlag FeatureFlag = "difc-proxy" +DIFCProxyFeatureFlag FeatureFlag = "difc-proxy" // Deprecated: use tools.github.integrity-proxy +CliProxyFeatureFlag FeatureFlag = "cli-proxy" +CopilotIntegrationIDFeatureFlag FeatureFlag = "copilot-integration-id" ```` ### Engine Names @@ -351,6 +384,9 @@ GitHub Actions runner images used across compiled workflows: /tmp/gh-aw/ ├── agent/ # Agent workspace ├── agent-stdio.log # Agent logs +├── agent_output.json # Agent output JSON +├── agent_usage.json # Token usage JSON +├── apm-bundle/ # APM bundle data ├── aw-prompts/ # Prompt storage │ └── prompt.txt ├── aw.patch # Git patch @@ -359,22 +395,35 @@ GitHub Actions runner images used across compiled workflows: ├── cache-memory-chroma/ # Vector DB cache ├── cache-memory-focus-areas/ # Focus areas cache ├── cache-memory-repo-audits/ # Audit cache +├── github_rate_limits.jsonl # GitHub API rate limit log ├── mcp-config/logs/ # MCP config logs ├── mcp-logs/ # MCP server logs ├── mcp-payloads/ # MCP gateway payloads +├── mcp-scripts/logs/ # MCP scripts logs +├── otel.jsonl # OpenTelemetry span mirror +├── proxy-logs/ # Network proxy logs ├── python/ # Python scripts/data │ ├── *.py │ ├── charts/*.png │ └── data/* +├── redacted-urls.log # Redacted URL log ├── repo-memory/ # Repository memory │ ├── campaigns/ │ └── default/ ├── safe-inputs/logs/ # Safe input logs +├── safe-output-items.jsonl # Safe output items manifest +├── safeoutputs.jsonl # Raw safe outputs NDJSON ├── safeoutputs/ # Safe output data │ └── assets/ -└── sandbox/ # Sandbox execution - ├── agent/logs/ - └── firewall/logs/ +├── sandbox/ # Sandbox execution +│ ├── agent/logs/ +│ └── firewall/ +│ ├── audit/ +│ └── logs/ +├── sarif/ # SARIF download path +├── temporary-id-map.json # Temporary ID map +└── threat-detection/ + └── detection.log # Threat detection log ```` ### GitHub Workflow Directory Structure @@ -419,9 +468,9 @@ This specification is automatically maintained by the **Layout Specification Mai 4. Updates this document with findings 5. Creates a PR with the changes -**Last extraction run**: 2026-04-06 -**Lock files analyzed**: 181 -**Patterns documented**: 240+ +**Last extraction run**: 2026-04-13 +**Lock files analyzed**: 187 +**Patterns documented**: 250+ ---