Skip to content

feat(ci): add script to transform workflows for local AWF testing #356

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
Mossaka wants to merge 19 commits into from
Closed

feat(ci): add script to transform workflows for local AWF testing #356

Mossaka wants to merge 19 commits into from

Conversation

@Mossaka
Copy link
Collaborator

@Mossaka Mossaka commented Jan 18, 2026

Summary

  • Add scripts/use-local-awf.sh script that transforms generated .lock.yml workflow files to use locally built AWF binaries instead of released versions from GHCR
  • This is useful for testing AWF changes in agentic workflows before releasing

What the script does

  1. Replaces curl-based AWF installation with local build commands:

    • Before: curl -sSL ... install.sh | sudo AWF_VERSION=v0.8.2 bash
    • After: git clone && npm ci && npm run build && sudo npm link

  2. Replaces --image-tag flag with --build-local:

    • Before: --image-tag 0.8.2
    • After: --build-local

Usage

# Transform all .lock.yml files in .github/workflows/
./scripts/use-local-awf.sh

# Preview changes without modifying files
./scripts/use-local-awf.sh --dry-run

# Transform specific files
./scripts/use-local-awf.sh .github/workflows/smoke-claude.lock.yml

Test plan

  • Created the script with dry-run and actual modification modes
  • Tested --dry-run mode detects all 14 lock files
  • Tested actual transformation on a copy of smoke-claude.lock.yml
  • Verified AWF install step is correctly replaced
  • Verified --image-tag X.Y.Z is replaced with --build-local

🤖 Generated with Claude Code

@Mossaka @claude
feat(scripts): add script to transform workflows for local AWF testing
70676ad 
Add use-local-awf.sh script that transforms generated .lock.yml workflow
files to use locally built AWF binaries instead of released versions.

The script:
- Replaces curl-based AWF installation with npm ci/build/link
- Replaces --image-tag with --build-local flag
- Supports --dry-run mode for previewing changes
- Can process all lock files or specific files

This is useful for testing AWF changes before releasing, allowing
agentic workflows to use the local development build.

Usage:
  ./scripts/use-local-awf.sh                    # Transform all workflows
  ./scripts/use-local-awf.sh --dry-run          # Preview changes
  ./scripts/use-local-awf.sh path/to/file.yml   # Transform specific file

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@github-actions
Copy link

github-actions bot commented Jan 18, 2026 •
edited
Loading

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

@github-actions
Copy link

github-actions bot commented Jan 18, 2026 •
edited
Loading

✅ Coverage Check Passed

Overall Coverage

Metric Base PR Delta
Lines 77.88% 77.88% ➡️ +0.00%
Statements 77.94% 77.94% ➡️ +0.00%
Functions 77.29% 77.29% ➡️ +0.00%
Branches 71.00% 71.00% ➡️ +0.00%

Coverage comparison generated by scripts/ci/compare-coverage.ts

@github-actions
Copy link

github-actions bot commented Jan 18, 2026 •
edited
Loading

🎬 THE END — Smoke Claude MISSION: ACCOMPLISHED! The hero saves the day! ✨

@github-actions
Copy link

github-actions bot commented Jan 18, 2026

Claude Smoke Test Results

Recent PRs:

Test Results:

  • ✅ GitHub MCP - Fetched 2 merged PRs
  • ✅ Playwright - Page title: "GitHub · Change is constant. GitHub keeps you ahead. · GitHub"
  • ✅ File Writing - Created test file successfully
  • ✅ Bash Tool - Verified file contents

Status: PASS

AI generated by Smoke Claude

@github-actions
Copy link

github-actions bot commented Jan 18, 2026

Smoke Test Results

Last 2 Merged PRs:

Test Results:

  • ✅ GitHub MCP: Reviewed last 2 merged PRs
  • ❌ Playwright: Missing system dependencies (libglib-2.0.so.0)
  • ✅ File Write: Created /tmp/gh-aw/agent/smoke-test-copilot-21120165245.txt
  • ✅ Bash: Verified file contents successfully

Overall Status: FAIL

cc: @Mossaka (PR author)

AI generated by Smoke Copilot

@Mossaka @claude
feat(ci): transform smoke-copilot to use local AWF build
8221bdf 
- Apply use-local-awf.sh transformation to smoke-copilot.lock.yml
- Replace curl-based AWF install with local git clone + npm build
- Replace --image-tag with --build-local flag
- Add PRD document for local AWF testing
- Add Ralph CI monitor script for tracking workflow status

This allows testing AWF changes before release by building from source.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@github-actions
Copy link

github-actions bot commented Jan 22, 2026 •
edited
Loading

📰 DEVELOPING STORY: Smoke Copilot reports was cancelled. Our correspondents are investigating the incident...

@github-actions
Copy link

github-actions bot commented Jan 22, 2026 •
edited
Loading

💫 TO BE CONTINUED... Smoke Claude was cancelled! Our hero faces unexpected challenges...

@Mossaka @claude
fix(scripts): use correct gh CLI field names in ralph prompt
6168ed9 
- Use state instead of conclusion (gh pr checks uses different fields)
- Update workflow run listing command

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@github-actions
Copy link

github-actions bot commented Jan 22, 2026 •
edited
Loading

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

@github-actions
Copy link

github-actions bot commented Jan 22, 2026 •
edited
Loading

🎬 THE END — Smoke Claude MISSION: ACCOMPLISHED! The hero saves the day! ✨

@github-actions
Copy link

github-actions bot commented Jan 22, 2026

Smoke Test Results

Recent merged PRs:

Test Results:

  • ✅ GitHub MCP: Retrieved last 2 merged PRs
  • ✅ Playwright: Navigated to github.com, title contains "GitHub"
  • ✅ File Write: Created /tmp/gh-aw/agent/smoke-test-claude-21231220885.txt
  • ✅ Bash: Verified file content

Status: PASS

AI generated by Smoke Claude

@github-actions
Copy link

github-actions bot commented Jan 22, 2026

Smoke Test Results

Last 2 Merged PRs:

Test Results:

  • ✅ GitHub MCP: Retrieved PR data
  • ✅ Playwright: GitHub page title verified (contains "GitHub")
  • ✅ File Writing: Test file created successfully
  • ✅ Bash Tool: File verified with cat

Overall Status: PASS

cc: @Mossaka (author/assignee)

AI generated by Smoke Copilot

@Mossaka @claude
fix(scripts): pipe prompt via stdin for claude --print
0404ef0 
The --print flag requires input via stdin, not as a positional argument.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@github-actions
Copy link

github-actions bot commented Jan 22, 2026 •
edited
Loading

🎬 THE END — Smoke Claude MISSION: ACCOMPLISHED! The hero saves the day! ✨

@github-actions
Copy link

github-actions bot commented Jan 22, 2026 •
edited
Loading

📰 DEVELOPING STORY: Smoke Copilot reports failed. Our correspondents are investigating the incident...

@github-actions
Copy link

github-actions bot commented Jan 22, 2026

Smoke Test Results

Last 2 Merged PRs:

Test Results:

  • ✅ GitHub MCP - Retrieved recent PRs
  • ✅ Playwright - Navigated to github.com, title contains "GitHub"
  • ✅ File Write - Created test file at /tmp/gh-aw/agent/smoke-test-claude-21231570083.txt
  • ✅ Bash Tool - Verified file content successfully

Overall: PASS

AI generated by Smoke Claude

@Mossaka @claude
refactor(scripts): simplify ralph loop to general pattern
311daef 
- Use claude -p with @file syntax for prompt
- Use --permission-mode acceptEdits
- Simpler iteration loop

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@github-actions
Copy link

github-actions bot commented Jan 22, 2026 •
edited
Loading

📰 DEVELOPING STORY: Smoke Copilot reports was cancelled. Our correspondents are investigating the incident...

@github-actions
Copy link

github-actions bot commented Jan 22, 2026 •
edited
Loading

💫 TO BE CONTINUED... Smoke Claude was cancelled! Our hero faces unexpected challenges...

@Mossaka @claude
feat(scripts): add task execution instructions to ralph loop
adf5421 
Include standard ralph loop instructions for task execution,
testing, PRD updates, and progress tracking.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@github-actions
Copy link

github-actions bot commented Jan 22, 2026 •
edited
Loading

💫 TO BE CONTINUED... Smoke Claude was cancelled! Our hero faces unexpected challenges...

@Mossaka @claude
refactor(scripts): make ralph loop accept PRD and progress file params
03dada0 
Usage: ./scripts/ralph-ci-monitor.sh <iterations> <prd_file> [progress_file]

- Takes PRD file path as required argument
- Progress file defaults to progress.txt in same directory as PRD
- Creates progress file if it doesn't exist

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@github-actions
Copy link

github-actions bot commented Jan 22, 2026 •
edited
Loading

📰 DEVELOPING STORY: Smoke Copilot reports was cancelled. Our correspondents are investigating the incident...

@github-actions
Copy link

github-actions bot commented Jan 22, 2026 •
edited
Loading

💫 TO BE CONTINUED... Smoke Claude was cancelled! Our hero faces unexpected challenges...

@Mossaka @claude
refactor(scripts): rename ralph-ci-monitor.sh to ralph
4d20038 
Simpler name for the general-purpose Ralph loop script.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@github-actions
Copy link

github-actions bot commented Jan 22, 2026 •
edited
Loading

🎬 THE END — Smoke Claude MISSION: ACCOMPLISHED! The hero saves the day! ✨

@github-actions
Copy link

github-actions bot commented Jan 22, 2026 •
edited
Loading

📰 DEVELOPING STORY: Smoke Copilot reports failed. Our correspondents are investigating the incident...

@github-actions github-actions bot mentioned this pull request Jan 22, 2026
Open
@github-actions
Copy link

github-actions bot commented Jan 22, 2026

Smoke Test Results (Claude)

Last 2 merged PRs:

Test Results:

  • ✅ GitHub MCP: Retrieved PR list
  • ✅ Playwright: Page title contains "GitHub"
  • ✅ File Write: Created test file
  • ✅ Bash: Verified file contents

Status: PASS

AI generated by Smoke Claude

@Mossaka @claude
fix(ci): replace --build-local with --image-tag for GHCR containers
68d788c 
The --build-local flag requires AWF source files to build containers, but
the workflow installs AWF as a pre-built binary via install_awf_binary.sh.
This caused container build failures:

  unable to prepare context: path "/snapshot/gh-aw-firewall/containers/agent" not found

Changed to --image-tag 0.10.0 to use pre-built containers from GHCR that
match the installed AWF version.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@github-actions
Copy link

github-actions bot commented Jan 22, 2026 •
edited
Loading

🎬 THE END — Smoke Claude MISSION: ACCOMPLISHED! The hero saves the day! ✨

@github-actions
Copy link

github-actions bot commented Jan 22, 2026 •
edited
Loading

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

@github-actions
Copy link

github-actions bot commented Jan 22, 2026

Smoke Test Results (Copilot)

Status: PASS ✅

cc @Mossaka

AI generated by Smoke Copilot

@github-actions
Copy link

github-actions bot commented Jan 22, 2026

Smoke Test Results

Last 2 Merged PRs:

Test Results:

  • ✅ GitHub MCP - Retrieved PRs successfully
  • ✅ Playwright - Navigated to github.com, title contains "GitHub"
  • ✅ File Writing - Created /tmp/gh-aw/agent/smoke-test-claude-21232751841.txt
  • ✅ Bash Tool - Verified file contents

Status: PASS

AI generated by Smoke Claude

@github-actions
Copy link

github-actions bot commented Jan 22, 2026

Security Review: Security Boundary Extension Detected

⚠️ Security Concern: Host Access Enablement

This PR introduces the --enable-host-access flag to workflow files, which when combined with the existing host.docker.internal domain in the allowlist, extends the security boundary to grant the AI agent access to ANY service running on the GitHub Actions runner host.

File and Lines:

  • .github/workflows/smoke-copilot.lock.yml (line ~717 in the new version)

Code Change:

- --image-tag 0.8.2 \
+ --enable-host-access --image-tag 0.10.0 \

Security Impact:

According to AWF's own documentation (docs/usage.md), this combination has explicit security warnings:

⚠️ Security Warning: When --enable-host-access is combined with host.docker.internal in --allow-domains, containers can access ANY service running on the host machine, including:

  • Local databases (PostgreSQL, MySQL, Redis)
  • Development servers
  • Any network service listening on localhost

Why This Matters:

  1. Expanded Attack Surface: A compromised or malicious AI agent could now probe and potentially exploit ANY service running on the GitHub Actions runner
  2. GitHub Actions Context: In GitHub Actions runners, this could include:
    • Docker daemon (if exposed)
    • Metadata services
    • Other workflow processes
    • Cached credentials or secrets in memory

Context & Justification:

The PR description and accompanying code suggest this is intended for MCP gateway support, which is a legitimate use case. The setup includes:

  • MCP gateway environment variables (MCP_GATEWAY_DOMAIN, MCP_GATEWAY_PORT)
  • API key generation for gateway authentication

However, the security boundary extension goes beyond just the MCP gateway - it allows access to ALL host services, not just port 80 (the configured MCP gateway port).

Recommended Actions:

  1. Add --allow-host-ports 80 flag: This would restrict host access to ONLY port 80 (the MCP gateway port), rather than the default ports 80 and 443. This follows the principle of least privilege.

    sudo -E awf \
  --enable-host-access \
  --allow-host-ports 80 \
  --allow-domains host.docker.internal,...

  2. Document the risk: Add a comment in the workflow explaining why host access is needed and acknowledging the security trade-off.

  3. Consider alternatives: If possible, evaluate whether the MCP gateway could run inside the firewall container instead of on the host.

Verdict:

While this change appears intentional for MCP gateway functionality, it represents a significant security boundary extension that should be:

  • Explicitly reviewed and approved by security-conscious maintainers
  • Documented in the workflow file
  • Potentially scoped down using --allow-host-ports to minimize the attack surface

The rest of the PR (the use-local-awf.sh script and version updates) does not introduce security concerns - it's purely a development/testing convenience tool.

AI generated by Security Guard

@Mossaka @claude
docs: mark PRD complete - GitHub MCP now working in smoke tests
c910493 
Workflow run 21232751884 passed with all tests:
- GitHub MCP: Retrieved last 2 merged PRs
- Playwright: Verified github.com title
- File Writing: Created test file
- Bash: Verified file creation

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@github-actions
Copy link

github-actions bot commented Jan 22, 2026 •
edited
Loading

🎬 THE END — Smoke Claude MISSION: ACCOMPLISHED! The hero saves the day! ✨

@github-actions
Copy link

github-actions bot commented Jan 22, 2026 •
edited
Loading

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

@github-actions
Copy link

github-actions bot commented Jan 22, 2026

Smoke Test Results

Recent PRs:

Tests:

  • ✅ GitHub MCP: Retrieved 2 PRs
  • ✅ Playwright: Page title contains "GitHub"
  • ✅ File Writing: Created /tmp/gh-aw/agent/smoke-test-claude-21232846922.txt
  • ✅ Bash Tool: Verified file content

Status: PASS

AI generated by Smoke Claude

@github-actions
Copy link

github-actions bot commented Jan 22, 2026

Smoke Test Results for Copilot

✅ GitHub MCP: #344 "feat: add test-coverage-improver agentic workflow", #340 "feat(ci): add CI Failure Doctor workflow"
✅ Playwright: GitHub page title verified
✅ File Write: /tmp/gh-aw/agent/smoke-test-copilot-21232846975.txt
✅ Bash Tool: File verified

Status: PASS

cc @Mossaka @Copilot

AI generated by Smoke Copilot

@Mossaka @claude
fix(ci): resolve workflow compilation errors in ci-doctor and release
60e61fe 
ci-doctor.md:
- Remove disallowed 'issues: write' permission (use safe-outputs instead)
- Replace disallowed expressions (workflow_run.name, head_branch) with
  allowed alternatives (run_number, event)
- Add required 'issues: read' permission for github toolsets

release.md:
- Fix template injection vulnerabilities by converting step outputs to
  environment variables
- Convert multi-line run blocks to single-line to avoid compiler's env
  block capture issue
- Use proper YAML quoting for commands containing colons

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@github-actions
Copy link

github-actions bot commented Jan 22, 2026 •
edited
Loading

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

@github-actions
Copy link

github-actions bot commented Jan 22, 2026 •
edited
Loading

🎬 THE END — Smoke Claude MISSION: ACCOMPLISHED! The hero saves the day! ✨

@github-actions
Copy link

github-actions bot commented Jan 22, 2026

Smoke Test Results

Last 2 merged PRs:

✅ GitHub MCP
✅ Playwright (title: "GitHub · Change is constant. GitHub keeps you ahead. · GitHub")
✅ File write/read
✅ Bash commands

Status: PASS

cc: @Mossaka

AI generated by Smoke Copilot

@github-actions
Copy link

github-actions bot commented Jan 22, 2026

Smoke Test Results - Claude ✅

Last 2 Merged PRs:

Test Results:

  • ✅ GitHub MCP - Successfully retrieved PR data
  • ✅ Playwright - Page title contains "GitHub"
  • ✅ File Write - Created test file successfully
  • ✅ Bash Tool - Verified file contents

Overall Status: PASS

AI generated by Smoke Claude

Mossaka and others added 4 commits January 22, 2026 18:49
@Mossaka @claude
refactor: remove unnecessary files from PR
72350ef 
Remove PRD tracking files and revert workflow source changes:
- prd/fix-smoke-test.md
- prd/progress.txt
- scripts/prd.md
- scripts/progress.txt
- scripts/ralph
- scripts/ralph-prompt.md
- Revert ci-doctor.md and release.md to main branch versions

Keep only .lock.yml changes and use-local-awf.sh script.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@Mossaka @claude
chore: recompile workflows with latest gh-aw (v0.37.10)
9e3bdcc 
Recompile all workflow lock files using the latest gh-aw compiler.
The new compiler automatically uses --build-local instead of
--image-tag for AWF installation.

Note: ci-doctor.lock.yml and release.lock.yml could not be recompiled
due to security issues in the source .md files (issues: write permission
and template injection vulnerabilities). These will need separate fixes
in the source files.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@Mossaka @claude
docs: add workflow compilation instructions to CLAUDE.md
6fb8e8f 
Document that agents should always run use-local-awf.sh after
compiling workflows with gh aw compile to transform them for
local AWF testing.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@Mossaka @claude
revert: restore lock.yml files to main branch versions
f4454fb 
The latest gh-aw compiler (v0.37.10) generates workflows that reference
./actions/setup, a local action that doesn't exist in this repository.
Revert all lock files to their working main branch versions.

Remove ci-doctor.lock.yml as it was a new file that also has this issue.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@github-actions
Copy link

github-actions bot commented Jan 22, 2026 •
edited
Loading

🎬 THE END — Smoke Claude MISSION: ACCOMPLISHED! The hero saves the day! ✨

@github-actions
Copy link

github-actions bot commented Jan 22, 2026 •
edited
Loading

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

@github-actions
Copy link

github-actions bot commented Jan 22, 2026

Smoke Test Results (Claude)

Last 2 merged PRs:

✅ GitHub MCP - Retrieved PRs successfully
✅ Playwright - Page title contains "GitHub"
✅ File Write - Created test file
✅ Bash - Verified file contents

Status: PASS

AI generated by Smoke Claude

@github-actions
Copy link

github-actions bot commented Jan 22, 2026

Smoke Test Results

Last 2 Merged PRs:

Test Results:

  • ✅ GitHub MCP: Fetched last 2 merged PRs
  • ❌ Playwright: Failed to verify GitHub page title
  • ✅ File Writing: Created test file successfully
  • ✅ Bash Tool: Verified file contents

Status: FAIL (3/4 tests passed)

cc: @Mossaka @Copilot

AI generated by Smoke Copilot

@Mossaka Mossaka closed this Jan 22, 2026
This was referenced Jan 22, 2026
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

smoke-claude smoke-copilot

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Mossaka