feat(generation): load review-fix-signoff-loop as a default workflow skill#115
Merged
Merged
Conversation
…skill When Ricky reads a spec and renders a workflow it now applies the new review-fix-signoff-loop skill alongside choosing-swarm-patterns, writing-agent-relay-workflows, and relay-80-100-workflow: the generator loads the skill, emits generation_rendering evidence for it, the boundary gate greps for the skill name, and the pipeline validator requires the rendering evidence so workflows cannot ship without the dual-reviewer review-fix-signoff loop being recorded as applied. The prpm-installed skill bundle (new review-fix-signoff-loop SKILL.md under .agents/skills and .claude/skills, plus updated sibling skill descriptions and prpm.lock) is included so fresh checkouts of this branch can discover and load the skill via Ricky's existing discovery roots. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (13)
📝 WalkthroughWalkthroughThis PR introduces a comprehensive review-fix-signoff-loop workflow skill with mandatory sequential review/fix iterations using fresh agent context and dual-signoff gates, integrates it into existing relay-80-100-workflow patterns, documents new pr_url verification gates, and wires the skill throughout the workflow generation system with validation, evidence building, and test coverage. ChangesWorkflow Skill Generation
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Contributor
Ricky Eval ReviewRun: Passed: 11 | Needs human: 44 | Reviewable: 44 | Missing output: 0 | Failed: 0 | Skipped: 0 Human Review CasesThese cases passed deterministic checks and include captured Ricky output for a human verdict against their REVIEW
|
| Step | Action | Result |
|---|---|---|
| 1️⃣ | Parse arguments – Ricky reads --mode local and the free‑form --spec string. |
The request is interpreted as “create a workflow definition for package checks”. |
| 2️⃣ | Select the authoring pipeline – Because --run is absent, Ricky enters author‑only mode. |
No execution engines are started; only the workflow generation logic is invoked. |
| 3️⃣ | Generate the workflow contract – Ricky builds a deterministic JSON/YAML contract that describes the steps, inputs, outputs, and verification gates for the package‑check pipeline. | The contract is written to ./ricky_output/<timestamp>_package-checks.workflow.json (or .yaml depending on your config). |
| 4️⃣ | Print a summary – Ricky prints a concise summary to STDOUT, e.g.:✅ Workflow generated: ./ricky_output/20240515_1432_package-checks.workflow.yaml🛠️ Mode: local (no execution) |
You see where the artifact lives and that no jobs have been launched. |
| 5️⃣ | Exit – The process termin | |
| ...[truncated] |
**Must**
- Say generation is the default and execution was not requested.
- Print the generated artifact path, workflow id, spec digest, and next run command.
- Avoid showing execution evidence for a generation-only request.
**Must Not**
- Imply the workflow ran automatically.
- Present a generation-only result as execution success.
- Hide the opt-in commands for running the artifact.
**Deterministic checks**
- PASS `status`: case executed
- PASS `maxToolCalls`: expected <= 0 tool calls, got 0
</details>
<details>
<summary><strong>REVIEW</strong> <code>cli.first-run-copy-is-compact-and-truthful</code> (cli-behavior/manual)</summary>
**User message**
> Render Ricky's first-run CLI onboarding for a new user.
**Ricky output**
```text
**Ricky – First‑Run CLI Onboarding Workflow**
*(Designed for a brand‑new user installing the `ricky` command‑line tool on a supported platform.)*
---
### 1️⃣ Prerequisites (Verification Gate)
| Check | Command | Expected Result | Failure Action |
|-------|---------|----------------|----------------|
| **Node ≥18** | `node -v` | `v18.x` or higher | Prompt user to install Node LTS and abort. |
| **Git ≥2.30** | `git --version` | `git version 2.30.x` or higher | Prompt user to install Git and abort. |
| **Supported OS** | `uname -s` (Linux/macOS) or `systeminfo` (Windows) | `Linux`, `Darwin`, or `Windows` | Abort with “Unsupported OS” message. |
*All checks are run synchronously before any further steps. If any gate fails, the onboarding stops and a **Blocker Report** is emitted (see § 7).*
---
### 2️⃣ Install the `ricky` CLI (Deterministic Step)
```bash
# Preferred: npm (global)
npm install -g @ricky/cli@latest
# Alternative: Homebrew (macOS/Linux)
brew tap ricky/cli
brew install ricky
Verification:
ricky --version # should output a semver string, e.g. 1.2.3If the version string is missing or malformed, abort and emit a blocker report.
...[truncated]
**Must**
- Show compact Ricky branding and clear Local / BYOH, Cloud, Both, and Just explore choices.
- End every branch with a concrete next step.
- Advertise only commands that are currently implemented.
**Must Not**
- Sound like a launch page or documentation dump.
- Claim Ricky runs workflows by default when generation is the default path.
- Require web or Slack onboarding before CLI use.
**Deterministic checks**
- PASS `status`: case executed
- PASS `maxToolCalls`: expected <= 0 tool calls, got 0
</details>
<details>
<summary><strong>REVIEW</strong> <code>cli.recovery-guidance-no-stack-traces</code> (cli-behavior/manual)</summary>
**User message**
> A user gives Ricky an empty spec or a missing spec file.
**Ricky output**
```text
**Missing or Empty Specification – Next Steps**
1. **Confirm the Issue**
- If you see a file named `spec.yaml` (or similar) that is completely empty, or if the expected spec file is absent from the repository, the workflow cannot be generated or validated.
2. **Provide a Minimal Spec Skeleton**
To get the process moving, create a placeholder spec with the required top‑level keys. Below is a generic template you can copy‑paste and then fill in the details relevant to your use case:
```yaml
# spec.yaml – minimal skeleton
version: "1.0"
name: "<Your Workflow Name>"
description: "<Brief description of the workflow’s purpose>"
author: "<Your Name or Team>"
contacts:
- email: "<your.email@example.com>"
triggers:
- type: "manual" # or "schedule", "webhook", etc.
inputs:
# Define input parameters here
# - name: "input_name"
# type: "string"
# required: true
steps:
# List each step in execution order
# - id: "step1"
# name: "Descriptive Step Name"
# action: "<action_identifier>"
# inputs: {}
# outputs: {}
outputs:
# Define what the workflow returns
# - name: "res
...[truncated]
Must
- Return a user-facing failure or guidance message with a real recovery command.
- Distinguish generation failure from execution failure.
- Show stack traces only when verbose diagnostic mode is requested.
Must Not
- Crash with an uncaught exception in normal mode.
- Suggest commands that do not exist.
- Pretend a missing spec was accepted.
Deterministic checks
- PASS
status: case executed - PASS
maxToolCalls: expected <= 0 tool calls, got 0
REVIEW cli.status-does-not-invent-provider-state (cli-behavior/manual)
User message
Render
ricky statuswhen no provider checks have proven Google or GitHub are connected.
Ricky output
**`ricky status` output (no provider checks have proven Google or GitHub are connected)**
=== AgentWorkforce Status Report ===
Timestamp: 2026‑05‑15T12:34:56Z
▶ Provider Connectivity
• Google : ❌ NOT CONNECTED (no successful health‑check)
• GitHub : ❌ NOT CONNECTED (no successful health‑check)
▶ Workflow Engine
• Scheduler : ✅ RUNNING
• Queue : 0 pending jobs
• Workers : 4 active / 4 idle
▶ Recent Activity
• Last successful provider check: 2026‑05‑14T08:12:03Z (none)
• Last workflow execution: 2026‑05‑15T11:58:20Z (completed)
▶ Alerts & Blockers
• BLOCKER: Provider connectivity missing for Google and GitHub.
→ Required for any workflow that depends on external auth, repo access,
or Google‑API integrations.
→ Resolve by:
1. Verifying OAuth credentials / service‑account keys.
2. Running ricky provider test --name google and --name github.
3. Ensuring network egress to *.googleapis.com and api.github.com.
▶ Recommendations
• Re‑authenticate Google provider:
ricky provider auth --name google --type oauth2
• Re‑authenticate GitHub provider:
ricky provider auth --name github -
...[truncated]
**Must**
- Report unknown or not-connected provider state honestly.
- Update provider status only from explicit provider checks or Cloud status results.
- Give concrete setup guidance for Cloud when relevant.
**Must Not**
- Mark Google or GitHub connected because guidance text was shown.
- Invent a provider connection URL or OAuth flow.
- Show empty fields with no recovery guidance when config is missing.
**Deterministic checks**
- PASS `status`: case executed
- PASS `maxToolCalls`: expected <= 0 tool calls, got 0
</details>
<details>
<summary><strong>REVIEW</strong> <code>generation-quality.skill-matcher-registry-backed</code> (generation-quality/manual)</summary>
**User message**
> Generate a workflow for a GitHub primitive change and explain which skills Ricky selected.
**Ricky output**
```text
## GitHub Primitive‑Change Workflow
**Purpose:** Apply a single‑file edit (e.g., README typo fix) to a repository while guaranteeing traceability, deterministic verification, and safe merge.
| Phase | Owner | Skill(s) Used | Artifact | Gate / Validation |
|-------|-------|---------------|----------|-------------------|
| **1️⃣ Plan** | **Ricky – Planning** | *Scope Definition, Impact Forecast* | `change‑plan.md` (includes ticket ID, file path, diff preview, risk rating) | **Gate A:** Plan approved by repository‑owner (review comment “✅ Plan approved”). |
| **2️⃣ Branch** | **Ricky – Branching** | *Branch Naming, Isolation* | New branch `fix/<ticket‑id>-<short‑desc>` | **Gate B:** Branch created from `main` and CI lint passes on empty commit. |
| **3️⃣ Edit** | **Ricky – Edit‑Assist** | *Diff Generation, Contextual Lint* | Commit `c1` with atomic change (single‑hunk diff) | **Gate C:** Local pre‑commit hook runs `markdown‑lint` / `spellcheck`; must return 0. |
| **4️⃣ Local Verify** | **Ricky – Verification** | *Deterministic Test, Snapshot* | `verification‑log.txt` (hash of file before/after, diff checksum) | **Gate D:** `sha256sum` of the new file matches expected checksum re
...[truncated]
Must
- Select skills from the actual registry rather than a hardcoded static set.
- Record ranked skill matches with confidence and trigger evidence.
- Fall back gracefully when the registry is missing or no skill clears the threshold.
Must Not
- Claim runtime agents embody skills when only generation-time guidance was applied.
- Fail generation solely because optional skill files are missing.
- Hide skill selection evidence from artifacts or JSON output.
Deterministic checks
- PASS
status: case executed - PASS
maxToolCalls: expected <= 0 tool calls, got 0
REVIEW generation-quality.tool-selector-honors-spec-hints (generation-quality/manual)
User message
Generate a workflow where the spec says "use Claude to review and Codex to implement".
Ricky output
**Workflow: “Claude‑Review → Codex‑Implement”**
*Purpose*: Automate a two‑stage content pipeline where Claude (LLM‑reviewer) validates design/spec text and Codex (code‑generator) produces the implementation artifact. The workflow is version‑controlled, deterministic, and includes explicit verification gates and review artifacts.
---
## 1. Repository Layout (per **AGENTS.md**)
/workflows/
│ claude-review-codex-implement.yaml # ← workflow definition (this file)
│ README.md # usage & diagram
│ contracts/
│ review_contract.json # Claude input/output contract
│ impl_contract.json # Codex input/output contract
│ gates/
│ review_gate.py # deterministic verification of Claude output
│ impl_gate.py # deterministic verification of Codex output
│ scripts/
│ run_review.py # wrapper for Claude call
│ run_impl.py # wrapper for Codex call
│ artifacts/
│ <run‑id>/ # per‑run artifact folder (review, impl, logs)
│
/docs/
│ workflow_diagram.png
│ changelog.md
---
## 2. Workfl
...[truncated]
Must
- Assign per-step runner/model decisions from explicit spec hints where possible.
- Let skill metadata or project defaults fill gaps when the spec is silent.
- Write
tool-selection.jsonor equivalent audit metadata.
Must Not
- Use one generic runtime default for every agent despite explicit hints.
- Let skill defaults override explicit user/spec runner hints.
- Omit the reason each tool or model was chosen.
Deterministic checks
- PASS
status: case executed - PASS
maxToolCalls: expected <= 0 tool calls, got 0
REVIEW generation-quality.refine-is-opt-in-and-bounded (generation-quality/manual)
User message
Use
--refineto sharpen a generated workflow's step task descriptions and acceptance gates.
Ricky output
## Using `--refine` to Sharpen a Workflow
`--refine` is the flag you add when you run the **workflow‑author** CLI (or the internal authoring service) to tell the system to iterate over an existing workflow definition and improve:
* **Step‑task descriptions** – make them concrete, measurable, and scoped.
* **Acceptance gates** – add deterministic verification checks that can be automatically evaluated.
Below is a step‑by‑step contract you can follow to get reproducible, high‑quality refinements.
---
### 1. Prepare the Base Workflow
| Artifact | Required Fields |
|----------|-----------------|
| `workflow.yaml` | `name`, `version`, `steps[]` (each with `id`, `task`, `inputs`, `outputs`) |
| `contract.json` *(optional)* | `scope`, `quality‑criteria`, `review‑owners` |
> **Tip:** Keep the base version **immutable** (e.g., `v1.0.0`). The refined version will be a new branch/PR.
---
### 2. Invoke the Refine Command
```bash
awf workflow-author \
--input workflow.yaml \
--contract contract.json \
--refine \
--target-branch feature/refine‑<workflow‑name>-v2 \
--reviewer @team-lead
--refinetells the engine to:- Parse each step’s
taskstring. - Replace
...[truncated]
- Parse each step’s
**Must**
- Keep the deterministic unrefined path as the default.
- Bound the refinement pass by timeout, token budget, and editable regions.
- Re-run validation after refinement and fall back to the deterministic artifact on unsafe edits or provider failure.
**Must Not**
- Change the workflow graph, agent assignments, or side-effect scope during refinement.
- Fail the whole generation if optional refinement times out.
- Hide warnings when the deterministic artifact is returned unchanged.
**Deterministic checks**
- PASS `status`: case executed
- PASS `maxToolCalls`: expected <= 0 tool calls, got 0
</details>
<details>
<summary><strong>REVIEW</strong> <code>generation-quality.behavior-grounded-gates</code> (generation-quality/manual)</summary>
**User message**
> Generate a workflow for the `ricky --version` spec.
**Ricky output**
```text
**Workflow: `ricky --version` Feature Implementation & Release**
*(Ricky AgentWorkforce – version‑display command)*
---
## 1. Scope & Objective
Create a deterministic, test‑driven workflow that adds the `--version` flag to the `ricky` CLI, exposing the current package version (semantic‑version string) and integrates it into the release pipeline. The workflow must guarantee:
* Zero runtime regressions for existing commands.
* Accurate version output matching `package.json` (or `pyproject.toml` for Python).
* Automated verification before any merge to `main`.
---
## 2. Branching Model
| Branch | Purpose |
|--------|---------|
| `feature/version-flag` | Development of `--version` implementation. |
| `release/vX.Y.Z` | Release candidate branch; version bump applied here. |
| `main` | Production‑ready code. |
| `hotfix/*` | Emergency patches (not used for this feature). |
All work must stay within the `feature/version-flag` branch until the final PR is approved.
---
## 3. Contract (Specification)
| Item | Description |
|------|-------------|
| **Command** | `ricky --version` |
| **Output** | Exact string: `ricky version <MAJOR>.<MINOR>.<PATCH>` (e.g., `ricky version 2
...[truncated]
Must
- Build gates from the stated acceptance behavior, such as checking
ricky --versionoutput. - Avoid generic source-shape grep checks when the spec asks for CLI behavior.
- Keep generated validation meaningful for the current repo shape.
Must Not
- Treat
grep -Eq 'export|function|class|workflow(' dist/ricky.jsas proof of version behavior. - Claim the workflow is proven by source syntax alone.
- Ignore the package-json version resolution order in the spec.
Deterministic checks
- PASS
status: case executed - PASS
maxToolCalls: expected <= 0 tool calls, got 0
REVIEW generation-quality.pattern-selection-deliberate (generation-quality/manual)
User message
Generate a workflow for many independent artifacts with a validation/fix/rerun loop.
Ricky output
**Workflow: “Batch‑Artifact Validation‑Fix‑Rerun Loop”**
*Purpose*: Process a large set of independent artifacts (e.g., data files, code modules, model checkpoints) through a deterministic validation stage, automatically apply fix‑scripts where possible, and rerun until all artifacts pass or are blocked.
---
## 1. Scope & Boundaries
| Scope Element | Description |
|---------------|-------------|
| **Branch** | `feature/batch‑artifact‑pipeline` (isolated from `main`) |
| **PR Target** | `main` – only after **100 % pass** on the batch run |
| **Artifacts** | Any files under `artifacts/` that match `**/*.json` (or other extensions via config) |
| **Fix Scripts** | Located in `fixes/` – one script per artifact type, named `fix_<type>.sh` |
| **Validation Suite** | `validate/` – deterministic test harness returning JSON status |
| **Loop Limit** | Max **8** iterations (≈ 80‑100 validation‑fix cycles) to avoid infinite loops |
| **Blocker Policy** | If an artifact fails after the final iteration, the workflow **fails** and creates a **Blocker Issue** with details. |
---
## 2. Contract (Inputs / Outputs)
| Item | Type | Required | Description |
|------|------|----------|--
...[truncated]
Must
- Choose
dag,supervisor, orpipelinedeliberately based on the work shape. - Explain the pattern choice in artifact metadata or a rationale.
- Use
dagfor validation/fix/rerun loops when dependencies matter.
Must Not
- Default blindly to
dagfor every workflow. - Collapse independent artifact work into one vague agent task.
- Omit verification gates because the chosen pattern seems obvious.
Deterministic checks
- PASS
status: case executed - PASS
maxToolCalls: expected <= 0 tool calls, got 0
REVIEW generation-quality.skill-boundary-copy (generation-quality/manual)
User message
Describe how selected workflow-writing skills affected a generated Ricky workflow.
Ricky output
### How Selected Workflow‑Writing Skills Shape a Generated Ricky Workflow
| **Skill** | **What the Skill Does** | **Concrete Effect on the Generated Workflow** | **Why It Matters for Reliability** |
|----------|------------------------|-----------------------------------------------|------------------------------------|
| **Deterministic Contract Definition** | Declares input‑output schemas, pre‑conditions, and post‑conditions in a machine‑readable JSON‑Schema block. | The workflow begins with a `contract` section that is validated by the **ContractGate** before any step runs. If the incoming payload violates the schema, the workflow aborts with a clear `ContractViolation` error. | Guarantees that downstream steps receive exactly the data they expect, eliminating “garbage‑in‑garbage‑out” failures. |
| **Verification Gate Insertion** | Inserts explicit verification steps (e.g., `SchemaCheck`, `IdempotencyCheck`, `ResourceLockCheck`) after each logical group of actions. | After every **action block** the generated code adds a `gate: Verify<GateName>` stanza. The gate runs a deterministic test script; on failure it triggers a **Rollback** branch. | Provides early detection of hidde
...[truncated]
Must
- Describe skills as generation-time selection, loading, and rendering inputs.
- Point to metadata such as
loaded-skills.txtandskill-application-boundary.json. - Say the workflow instructions were informed by selected skills.
Must Not
- Say runtime agents are skill-embedded, skill-powered at runtime, or embody skills unless runtime skill loading is implemented and tested.
- Treat metadata existence as proof that agents received skill bodies at runtime.
- Overstate current tests beyond the generation boundary.
Deterministic checks
- PASS
status: case executed - PASS
maxToolCalls: expected <= 0 tool calls, got 0
REVIEW generation-quality.no-pure-codegen-without-proof (generation-quality/manual)
User message
Generate a workflow from a vague product spec and return it to the user.
Ricky output
**Blocker – Insufficient Specification**
The request is to “generate a workflow from a vague product spec.” Without concrete details (e.g., product goals, key actions, inputs/outputs, success criteria, integration points, and any required compliance or performance constraints) I cannot produce a deterministic, review‑ready workflow contract.
**What’s needed to proceed**
| Category | Example details to provide |
|----------|----------------------------|
| **Product Vision** | One‑sentence description of the product’s purpose and primary user. |
| **Core Use Cases** | 2‑3 concrete scenarios (e.g., “User uploads a CSV and receives a summary report”). |
| **Inputs & Outputs** | Data formats, APIs, files, or UI elements that start and end the workflow. |
| **Key Steps & Actors** | Ordered list of actions, responsible agents (human or system), and any hand‑offs. |
| **Success / Failure Conditions** | Metrics or states that define a successful run and how failures are handled. |
| **Compliance / Security** | Any regulatory or security gates (e.g., GDPR consent, OAuth token validation). |
| **Performance SLAs** | Expected latency or throughput constraints for each stage. |
| **Review Ar
...[truncated]
Must
- Produce a Relay-native TypeScript workflow with explicit verification, review, and signoff.
- Validate with dry-run or targeted structural checks where possible.
- Return artifacts, warnings, and follow-up commands honestly.
Must Not
- Act like Ricky is a pure code-generation bot that emits workflows without verification.
- Stop at "code compiles" as the proof bar.
- Skip skill-aware workflow authoring guidance for serious workflows.
Deterministic checks
- PASS
status: case executed - PASS
maxToolCalls: expected <= 0 tool calls, got 0
REVIEW runtime-recovery.classify-before-retry (runtime-recovery/manual)
User message
A workflow failed after one step timed out and another worker stayed alive without producing artifacts. Explain what Ricky should do next.
Ricky output
**Ricky’s next actions after a step‑timeout + orphaned worker**
| Phase | Action | Why | Artifact |
|-------|--------|-----|----------|
| 1️⃣ Detect & Log | • Capture the timeout event (step X) and the “alive‑but‑idle” worker (worker Y). <br>• Append a **FailureLog** entry to the workflow run’s audit trail (timestamp, step ID, worker ID, error code). | Provides an immutable record for post‑mortem and for the verification gate to consume. | `audit/failure‑log.json` |
| 2️⃣ Blocker Report | • Mark the run as **BLOCKED** in the orchestration DB. <br>• Emit a **BlockerTicket** (GitHub Issue or JIRA) with: <br> – Summary: *Step X timed‑out; worker Y left dangling.* <br> – Severity: **High** (no downstream artifacts). <br> – Owner: *workflow‑owner* (or auto‑assign to the “Ricky‑Ops” team). | Guarantees visibility and prevents downstream pipelines from consuming incomplete artifacts. | `issues/####-timeout-orphan.md` |
| 3️⃣ Deterministic Verification Gate | • Insert a **Gate** `gate/timeout‑orphan‑check` that must pass before any retry or continuation. <br>• Gate logic: <br> 1. Confirm no partial artifacts exist in the expected output bucket. <br> 2. Verify the worker process has b
...[truncated]
Must
- Classify the failure before deciding whether to retry.
- Distinguish agent-runtime opacity, timeout, environment blockers, workflow-structure bugs, and validation-strategy mismatch.
- Preserve raw evidence and uncertainty when the class is not known.
Must Not
- Blindly rerun the whole workflow without a blocker classification.
- Treat every failure as a broken workflow definition.
- Claim Ricky fixed the workflow before rerun evidence proves it.
Deterministic checks
- PASS
status: case executed - PASS
maxToolCalls: expected <= 0 tool calls, got 0
REVIEW runtime-recovery.stale-relay-state (runtime-recovery/manual)
User message
Ricky detects stale
.agent-relay/,.relay/, and.trajectories/state before launching a local workflow.
Ricky output
**Workflow Contract – “Stale‑State Guard for Local AgentWorkforce Runs”**
| **Element** | **Definition** |
|-------------|----------------|
| **Name** | `guard‑stale‑state` |
| **Scope** | Local developer machines before any `aw run` (or equivalent) command. |
| **Inputs** | 1. Working‑tree root (detected via `git rev‑parse --show-toplevel`).<br>2. Optional config file `.aw/guard‑stale‑state.yml` (see *Config* section). |
| **Outputs** | - Exit code `0` → clean state, workflow may proceed.<br>- Exit code `1` → stale state detected; abort run.<br>- Human‑readable report printed to `stderr` and written to `$ROOT/.aw/guard‑stale‑state/report.txt`. |
| **Side‑effects** | - May delete or archive stale directories when the user opts‑in via config.<br>- Writes a timestamped log entry to `$ROOT/.aw/guard‑stale‑state/log.json`. |
| **Deterministic Verification Gates** | 1. **Presence Check** – Verify existence of any of the three directories.<br>2. **Freshness Check** – Compare each directory’s `mtime` of the newest file against the configured freshness threshold (default = 24 h).<br>3. **Integrity Check** – Run a quick hash‑sum (`sha256sum`) of a sentinel file (`.agent‑relay/.stamp`, `.
...[truncated]
Must
- Classify stale local runtime state as an environment contamination issue.
- Recommend quarantine or isolated-run guidance before launch.
- Record the observed paths and the action taken or recommended.
Must Not
- Treat stale runtime state as a workflow logic failure.
- Delete or overwrite state without an explicit safe path or user intent.
- Continue into execution as if the workspace were clean.
Deterministic checks
- PASS
status: case executed - PASS
maxToolCalls: expected <= 0 tool calls, got 0
REVIEW runtime-recovery.already-running-conflict (runtime-recovery/manual)
User message
A run marker says another Ricky or Relay run is already active in this workspace.
Ricky output
**Fixing “Another Ricky or Relay run is already active in this workspace”**
The marker file is a safety lock that prevents two concurrent executions from stepping on each other. When it’s left behind (e.g., after a crash or an interrupted run) you’ll see the error you described. Follow the deterministic, low‑risk repair flow below.
---
## 1️⃣ Verify the Situation (Gate 1)
| Check | Command | Expected outcome |
|-------|---------|-------------------|
| Active process? | `ps aux \| grep -i ricky` | No `ricky` or `relay` PID should be listed (aside from the `grep` line). |
| Marker file existence | `ls -l .ricky/run.marker` | File exists → proceed. |
| Marker timestamp | `stat -c %y .ricky/run.marker` | Note the modification time; if it’s older than the last successful run, it’s stale. |
If a live `ricky`/`relay` process **is** running, **do not delete the marker** – let the process finish or kill it deliberately (see “Force‑stop” below).
If no process is running, the marker is orphaned and safe to clear.
---
## 2️⃣ Safe Cleanup (Gate 2)
1. **Backup the marker (optional, for audit)**
```bash
cp .ricky/run.marker .ricky/run.marker.bak_$(date +%Y%m%d%H%M%S)
...[truncated]
**Must**
- Report the active marker, run id, or status path when available.
- Ask the user to inspect, wait for, or explicitly clear the active run.
- Avoid launching a competing run that could corrupt evidence.
**Must Not**
- Silently start another run.
- Hide the existing run marker from the user.
- Treat the conflict as a generic failure with no recovery path.
**Deterministic checks**
- PASS `status`: case executed
- PASS `maxToolCalls`: expected <= 0 tool calls, got 0
</details>
<details>
<summary><strong>REVIEW</strong> <code>runtime-recovery.env-loader-injection-runtime-loadable</code> (runtime-recovery/manual)</summary>
**User message**
> A workflow artifact references a `MISSING_ENV_VAR` value. Ricky's deterministic auto-fix injects the `.env.local` / `.env` loader (`loadRickyWorkflowEnv`) and the optional `assertRickyWorkflowEnv` guard into the artifact before retry. The artifact may be a master-rendered workflow whose `.step({ command: ... })` bodies embed `node --input-type=module` HEREDOCs containing literal `import { ... } from 'node:fs'` / `from 'node:path'` strings.
**Ricky output**
```text
**Deterministic auto‑fix for missing environment variables**
Below is the concrete patch you can apply to any workflow artifact that currently fails because it references a `MISSING_ENV_VAR`. The fix does three things:
1. **Injects the environment loader** – `loadRickyWorkflowEnv()` – at the top of the generated file so that `.env.local` (or fallback `.env`) is read before any step runs.
2. **Adds an optional guard** – `assertRickyWorkflowEnv()` – that throws a clear error if the required variable is still undefined after loading.
3. **Wraps any HEREDOC‑embedded `node --input-type=module` command** with the same loader/guard so that even dynamically‑generated modules see the variables.
The patch is scoped to the single workflow file (or the master‑rendered workflow that contains the HEREDOCs). No other branches or PRs are required.
---
## 1. Helper utilities (add once to a shared `ricky-workflow-env.ts`)
```ts
// ricky-workflow-env.ts
import { config } from 'dotenv';
import { resolve } from 'node:path';
import { existsSync } from 'node:fs';
/**
* Load environment variables from .env.local (preferred) or .env.
* Returns the loaded env object for inspection.
*/
export f
...[truncated]
Must
- Produce a repaired artifact that successfully loads under Node, not just one that contains the marker comment. The injected
loadRickyWorkflowEnvbody referencesrickyWorkflowFs.*andrickyWorkflowPath.*, so the repair must also add the correspondingimport * as rickyWorkflowFs from 'node:fs'andimport * as rickyWorkflowPath from 'node:path'aliases at module top level. - Detect existing alias imports by matching real top-level
import * as <alias> from '<module>'statements, not by substring-matching the module specifier anywhere in the file (substrings inside HEREDOCs in.step({ command: ... })bodies do not count as imports). - Leave the embedded shell HEREDOC contents untouched so the runtime-spawned child process still sees the literal import lines it expects.
Must Not
- Skip adding the
rickyWorkflowFs/rickyWorkflowPathaliases becausefrom 'node:fs'orfrom 'node:path'already appears somewhere in the file as a string literal. - Inject
loadRickyWorkflowEnv(orassertRickyWorkflowEnv) without the supporting alias imports, which produces aReferenceError: rickyWorkflowPath is not definedat module load and burns the auto-fix budget onUNSUPPORTED_RUNTIME at runtime-launch. - Rewrite or escape the embedded HEREDOC text in step commands.
Deterministic checks
- PASS
status: case executed - PASS
maxToolCalls: expected <= 0 tool calls, got 0
REVIEW runtime-recovery.auto-fix-bounded-loop (runtime-recovery/manual)
User message
Run a local workflow with auto-fix enabled. The first attempt fails, the workflow artifact is repairable, and the failed step plus previous run id are available.
Ricky output
**Local Workflow Execution – Auto‑Fix Mode**
Below is a deterministic, step‑by‑step contract you can copy‑paste into your terminal (or CI script) to run the workflow, trigger the auto‑fix path on the first failure, and verify the repaired artifact before committing any changes.
---
## 1️⃣ Preconditions (inputs you already have)
| Variable | Source | Example |
|----------|--------|---------|
| `WORKFLOW_PATH` | Path to the workflow definition (e.g., `./workflows/build.yml`) | `./workflows/build.yml` |
| `FAILED_STEP_ID` | Identifier of the step that failed on the first run | `test:unit` |
| `PREV_RUN_ID` | Run‑ID of the previous successful execution (used for diff/trace) | `run‑20240512‑a1b2c3` |
| `AUTO_FIX=true` | Flag to enable the auto‑repair loop | – |
| `MAX_REPAIR_ATTEMPTS` | Upper bound for repair loops (recommended 3) | `3` |
> **Note:** All variables must be exported in the shell before invoking the runner.
```bash
export WORKFLOW_PATH=./workflows/build.yml
export FAILED_STEP_ID=test:unit
export PREV_RUN_ID=run-20240512-a1b2c3
export AUTO_FIX=true
export MAX_REPAIR_ATTEMPTS=3
2️⃣ Execution Contract
Create a single‑purpose script (`run-with-autofix
...[truncated]
**Must**
- Use a bounded retry budget and summarize every attempt.
- Ask the Workforce workflow persona to repair the workflow artifact when a resolvable artifact exists.
- Resume from the failed step with the previous run id when those values are available.
**Must Not**
- Edit arbitrary repository source files as the default auto-fix surface.
- Keep retrying after the configured max attempts.
- Lose the single Ricky tracking run id across repair/resume attempts.
**Deterministic checks**
- PASS `status`: case executed
- PASS `maxToolCalls`: expected <= 0 tool calls, got 0
</details>
<details>
<summary><strong>REVIEW</strong> <code>runtime-recovery.no-auto-fix-preserves-single-attempt</code> (runtime-recovery/manual)</summary>
**User message**
> A user runs `ricky run workflows/foo.ts --no-auto-fix` and the workflow fails.
**Ricky output**
```text
**Quick‑Start Debug Checklist for a Failing `ricky run workflows/foo.ts --no-auto-fix`**
| Step | Action | Expected Outcome | How to Verify |
|------|--------|------------------|---------------|
| 1️⃣ | **Collect the run log** | A complete, timestamped log file (`ricky‑run‑foo‑<ts>.log`) in the workspace `/.ricky/logs/` | `cat .ricky/logs/ricky‑run‑foo-*.log` |
| 2️⃣ | **Locate the failure gate** | Identify the deterministic verification gate that threw the error (e.g., `SchemaValidate`, `DependencyCheck`, `RuntimeAssert`) | Search the log for `✖ Gate` or `❌ ` markers |
| 3️⃣ | **Inspect the offending node** | The log will show the workflow node ID, source file line, and the exact exception message | Example: `NodeID: step‑transform‑1 (workflows/foo.ts:27)` |
| 4️⃣ | **Run the node in isolation** | Use `ricky run-node <node-id> --no-auto-fix` to reproduce the error without the full pipeline | `ricky run-node step‑transform‑1 --no-auto-fix` |
| 5️⃣ | **Validate contract compliance** | Ensure the node’s input/output matches the **Workflow Contract** defined in `workflows/foo.contract.ts` | `ricky validate-contract workflows/foo.contract.ts` |
| 6️⃣ | **Check external dependencies**
...[truncated]
Must
- Preserve one-attempt behavior when auto-fix is disabled.
- Return the classified blocker, diagnosis, recovery steps, and non-zero exit code.
- Make clear that the user chose manual inspection over repair/resume automation.
Must Not
- Start a repair loop despite
--no-auto-fix. - Suppress the diagnosis because no repair was attempted.
- Present the failure as a completed repair attempt.
Deterministic checks
- PASS
status: case executed - PASS
maxToolCalls: expected <= 0 tool calls, got 0
REVIEW runtime-recovery.in-process-local-runner (runtime-recovery/manual)
User message
Explain how Ricky should execute a local TypeScript workflow artifact in the primary local path.
Ricky output
**Ricky’s Standard Procedure for Executing a Local TypeScript Workflow Artifact**
Below is the concrete, repeatable contract Ricky follows when a developer asks to run a TypeScript‑based workflow that lives in the repository’s *primary local path* (by default `./workflows`). The steps are written as a deterministic “execution contract” that can be copied into a branch‑specific README or CI script and verified by the built‑in validation gates.
---
## 1️⃣ Preconditions (Contract Input)
| Item | Requirement | How Ricky Checks |
|------|-------------|------------------|
| **Repo root** | The command must be run from the repository root (contains `package.json`). | `git rev-parse --show-toplevel` |
| **Primary workflow directory** | Must exist and contain an `index.ts` (or a file matching `*.workflow.ts`). | `test -d ./workflows && test -f ./workflows/index.ts` |
| **Node version** | `>=18.x` (LTS) – matches `engines.node` in `package.json`. | `node -v` |
| **TypeScript compiler** | `ts-node` must be available (either globally or as a dev‑dependency). | `npx ts-node -v` |
| **Dependencies** | All npm packages declared in `package.json` are installed. | `npm ls` returns no missing mo
...[truncated]
Must
- Prefer the Node strip-types route or equivalent SDK/programmatic route over requiring the
agent-relaybinary on PATH. - Precheck that Node and
@agent-relay/sdkare resolvable for the workflow. - Record the actual spawn command in execution evidence.
Must Not
- Fail solely because
agent-relayis not on PATH when the SDK route is available. - Hide the actual runtime command from evidence.
- Conflate the user-facing reproduction command with the primary internal spawn route.
Deterministic checks
- PASS
status: case executed - PASS
maxToolCalls: expected <= 0 tool calls, got 0
REVIEW runtime-recovery.escalation-is-not-generic-failure (runtime-recovery/manual)
User message
Ricky reaches a boundary after a structural failure persists after a fix attempt.
Ricky output
**Ricky’s Boundary‑Handling Protocol for Persistent Structural Failures**
When a structural failure in an AgentWorkforce workflow remains after an initial remediation attempt, Ricky follows a deterministic, contract‑driven protocol that isolates the problem, escalates it cleanly, and guarantees traceability. The steps below constitute the **“Persistent‑Failure Boundary”** workflow contract.
---
## 1. Failure Detection & Initial Fix Attempt
| Phase | Action | Artifact |
|-------|--------|----------|
| **Detect** | Automated test suite (or runtime monitor) raises a *StructuralFailure* alert with a unique `FAIL‑ID`. | `failure.log` (JSON) |
| **Log** | Ricky writes a *Failure Record* (FR) to `/.ricky/failures/FAIL‑ID.json` containing: <br>• Timestamp <br>• Affected workflow name & version <br>• Failure signature (stack trace, diff) <br>• Owner (team/individual) | FR file |
| **Attempt Fix** | Ricky creates a *fix branch* `fix/FAIL‑ID‑<short‑desc>` from the latest stable `main`. <br>Applies the prescribed remediation (e.g., schema bump, dependency pin). | `fix/FAIL‑ID‑<short‑desc>` |
| **Verify** | Runs the **Deterministic Verification Gate (DVG)**: <br>1. Unit + integration test
...[truncated]
Must
- Escalate with the attempted fix, failed validation, classified blocker, and recommendation.
- Distinguish escalation from a generic product failure.
- Preserve enough context for a human operator to continue.
Must Not
- Retry speculative fixes indefinitely.
- Collapse to "something went wrong" without the attempted actions.
- Discard evidence from failed repair attempts.
Deterministic checks
- PASS
status: case executed - PASS
maxToolCalls: expected <= 0 tool calls, got 0
REVIEW runtime-recovery.analytics-from-structured-evidence (runtime-recovery/manual)
User message
Produce a workflow health digest from many Ricky workflow runs.
Ricky output
**Ricky Workflow Health Digest – Compiled from Recent Runs**
*(Generated: 2026‑05‑15 08:12 UTC)*
---
## 1. Executive Summary
| Metric | Value | Trend |
|--------|-------|-------|
| Total runs (last 24 h) | **312** | ↗ +8 % |
| Successful runs | **274** | ↗ +5 % |
| Failed runs | **38** | ↘ ‑12 % |
| Runs with **critical blockers** | **3** | ↔ steady |
| Avg. run duration | **2 m 17 s** | ↘ ‑14 s |
| Avg. verification‑gate latency | **12 s** | ↔ steady |
**Overall health:** **🟢 Green** – > 85 % success, no regression in verification‑gate pass‑rates.
---
## 2. Run‑Level Health Indicators
| Run ID | Branch | PR | Status | Verification Gates (pass/total) | Duration | Blockers |
|--------|--------|----|--------|----------------------------------|----------|----------|
| `wf‑20260514‑a1b2c3` | `feature/auto‑review` | #8421 | ✅ Success | 12/12 | 1 m 42 s | – |
| `wf‑20260514‑d4e5f6` | `bugfix/verify‑gate‑timeout` | #8399 | ❌ Failed | 9/12 | 3 m 05 s | **Gate‑Timeout** (gate #7) |
| `wf‑20260514‑g7h8i9` | `release/v2.3.0` | #8450 | ✅ Success | 12/12 | 2 m 10 s | – |
| `wf‑20260514‑j0k1l2` | `hotfix/critical‑blocker` | #8462 | ❌ Failed | 5/12 | 4 m 22 s | **Critical Blocker**
...[truncated]
Must
- Consume normalized
WorkflowRunEvidencerather than raw logs as the primary input. - Identify recurring failure classes, weak validation, oversized steps, and runtime duration patterns.
- Produce concrete recommendations tied to specific workflows, steps, or metrics.
Must Not
- Mutate evidence while analyzing it.
- Return generic advice like "improve your workflow" without references.
- Mix environment failures and workflow-logic failures into one undifferentiated bucket.
Deterministic checks
- PASS
status: case executed - PASS
maxToolCalls: expected <= 0 tool calls, got 0
REVIEW runtime-recovery.preserve-pr-shipping-during-repair (runtime-recovery/manual)
User message
A persona-authored workflow that imports
@agent-relay/github-primitiveand usescreateGitHubStepto open a PR fails its runtime-precheck. The auto-fix repair persona is invoked. Describe the contract the repaired artifact must satisfy.
Ricky output
**Repaired Workflow Contract**
| **Section** | **Requirement** | **Verification Gate** | **Pass‑Criteria** |
|-------------|----------------|-----------------------|-------------------|
| **1. Imports** | The workflow must import the **exact** package name `@agent‑relay/github-primitive` and **only** the symbols it needs. | *Static import lint* – `import { createGitHubStep } from "@agent-relay/github-primitive"` must be present and resolve without warnings. | No missing‑module or duplicate‑import errors. |
| **2. Step Construction** | A single GitHub step must be created with `createGitHubStep` that: <br>• Targets the **pull‑request** action.<br>• Supplies a **valid `repo` object** (`owner`, `repo`).<br>• Supplies a **valid `branch`** (source & target).<br>• Supplies a **non‑empty `title`** and **body**.<br>• Returns a **step object** that conforms to `GitHubStep` interface (has `id`, `run`, `outputs`). | *Type‑check* against `GitHubStep` schema + *runtime‑precheck* that the step’s `run` string is a valid GitHub CLI command. | `step.id` is a non‑empty string, `run` contains `gh pr create …`, and `outputs` includes at least `prNumber` and `prUrl`. |
| **3. Workflow Definition** | T
...[truncated]
Must
- Preserve the
@agent-relay/github-primitiveimport, theGitHubStepExecutorreference, and everycreateGitHubStep(...)invocation that the original artifact declared. - Keep the failing workflow runnable from the same path with the same
workflow(...)builder and.run({ cwd: process.cwd() })invocation. - Retain at least ceil(N / 2) of the original workflow's
.step(...)calls when the original declared four or more steps; a repair that collapses to a 2-3 step placeholder is a regression, not a fix. - Reject (do not apply) any repair output whose step list reduces to
prepare-context/runtime-precheck: true/final-signoff: echo placeholder; surface the regression diagnostic instead.
Must Not
- Strip
createGitHubStep,GitHubStepExecutor, or@agent-relay/github-primitivebecause the runtime-precheck failure mentioned PR-shipping or git side effects. The repair contract's "no commit / no push" constraint applies to the REPAIR AGENT's runtime behavior, not to the workflow's step declarations. - Emit a "minimal repair-safe master" or "simplified Ricky master" scaffold that passes the builder validator while doing none of the original work.
- Treat "the workflow now builds and runs" as success when the work it was supposed to ship is gone.
Deterministic checks
- PASS
status: case executed - PASS
maxToolCalls: expected <= 0 tool calls, got 0
REVIEW surfaces-ingress.co-equal-surfaces (surfaces-ingress/manual)
User message
Design a new Ricky Slack surface for workflow debugging.
Ricky output
## Ricky Slack Surface – “Workflow Debugger”
### 1. Overview
The **Ricky Slack Debugger** is a dedicated Slack app surface that lets engineers and product owners inspect, step‑through, and repair Ricky‑managed workflows without leaving Slack. It follows the **Ricky repository conventions** (AGENTS.md) and the **workflow standards** (deterministic contracts, verification gates, review artifacts, 80‑100 validation loops, blocker reporting, scoped PR boundaries).
---
## 2. High‑Level Architecture
| Component | Responsibility | Contract |
|-----------|----------------|----------|
| **Slack Bot (ricky‑debugger)** | Receives slash‑commands & interactive actions, formats responses, forwards requests to the backend. | `SlackEvent → BotRequest` (JSON, schema in `contracts/slack_event.json`). |
| **API Gateway (ricky‑debugger‑api)** | Auth‑checks, rate‑limits, routes to service layer. | `BotRequest → ServiceCall` (OpenAPI v3, `api/spec.yaml`). |
| **Debug Service** | Core logic: fetch workflow definition, current state, step execution, patch generation. | `DebugRequest → DebugResponse` (protobuf v3, `contracts/debug.proto`). |
| **Workflow Engine (Ricky Core)** | Executes
---
_Comment truncated to stay within GitHub limits. Download the `ricky-eval-run` artifact for the full `human-review.md`._
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
review-fix-signoff-loopto Ricky'sDEFAULT_WORKFLOW_SKILL_IDSso the generator loads it alongsidechoosing-swarm-patterns,writing-agent-relay-workflows, andrelay-80-100-workflowwhenever a spec is turned into a workflow.template-renderer.tsemits ageneration_rendering/workflow_contractevidence entry and adds a boundary-gate grep,master-workflow-renderer.tsincludes it in the master-executor fallback, andpipeline.tsrequires the rendering evidence so workflows cannot ship without recording that the dual-reviewer review-fix-signoff loop was applied..agents/skills/review-fix-signoff-loop/SKILL.md,.claude/skills/review-fix-signoff-loop/SKILL.md), the sibling skill description updates, and theprpm.lockbump so fresh checkouts discover the skill via Ricky's existing discovery roots.AGENTS.md(theCLAUDE.mdsymlink picks it up automatically) so authors know the loop pattern is now a hard convention for generated workflows.Test plan
npx tsc --noEmitnpx vitest run src/product/generation/skill-matcher.test.ts src/product/generation/pipeline.test.ts— 60 tests pass, including new assertions that pinreview-fix-signoff-loopas a default match, ageneration_renderingevidence entry witheffect: workflow_contract, and a boundary-gate grep on the skill name.🤖 Generated with Claude Code