[Phase 2.5-F] End-to-end CLI validation with cf-test project

[frankbria]

Summary

Validate the ReAct engine end-to-end by running it against the cf-test project (task tracker CLI from requirements.md). This is the primary acceptance test for the entire Phase 2.5 redesign.

Acceptance Criteria

• Clean cf-test directory: remove previous .codeframe workspace and generated code
• Run full Golden Path with ReAct engine:
  1. `cf init ~/projects/cf-test --detect`
  2. `cf prd add requirements.md`
  3. `cf tasks generate`
  4. Mark tasks READY
  5. `cf work start --execute --engine react` for each task
• Success criteria (from unified plan):
  • Build working task tracker CLI with tests, on first attempt
  • 0 ruff lint errors
  • pyproject.toml preserved (not overwritten)
  • No cross-file naming mismatches
  • Lint errors caught per-edit and fixed in next iteration
  • Each task completes within 30 iterations
• Document results: success rate, iteration counts, token usage, any failures
• Compare with Plan-and-Execute engine on same tasks (cost, success rate, quality)
• All existing codeframe tests still pass

Implementation Notes

• Use the existing cf-test project at ~/projects/cf-test with requirements.md
• Source ANTHROPIC_API_KEY from /home/frankbria/projects/codeframe/.env
• Run with --verbose for detailed output
• Document any issues found and file follow-up issues if needed

Dependencies

• All of Phases A-E must be complete ([Phase 2.5-A] Implement SearchReplaceEditor with fuzzy matching #343-[Phase 2.5-E] Implement token budget tracking and message compaction #352)

Reference

• docs/AGENT_V3_UNIFIED_PLAN.md (Success Criteria)
• ~/projects/cf-test/requirements.md (test project PRD)

[traycerai]

Plan

Observations

The codebase currently implements a Plan-and-Execute pattern where file:codeframe/core/planner.py generates a complete implementation plan upfront, then file:codeframe/core/executor.py executes each step sequentially. The cf work start command in file:codeframe/cli/app.py (lines 1957-2091) has an --execute flag but no --engine parameter. Phase 2.5 with the ReAct engine and issues #343-#352 (Phases A-E) are not yet implemented - this issue describes validation work that depends on completing those prerequisite phases first. The cf-test project at ~/projects/cf-test with requirements.md has been used for CLI testing before (referenced in file:docs/CLI_V2_TEST_REPORT.md).

Approach

This plan assumes Phases 2.5-A through 2.5-E are complete and focuses on Phase 2.5-F (end-to-end validation). The approach implements a comprehensive test harness that runs the Golden Path workflow against cf-test using both the ReAct engine (new) and Plan-and-Execute engine (baseline), then compares results across multiple dimensions: success rate, iteration count, token usage, code quality (ruff lint), and architectural preservation (pyproject.toml). The validation framework will be reusable for future engine comparisons and regression testing.

Implementation Steps

1. Prerequisites Verification

Verify Phase 2.5-A through 2.5-E completion:

• Confirm ReAct engine implementation exists in file:codeframe/core/ (likely react_agent.py or similar)
• Verify --engine flag added to cf work start command in file:codeframe/cli/app.py
• Check for docs/AGENT_V3_UNIFIED_PLAN.md with success criteria definitions
• Validate that ReAct engine supports the same file:codeframe/core/agent.py interface (run/resume methods)
• Ensure both engines emit compatible events for monitoring

2. Test Environment Setup

Create validation test harness:

• Add new module file:tests/e2e/test_phase25_validation.py for end-to-end validation
• Create helper script file:scripts/validate_phase25.py for manual testing
• Define test fixtures in file:tests/e2e/conftest.py:
  • cf_test_project_path fixture pointing to ~/projects/cf-test
  • clean_cf_test fixture that removes .codeframe/ and generated code
  • anthropic_api_key fixture loading from /home/frankbria/projects/codeframe/.env

Prepare cf-test project:

• Document expected cf-test structure in file:tests/e2e/cf_test_structure.md
• Create baseline snapshot of ~/projects/cf-test/pyproject.toml for preservation checks
• Define expected task list from requirements.md parsing

3. Golden Path Workflow Implementation

Implement automated Golden Path runner:

• Create file:tests/e2e/golden_path_runner.py with GoldenPathRunner class
• Implement workflow steps as methods:
  • run_init() - Execute cf init ~/projects/cf-test --detect
  • add_prd() - Execute cf prd add requirements.md
  • generate_tasks() - Execute cf tasks generate
  • mark_tasks_ready() - Set all generated tasks to READY status
  • execute_task(task_id, engine) - Run cf work start <task-id> --execute --engine <engine>
• Add result collection for each step (exit codes, stdout/stderr, timing)
• Implement verbose logging mode matching --verbose flag behavior

4. Success Criteria Validation

Implement validation checks from unified plan:

• Create file:tests/e2e/validators.py with validation functions:
  • validate_build_success() - Verify task tracker CLI builds and runs
  • validate_ruff_lint() - Run ruff check . and assert 0 errors
  • validate_pyproject_preserved() - Compare against baseline snapshot
  • validate_no_naming_mismatches() - Check cross-file import consistency
  • validate_iteration_count(max_iterations=30) - Ensure tasks complete within limit
• Parse agent logs to extract per-edit lint error detection
• Implement test execution check (verify tests exist and pass)

5. Metrics Collection Framework

Build comprehensive metrics tracking:

• Create file:codeframe/core/metrics.py with EngineMetrics dataclass:
  • success_rate: float - Percentage of tasks completed successfully
  • iteration_counts: Dict[str, int] - Iterations per task
  • token_usage: Dict[str, int] - Total tokens by purpose (planning/execution)
  • duration_seconds: float - Total execution time
  • lint_errors_initial: int - Errors before self-correction
  • lint_errors_final: int - Errors after completion
• Integrate with existing file:codeframe/adapters/llm.py token tracking
• Add metrics export to JSON for comparison analysis

6. Engine Comparison Implementation

Create side-by-side comparison:

• Implement file:tests/e2e/engine_comparator.py with EngineComparator class
• Run Golden Path twice (once per engine) with clean cf-test between runs
• Collect metrics for both engines using EngineMetrics
• Generate comparison report with:
  • Success rate delta
  • Average iteration count comparison
  • Token usage breakdown (total, per-task, by purpose)
  • Code quality comparison (lint errors, test coverage)
  • Execution time comparison
• Create visualization using rich tables for CLI output

7. Results Documentation

Implement automated reporting:

• Create file:tests/e2e/report_generator.py to generate markdown reports
• Template structure in file:tests/e2e/templates/phase25_report.md.j2:
  • Executive summary (pass/fail, key metrics)
  • Detailed results per engine
  • Task-by-task breakdown
  • Failure analysis (if any)
  • Token usage breakdown
  • Recommendations
• Save reports to file:.codeframe/validation_reports/phase25_<timestamp>.md
• Include raw metrics JSON for further analysis

8. CI Integration

Add to existing test suite:

• Update file:pytest.ini to mark phase25 tests with @pytest.mark.phase25
• Add file:.github/workflows/phase25_validation.yml (if CI exists):
  • Trigger on manual dispatch or release tags
  • Set up cf-test project in CI environment
  • Run validation with both engines
  • Upload reports as artifacts
• Document manual validation procedure in file:docs/PHASE25_VALIDATION.md

9. Failure Handling & Debugging

Implement robust error handling:

• Add detailed logging in GoldenPathRunner for each step
• Capture full agent logs when --execute runs
• Implement checkpoint/resume for long-running validations
• Create file:scripts/debug_phase25_failure.py to analyze failed runs:
  • Parse agent logs for blocker patterns
  • Extract LLM prompts and responses
  • Identify iteration where failure occurred
• Add --continue-on-failure flag to run all tasks even if some fail

10. Regression Test Suite

Ensure existing tests pass:

• Run full test suite: pytest tests/ -v
• Specifically verify:
  • file:tests/cli/test_v2_cli_integration.py - CLI integration tests
  • file:tests/core/test_executor.py - Executor tests
  • file:tests/core/test_planner.py - Planner tests
  • file:tests/core/test_agent.py - Agent orchestration tests
• Fix any regressions introduced by Phase 2.5 changes
• Add regression tests for --engine flag parameter validation

sequenceDiagram
    participant Test as Test Harness
    participant CF as cf CLI
    participant React as ReAct Engine
    participant PlanExec as Plan-Execute Engine
    participant Validator as Validators
    participant Report as Report Generator

    Note over Test,Report: Phase 2.5-F Validation Flow

    Test->>CF: cf init ~/projects/cf-test --detect
    Test->>CF: cf prd add requirements.md
    Test->>CF: cf tasks generate
    Test->>Test: Mark all tasks READY

    rect rgb(200, 220, 255)
        Note over Test,React: ReAct Engine Run
        loop For each task
            Test->>CF: cf work start <task-id> --execute --engine react
            CF->>React: Execute with ReAct pattern
            React-->>CF: Task result + metrics
            CF-->>Test: Completion status
        end
        Test->>Validator: Validate success criteria
        Validator-->>Test: ReAct metrics
    end

    Test->>Test: Clean cf-test workspace

    rect rgb(255, 220, 200)
        Note over Test,PlanExec: Plan-Execute Engine Run
        loop For each task
            Test->>CF: cf work start <task-id> --execute --engine plan-execute
            CF->>PlanExec: Execute with Plan-Execute pattern
            PlanExec-->>CF: Task result + metrics
            CF-->>Test: Completion status
        end
        Test->>Validator: Validate success criteria
        Validator-->>Test: Plan-Execute metrics
    end

    Test->>Report: Generate comparison report
    Report-->>Test: Markdown + JSON output

Loading

Validation Checklist

Before running validation:

• Verify ANTHROPIC_API_KEY in /home/frankbria/projects/codeframe/.env
• Confirm ~/projects/cf-test/requirements.md exists and is up-to-date
• Ensure ruff is installed: pip install ruff
• Back up cf-test project if it contains important work
• Review docs/AGENT_V3_UNIFIED_PLAN.md for latest success criteria

During validation:

• Monitor token usage to avoid unexpected API costs
• Check verbose output for blocker creation patterns
• Verify self-correction loop activates on lint errors
• Confirm pyproject.toml is not overwritten

After validation:

• Review generated code quality manually
• Run task tracker CLI to verify functionality
• Compare iteration counts between engines
• Document any unexpected failures or issues
• File follow-up issues for any bugs discovered

Import In IDE

🤖 Prompt for AI Agents

## Observations

The codebase currently implements a **Plan-and-Execute** pattern where `file:codeframe/core/planner.py` generates a complete implementation plan upfront, then `file:codeframe/core/executor.py` executes each step sequentially. The `cf work start` command in `file:codeframe/cli/app.py` (lines 1957-2091) has an `--execute` flag but **no `--engine` parameter**. Phase 2.5 with the ReAct engine and issues #343-#352 (Phases A-E) are **not yet implemented** - this issue describes validation work that depends on completing those prerequisite phases first. The cf-test project at `~/projects/cf-test` with `requirements.md` has been used for CLI testing before (referenced in `file:docs/CLI_V2_TEST_REPORT.md`).

## Approach

This plan assumes **Phases 2.5-A through 2.5-E are complete** and focuses on Phase 2.5-F (end-to-end validation). The approach implements a comprehensive test harness that runs the Golden Path workflow against cf-test using both the ReAct engine (new) and Plan-and-Execute engine (baseline), then compares results across multiple dimensions: success rate, iteration count, token usage, code quality (ruff lint), and architectural preservation (pyproject.toml). The validation framework will be reusable for future engine comparisons and regression testing.

## Implementation Steps

### 1. Prerequisites Verification

**Verify Phase 2.5-A through 2.5-E completion:**
- Confirm ReAct engine implementation exists in `file:codeframe/core/` (likely `react_agent.py` or similar)
- Verify `--engine` flag added to `cf work start` command in `file:codeframe/cli/app.py`
- Check for `docs/AGENT_V3_UNIFIED_PLAN.md` with success criteria definitions
- Validate that ReAct engine supports the same `file:codeframe/core/agent.py` interface (run/resume methods)
- Ensure both engines emit compatible events for monitoring

### 2. Test Environment Setup

**Create validation test harness:**
- Add new module `file:tests/e2e/test_phase25_validation.py` for end-to-end validation
- Create helper script `file:scripts/validate_phase25.py` for manual testing
- Define test fixtures in `file:tests/e2e/conftest.py`:
  - `cf_test_project_path` fixture pointing to `~/projects/cf-test`
  - `clean_cf_test` fixture that removes `.codeframe/` and generated code
  - `anthropic_api_key` fixture loading from `/home/frankbria/projects/codeframe/.env`

**Prepare cf-test project:**
- Document expected cf-test structure in `file:tests/e2e/cf_test_structure.md`
- Create baseline snapshot of `~/projects/cf-test/pyproject.toml` for preservation checks
- Define expected task list from `requirements.md` parsing

### 3. Golden Path Workflow Implementation

**Implement automated Golden Path runner:**
- Create `file:tests/e2e/golden_path_runner.py` with `GoldenPathRunner` class
- Implement workflow steps as methods:
  - `run_init()` - Execute `cf init ~/projects/cf-test --detect`
  - `add_prd()` - Execute `cf prd add requirements.md`
  - `generate_tasks()` - Execute `cf tasks generate`
  - `mark_tasks_ready()` - Set all generated tasks to READY status
  - `execute_task(task_id, engine)` - Run `cf work start <task-id> --execute --engine <engine>`
- Add result collection for each step (exit codes, stdout/stderr, timing)
- Implement verbose logging mode matching `--verbose` flag behavior

### 4. Success Criteria Validation

**Implement validation checks from unified plan:**
- Create `file:tests/e2e/validators.py` with validation functions:
  - `validate_build_success()` - Verify task tracker CLI builds and runs
  - `validate_ruff_lint()` - Run `ruff check .` and assert 0 errors
  - `validate_pyproject_preserved()` - Compare against baseline snapshot
  - `validate_no_naming_mismatches()` - Check cross-file import consistency
  - `validate_iteration_count(max_iterations=30)` - Ensure tasks complete within limit
- Parse agent logs to extract per-edit lint error detection
- Implement test execution check (verify tests exist and pass)

### 5. Metrics Collection Framework

**Build comprehensive metrics tracking:**
- Create `file:codeframe/core/metrics.py` with `EngineMetrics` dataclass:
  - `success_rate: float` - Percentage of tasks completed successfully
  - `iteration_counts: Dict[str, int]` - Iterations per task
  - `token_usage: Dict[str, int]` - Total tokens by purpose (planning/execution)
  - `duration_seconds: float` - Total execution time
  - `lint_errors_initial: int` - Errors before self-correction
  - `lint_errors_final: int` - Errors after completion
- Integrate with existing `file:codeframe/adapters/llm.py` token tracking
- Add metrics export to JSON for comparison analysis

### 6. Engine Comparison Implementation

**Create side-by-side comparison:**
- Implement `file:tests/e2e/engine_comparator.py` with `EngineComparator` class
- Run Golden Path twice (once per engine) with clean cf-test between runs
- Collect metrics for both engines using `EngineMetrics`
- Generate comparison report with:
  - Success rate delta
  - Average iteration count comparison
  - Token usage breakdown (total, per-task, by purpose)
  - Code quality comparison (lint errors, test coverage)
  - Execution time comparison
- Create visualization using `rich` tables for CLI output

### 7. Results Documentation

**Implement automated reporting:**
- Create `file:tests/e2e/report_generator.py` to generate markdown reports
- Template structure in `file:tests/e2e/templates/phase25_report.md.j2`:
  - Executive summary (pass/fail, key metrics)
  - Detailed results per engine
  - Task-by-task breakdown
  - Failure analysis (if any)
  - Token usage breakdown
  - Recommendations
- Save reports to `file:.codeframe/validation_reports/phase25_<timestamp>.md`
- Include raw metrics JSON for further analysis

### 8. CI Integration

**Add to existing test suite:**
- Update `file:pytest.ini` to mark phase25 tests with `@pytest.mark.phase25`
- Add `file:.github/workflows/phase25_validation.yml` (if CI exists):
  - Trigger on manual dispatch or release tags
  - Set up cf-test project in CI environment
  - Run validation with both engines
  - Upload reports as artifacts
- Document manual validation procedure in `file:docs/PHASE25_VALIDATION.md`

### 9. Failure Handling & Debugging

**Implement robust error handling:**
- Add detailed logging in `GoldenPathRunner` for each step
- Capture full agent logs when `--execute` runs
- Implement checkpoint/resume for long-running validations
- Create `file:scripts/debug_phase25_failure.py` to analyze failed runs:
  - Parse agent logs for blocker patterns
  - Extract LLM prompts and responses
  - Identify iteration where failure occurred
- Add `--continue-on-failure` flag to run all tasks even if some fail

### 10. Regression Test Suite

**Ensure existing tests pass:**
- Run full test suite: `pytest tests/ -v`
- Specifically verify:
  - `file:tests/cli/test_v2_cli_integration.py` - CLI integration tests
  - `file:tests/core/test_executor.py` - Executor tests
  - `file:tests/core/test_planner.py` - Planner tests
  - `file:tests/core/test_agent.py` - Agent orchestration tests
- Fix any regressions introduced by Phase 2.5 changes
- Add regression tests for `--engine` flag parameter validation

```mermaid
sequenceDiagram
    participant Test as Test Harness
    participant CF as cf CLI
    participant React as ReAct Engine
    participant PlanExec as Plan-Execute Engine
    participant Validator as Validators
    participant Report as Report Generator

    Note over Test,Report: Phase 2.5-F Validation Flow

    Test->>CF: cf init ~/projects/cf-test --detect
    Test->>CF: cf prd add requirements.md
    Test->>CF: cf tasks generate
    Test->>Test: Mark all tasks READY

    rect rgb(200, 220, 255)
        Note over Test,React: ReAct Engine Run
        loop For each task
            Test->>CF: cf work start <task-id> --execute --engine react
            CF->>React: Execute with ReAct pattern
            React-->>CF: Task result + metrics
            CF-->>Test: Completion status
        end
        Test->>Validator: Validate success criteria
        Validator-->>Test: ReAct metrics
    end

    Test->>Test: Clean cf-test workspace

    rect rgb(255, 220, 200)
        Note over Test,PlanExec: Plan-Execute Engine Run
        loop For each task
            Test->>CF: cf work start <task-id> --execute --engine plan-execute
            CF->>PlanExec: Execute with Plan-Execute pattern
            PlanExec-->>CF: Task result + metrics
            CF-->>Test: Completion status
        end
        Test->>Validator: Validate success criteria
        Validator-->>Test: Plan-Execute metrics
    end

    Test->>Report: Generate comparison report
    Report-->>Test: Markdown + JSON output
```

### Validation Checklist

**Before running validation:**
- [ ] Verify ANTHROPIC_API_KEY in `/home/frankbria/projects/codeframe/.env`
- [ ] Confirm `~/projects/cf-test/requirements.md` exists and is up-to-date
- [ ] Ensure ruff is installed: `pip install ruff`
- [ ] Back up cf-test project if it contains important work
- [ ] Review `docs/AGENT_V3_UNIFIED_PLAN.md` for latest success criteria

**During validation:**
- [ ] Monitor token usage to avoid unexpected API costs
- [ ] Check verbose output for blocker creation patterns
- [ ] Verify self-correction loop activates on lint errors
- [ ] Confirm pyproject.toml is not overwritten

**After validation:**
- [ ] Review generated code quality manually
- [ ] Run task tracker CLI to verify functionality
- [ ] Compare iteration counts between engines
- [ ] Document any unexpected failures or issues
- [ ] File follow-up issues for any bugs discovered

---

Execution Information

Branch: main
Commit: 99f0ea9

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

🔗 Related PRs

#125 - Implement WorkerAgent execute_task method [merged]
#191 - feat(e2e): Enable conditional tests and fix discovery startup [merged]
#229 - feat(e2e): add state reconciliation tests for late-joining users [merged]
#246 - feat(agents): trigger multi-agent execution after task approval [merged]
#317 - feat(cli): add interactive PRD generation via Socratic discovery [merged]

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!