feat: add exhaustive Factory feature verification#86
Conversation
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
|
Warning Review limit reached
Next review available in: 27 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughAdds a checked-in Factory feature catalog, tiered verification documentation, an hourly Slack feature guardian with persisted progress, and a workflow that runs deterministic and optional live verification tiers with aggregated results. ChangesFactory verification
Estimated code review effort: 5 (Critical) | ~90+ minutes Sequence Diagram(s)sequenceDiagram
participant Scheduler
participant GuardianHandler
participant ManifestLoader
participant WorkspaceMemory
participant Slack
Scheduler->>GuardianHandler: invoke hourly handler
GuardianHandler->>ManifestLoader: load and validate manifest
GuardianHandler->>WorkspaceMemory: read checked feature progress
GuardianHandler->>Slack: post selected feature check
GuardianHandler->>WorkspaceMemory: persist updated cycle progress
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Catalog Factory's full CLI, orchestration, integration, API, and configuration surface; document critical paths and tiered procedures; and add proactive guardian and deterministic verification workflows. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
8029f30 to
7c06cc9
Compare
|
@coderabbitai review |
✅ Action performedReview finished.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.agentworkforce/agents/factory-feature-guardian/agent.ts:
- Around line 79-100: Clear the inherited categoryCriticality state at each
category heading in the guardian parser before processing the heading’s
criticality, so a missing value is rejected instead of reusing the previous
category’s value. Also update workflows/verify-features.ts lines 137-159 to
validate every category criticality against critical, hot, or standard,
preserving the existing manifest validation flow.
- Around line 286-299: Update the fallback message in the LLM completion catch
block to include feature.location alongside featureSurface(feature), preserving
the guardian’s exact source-surface information while leaving the existing
fallback structure unchanged.
In `@workflows/verify-features.ts`:
- Around line 424-464: Make the tier parsing in the verification collector fail
closed: when processing tier N, only accept an exact final marker for that same
tier with no suffix (for example, TIER1_PASS), and classify mismatched or
malformed markers such as TIER2_PASS or TIER1_PASS_EXTRA as UNKNOWN with FAIL=1.
Update the NOT_RUN self-test invocation to pass zero so it verifies
overall_result’s grep guard rather than being unconditionally failed by the
supplied fail_count.
- Around line 476-496: Update the workflow completion handling after wf.run() so
a report command that exits nonzero propagates failure to the process exit code.
Either enable failure propagation on the report step or inspect the returned
WorkflowRunRow and throw when its status is not completed, while preserving
successful completion behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: af119005-7d2c-4647-814b-c70d2ff9a9d5
📒 Files selected for processing (7)
.agentworkforce/agents/factory-feature-guardian/agent.ts.agentworkforce/agents/factory-feature-guardian/persona.json.agentworkforce/features/critical-paths.md.agentworkforce/features/manifest.yaml.agentworkforce/features/verify/procedures.md.gitignoreworkflows/verify-features.ts
2 participants
Summary
Final-main coverage
~in cloneRoot/clonePaths, and default clonePath to cwd for single-repo local runs #78 clone path home expansion, one-repo cwd inference, and local checkout preflightdispatch <number>can't resolve GitHub issues: resolver reads one page of listTree; issueSource auto-detect silently picks Linear #81 numeric GitHub issue resolution, source selection, compact/nested/by-id/meta/flat path shapes, ambiguity, and provider failuresar-<number>-reviewis not repo-qualified (implementer is) #82 repo-qualified GitHub agent identities and composite repo/issue lifecycle state[factory] error {}— raw Errors serialize to {} (message/stack are non-enumerable); describeError() already exists one line away #83 bounded, redacted, hook-safe Error/log serializationValidation
Independent audit