-
Notifications
You must be signed in to change notification settings - Fork 26
feat: add rule engine integration, risk labels, load balancing, and missing risk signals #66
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
22 commits
Select commit
Hold shift + click to select a range
9422eef
feat: AI-powered reviewer recommendation based on code ownership and …
leonardo1229 20ff9d2
feat: add rule engine integration, risk labels, load balancing, and m…
leonardo1229 2104c88
refactor: add RankedReviewer model and harden get_commits_for_file
leonardo1229 83ca743
fix: escape @ mentions in risk signal descriptions to prevent uninten…
leonardo1229 738515b
fix: address early feedback on reviewer recommendation agent
leonardo1229 f685a85
fix: separate cooldown check from mutation in slash command handler
leonardo1229 417dd35
test: fix ordering-dependent flakes in slash command tests
leonardo1229 408cdf6
feat: complete reviewer recommendation agent with real-world hardening
leonardo1229 0111ce1
docs: add CHANGELOG entry for reviewer recommendation agent
leonardo1229 e133012
feat: add reviewer acceptance rate tracking and recommendation succes…
leonardo1229 00ec24c
fix: show accurate rule count with truncation note in risk signals
leonardo1229 7fa74aa
refactor: fixed risk detection logic
leonardo1229 4067b82
fix: updated LLM prompt according to tc-10 test
leonardo1229 9da7440
fix: fixed some minor issue for tc-10
leonardo1229 1451952
fix: cooldown logic was be updated
leonardo1229 d80b6ee
fix: updated some risk detect function
leonardo1229 8782595
fix: always remove all stale risk labels before applying new one
leonardo1229 b0ed323
fix: udated some minor according to unit test
leonardo1229 ee7b14f
fix: updated some add_lables_to_issue issue
leonardo1229 21358c1
fix: fixed minor warning issue
leonardo1229 5cf836c
fix: added how to use /risk and /riewers in quick-start.md
leonardo1229 1e40455
fix: updated how to run github shell command
leonardo1229 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| from src.agents.reviewer_recommendation_agent.agent import ReviewerRecommendationAgent | ||
|
|
||
| __all__ = ["ReviewerRecommendationAgent"] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,96 @@ | ||
| # File: src/agents/reviewer_recommendation_agent/agent.py | ||
|
|
||
| from typing import Any | ||
|
|
||
| import structlog | ||
| from langgraph.graph import END, StateGraph | ||
|
|
||
| from src.agents.base import AgentResult, BaseAgent | ||
| from src.agents.reviewer_recommendation_agent import nodes | ||
| from src.agents.reviewer_recommendation_agent.models import RecommendationState | ||
|
|
||
| logger = structlog.get_logger() | ||
|
|
||
|
|
||
| class ReviewerRecommendationAgent(BaseAgent): | ||
| """ | ||
| Agent that recommends reviewers for a PR based on: | ||
| 1. CODEOWNERS ownership of changed files | ||
| 2. Commit history expertise (who recently touched the same files) | ||
| 3. Deterministic risk assessment (file count, sensitive paths, contributor status) | ||
| 4. LLM-powered ranking with natural-language reasoning | ||
|
|
||
| Outputs both a risk breakdown and ranked reviewer suggestions. | ||
| """ | ||
|
|
||
| def __init__(self) -> None: | ||
| super().__init__(agent_name="reviewer_recommendation") | ||
|
|
||
| def _build_graph(self) -> Any: | ||
| workflow: StateGraph[RecommendationState] = StateGraph(RecommendationState) | ||
|
|
||
| llm = self.llm | ||
|
|
||
| async def _recommend_reviewers(state: RecommendationState) -> RecommendationState: | ||
| return await nodes.recommend_reviewers(state, llm) | ||
|
|
||
| workflow.add_node("fetch_pr_data", nodes.fetch_pr_data) | ||
| workflow.add_node("assess_risk", nodes.assess_risk) | ||
| workflow.add_node("recommend_reviewers", _recommend_reviewers) | ||
|
|
||
| workflow.set_entry_point("fetch_pr_data") | ||
| workflow.add_edge("fetch_pr_data", "assess_risk") | ||
| workflow.add_edge("assess_risk", "recommend_reviewers") | ||
| workflow.add_edge("recommend_reviewers", END) | ||
|
|
||
| return workflow.compile() | ||
|
|
||
| async def execute(self, **kwargs: Any) -> AgentResult: | ||
| """ | ||
| Args: | ||
| repo_full_name: str — owner/repo | ||
| pr_number: int — PR number | ||
| installation_id: int — GitHub App installation ID | ||
| """ | ||
| repo_full_name: str | None = kwargs.get("repo_full_name") | ||
| pr_number: int | None = kwargs.get("pr_number") | ||
| installation_id: int | None = kwargs.get("installation_id") | ||
|
|
||
| if not repo_full_name or not pr_number or not installation_id: | ||
| return AgentResult(success=False, message="repo_full_name, pr_number, and installation_id are required") | ||
|
|
||
| initial_state = RecommendationState( | ||
| repo_full_name=repo_full_name, | ||
| pr_number=pr_number, | ||
| installation_id=installation_id, | ||
| ) | ||
|
|
||
| try: | ||
| result = await self._execute_with_timeout(self.graph.ainvoke(initial_state), timeout=45.0) | ||
| final_state = RecommendationState(**result) if isinstance(result, dict) else result | ||
|
|
||
| if final_state.error: | ||
| return AgentResult(success=False, message=final_state.error) | ||
|
|
||
| return AgentResult( | ||
| success=True, | ||
| message="Recommendation complete", | ||
| data={ | ||
| "risk_level": final_state.risk_level, | ||
| "risk_score": final_state.risk_score, | ||
| "risk_signals": [s.model_dump() for s in final_state.risk_signals], | ||
| "candidates": [c.model_dump() for c in final_state.candidates], | ||
| "llm_ranking": final_state.llm_ranking.model_dump() if final_state.llm_ranking else None, | ||
| "pr_files_count": len(final_state.pr_files), | ||
| "pr_author": final_state.pr_author, | ||
| "codeowners_team_slugs": final_state.codeowners_team_slugs, | ||
| "pr_base_branch": final_state.pr_base_branch, | ||
| }, | ||
| ) | ||
|
|
||
| except TimeoutError: | ||
| logger.error("agent_execution_timeout", agent="reviewer_recommendation", repo=repo_full_name) | ||
| return AgentResult(success=False, message="Recommendation timed out after 45 seconds") | ||
| except Exception as e: | ||
| logger.exception("agent_execution_failed", agent="reviewer_recommendation", error=str(e)) | ||
| return AgentResult(success=False, message=str(e)) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,85 @@ | ||
| # File: src/agents/reviewer_recommendation_agent/models.py | ||
|
|
||
| from typing import Any | ||
|
|
||
| from pydantic import BaseModel, Field | ||
|
|
||
|
|
||
| class ReviewerCandidate(BaseModel): | ||
| """A candidate reviewer with a score and reasons for recommendation.""" | ||
|
|
||
| username: str | ||
| score: int = 0 | ||
| ownership_pct: int = 0 # % of changed files they own or recently touched | ||
| reasons: list[str] = Field(default_factory=list) | ||
|
|
||
|
|
||
| class RiskSignal(BaseModel): | ||
| """A single contributing factor to the PR risk score.""" | ||
|
|
||
| label: str | ||
| description: str | ||
| points: int | ||
|
|
||
|
|
||
| class RankedReviewer(BaseModel): | ||
| """A single reviewer entry in the LLM ranking output.""" | ||
|
|
||
| username: str = Field(description="GitHub username of the reviewer") | ||
| reason: str = Field(description="Short explanation of why this reviewer is recommended") | ||
|
|
||
|
|
||
| class LLMReviewerRanking(BaseModel): | ||
| """Structured output from the LLM reviewer ranking step.""" | ||
|
|
||
| ranked_reviewers: list[RankedReviewer] = Field(description="Ordered list of reviewers, best match first") | ||
| summary: str = Field(description="One-line overall recommendation summary") | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
|
|
||
|
|
||
| class RecommendationState(BaseModel): | ||
| """Shared state (blackboard) for the ReviewerRecommendationAgent graph.""" | ||
|
|
||
| # --- Inputs --- | ||
| repo_full_name: str | ||
| pr_number: int | ||
| installation_id: int | ||
|
|
||
| # --- Collected Data --- | ||
| pr_files: list[str] = Field(default_factory=list) | ||
| pr_author: str = "" | ||
| pr_additions: int = 0 | ||
| pr_deletions: int = 0 | ||
| pr_commits_count: int = 0 | ||
| pr_author_association: str = "NONE" | ||
| codeowners_content: str | None = None | ||
| contributors: list[dict[str, Any]] = Field(default_factory=list) | ||
| # file_path -> list of recent committer logins | ||
| file_experts: dict[str, list[str]] = Field(default_factory=dict) | ||
| # Matched Watchflow rules (description, severity) loaded from .watchflow/rules.yaml | ||
| matched_rules: list[dict[str, str]] = Field(default_factory=list) | ||
| # Recent review activity: login -> count of reviews on recent PRs (for load balancing) | ||
| reviewer_load: dict[str, int] = Field(default_factory=dict) | ||
| # Reviewer acceptance rates: login -> approval rate (0.0–1.0) from recent PRs | ||
| reviewer_acceptance_rates: dict[str, float] = Field(default_factory=dict) | ||
| # PR title (for revert detection) | ||
| pr_title: str = "" | ||
|
|
||
| # --- Risk Assessment --- | ||
| risk_score: int = 0 | ||
| risk_level: str = "low" # low / medium / high / critical | ||
| risk_signals: list[RiskSignal] = Field(default_factory=list) | ||
|
|
||
| # --- Recommendations --- | ||
| candidates: list[ReviewerCandidate] = Field(default_factory=list) | ||
| llm_ranking: LLMReviewerRanking | None = None | ||
|
|
||
| # PR base branch (used when writing .watchflow/expertise.json) | ||
| pr_base_branch: str = "main" | ||
| # Team slugs extracted from CODEOWNERS (@org/team entries) — used to split | ||
| # reviewer assignment into `reviewers` vs `team_reviewers` GitHub API fields | ||
| codeowners_team_slugs: list[str] = Field(default_factory=list) | ||
| # Persisted expertise profiles loaded from .watchflow/expertise.json | ||
| expertise_profiles: dict[str, Any] = Field(default_factory=dict) | ||
|
|
||
| # --- Execution Metadata --- | ||
| error: str | None = None | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Emit the required confidence contract before downstream automation.
The returned payload has risk/reviewer data only. It never exposes
decision,confidence,reasoning,recommendations, orstrategy_used, so webhook handlers cannot enforce the required<0.5human-in-the-loop policy before posting comments and labels. Add those fields to the result payload and gate auto-actions on confidence.As per coding guidelines, "Agent outputs must include:
decision,confidence (0..1), shortreasoning,recommendations,strategy_used" and "Implement confidence policy: reject or route to human-in-the-loop when confidence < 0.5".🤖 Prompt for AI Agents