Feedback LLM for Automated Intelligent Review is a reusable GitHub Action that integrates an LLM (such as Qwen2.5-coder) into your CI/CD pipeline to automatically analyze code diffs and post review comments on pull requests. It pre-processes diffs (adding explicit line numbers), filters out unwanted files (e.g., tests), intelligently splits large diffs, queries the LLM for detailed feedback, adjusts reported line numbers based on diff hunks, and finally posts formatted comments (with code context) to the pull request.
- Features
- Architecture & Directory Structure
- Prerequisites
- Installation
- Configuration
- Local Usage
- Using as a GitHub Action
- Cleanup Jobs
- Testing
- Contributing
- License
- Contact
-
Automated Code Review:
Analyze pull request diffs using an LLM for actionable, detailed feedback. -
Diff Preprocessing:
- Annotates diffs with explicit line numbers for clarity.
- Filters out entire file blocks based on configurable exclude patterns (e.g., test files).
-
Intelligent Diff Splitting:
Splits large diffs into manageable chunks based on file blocks and line counts. -
LLM Integration:
Constructs a detailed prompt and extracts JSON feedback (even when extra text is present). -
Line Number Adjustment:
Post-processes LLM-reported line numbers by analyzing diff hunk headers. -
Formatted Comment Publishing:
Publishes markdown-formatted comments with code context (retrieved via the GitHub API). -
Cleanup Jobs:
Provides jobs to manually delete all PR comments or only those generated by the workflow.
llm-code-review-action/
├── .env.example # Example environment variables file
├── action.yml # Action definition file for GitHub Action
├── action_usage.yml.example # Example usage file for consumers
├── requirements.txt # Python dependencies
├── README.md # This file
├── src/
│ ├── __init__.py # Marks src as a package
│ ├── config.py # Loads configuration from .env
│ ├── diff_extractor.py # Functions to retrieve, filter, and preprocess diffs
│ ├── llm_client.py # Functions for querying the LLM and processing responses
│ ├── comment_publisher.py # Main module to post comments (delegates to GitHub-specific publisher)
│ ├── comment_publisher_github.py # Publishes formatted comments to GitHub PRs; retrieves code context
│ ├── delete_pr_comments.py # Script to delete all PR comments (for cleanup)
│ ├── delete_workflow_comments.py # Script to delete workflow-generated comments (for cleanup)
│ └── (Other utility modules as needed)
└── tests/
├── test_comment_publisher_github.py
├── test_diff_extractor.py
├── test_llm_client.py
└── (Additional tests)
- Python 3.7+
- Git
- GitHub Account and Repository Access
- A valid GitHub token with appropriate permissions (read/write issues, pull-requests, etc.)
- An operational LLM server (e.g., a running instance of llama.cpp)
-
Clone the repository:
git clone https://github.com/jo15122002/flair.git cd flair -
Install dependencies:
pip install -r requirements.txt
-
Create a
.envfile:
Use the provided.env.exampleas a starting point. For example:# CI/CD Platform (use "github" or "gitlab") CI_PLATFORM=github ############################################## # GitHub Variables ############################################## REPOSITORY_GITHUB=your-github-username/your-repo # Uncomment the following line for local testing if needed # PR_NUMBER_GITHUB=1 GITHUB_TOKEN=your_github_token_here GITHUB_API_URL=https://api.github.com ############################################## # LLM Configuration ############################################## LLM_ENDPOINT=http://localhost:8080/completion LLM_MAX_TOKENS=600 ############################################## # Diff Configuration ############################################## DIFF_CHUNK_SIZE=10000 EXCLUDE_PATTERNS=test,tests,spec,example ############################################## # GitLab Variables (for future use) ############################################## GITLAB_API_URL=https://gitlab.example.com/api/v4 GITLAB_PRIVATE_TOKEN=your_gitlab_token_here
-
Ensure
src/__init__.pyexists:
This file can be empty but is required for Python to treatsrcas a package.
-
Run the Code Review Process:
python src/main.py
-
Run Tests:
pytest
This project is designed as a reusable GitHub Action. The action definition is in action.yml at the repository root.
Create a workflow file (e.g., .github/workflows/llm_code_review.yml) in your target repository with the following content:
name: "Automated Code Review"
on:
pull_request:
types: [opened, synchronize]
permissions:
issues: write
pull-requests: write
jobs:
code_review:
runs-on: ubuntu-latest
steps:
- name: Checkout Repository
uses: actions/checkout@v3
- name: Run LLM Code Review Action
uses: jo15122002/flair@v1.0.0 # Use a specific version or branch (e.g., "main")
with:
llm-endpoint: ${{ secrets.LLM_ENDPOINT }}
diff-chunk-size: 10000
exclude-patterns: "test,tests,spec"
github-token: ${{ secrets.GITHUB_TOKEN }}This will run the review process on every new pull request.
Two additional workflows are provided to clean up comments manually:
This workflow is triggered manually using workflow_dispatch and offers two jobs:
- Delete All PR Comments: Deletes every comment on the PR.
- Delete Workflow Comments: Deletes only the comments generated by this review workflow (filtered by a marker, e.g., "LLM Code Review").
Example file: .github/workflows/cleanup_comments.yml
name: "Cleanup PR Comments"
on:
workflow_dispatch:
inputs:
pr_number:
description: "Pull request number to clean up"
required: true
default: ""
jobs:
delete-pr-comments:
name: "Delete All PR Comments"
runs-on: ubuntu-latest
steps:
- name: Checkout Repository
uses: actions/checkout@v3
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: "3.8"
- name: Install Dependencies
run: pip install requests
- name: Delete PR Comments
env:
REPOSITORY_GITHUB: ${{ github.repository }}
PR_NUMBER_GITHUB: ${{ github.event.inputs.pr_number != '' && github.event.inputs.pr_number || github.event.pull_request.number }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: python src/delete_pr_comments.py
delete-workflow-comments:
name: "Delete Workflow Comments"
runs-on: ubuntu-latest
steps:
- name: Checkout Repository
uses: actions/checkout@v3
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: "3.8"
- name: Install Dependencies
run: pip install requests
- name: Delete Workflow Comments
env:
REPOSITORY_GITHUB: ${{ github.repository }}
PR_NUMBER_GITHUB: ${{ github.event.inputs.pr_number != '' && github.event.inputs.pr_number || github.event.pull_request.number }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: python src/delete_workflow_comments.pyTrigger this workflow manually from the GitHub Actions tab to perform comment cleanup.
-
Unit Tests:
Runpytestto execute all unit tests in thetestsdirectory. -
Test Coverage:
Ensure all modules (config, diff_extractor, llm_client, comment_publisher, etc.) have adequate test coverage.
Contributions are welcome! To contribute:
- Fork this repository.
- Create a new branch for your feature or fix.
- Write tests and ensure all tests pass.
- Submit a pull request with detailed explanations.
This project is licensed under the MIT License.
For questions or support, please open an issue in this repository.
This README.md is designed to be as useful as possible for new users and contributors, detailing every aspect of the project from setup to deployment and maintenance. Feel free to modify it to better suit your project's needs.