Skip to content

Add feedback reminder for CodeRabbit#4932

Merged
ykmr1224 merged 1 commit intoopensearch-project:mainfrom
ykmr1224:coderabbit-feedback
Dec 11, 2025
Merged

Add feedback reminder for CodeRabbit#4932
ykmr1224 merged 1 commit intoopensearch-project:mainfrom
ykmr1224:coderabbit-feedback

Conversation

@ykmr1224
Copy link
Copy Markdown
Collaborator

Description

  • Add feedback reminder for CodeRabbit
    • When CodeRabbit made review comment, it automatically adds reminder message to leave feedback. 👋 Leave emoji reaction (👍/👎) to track effectiveness of CodeRabbit.
    • Later we can take stats on how many comments got positive/negative feedback via Github API, and make improvement based on the result.

Related Issues

n/a

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • New PPL command checklist all confirmed.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff or -s.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Dec 10, 2025

📝 Walkthrough

Summary by CodeRabbit

  • Chores
    • Added an automated feedback mechanism to code review comments to improve review process efficiency.

✏️ Tip: You can customize this high-level summary in your review settings.

Walkthrough

A new GitHub Actions workflow is added that automatically appends a feedback prompt to code review comments created by coderabbitai[bot]. The workflow triggers on pull request review comments and invites users to leave emoji reactions to gauge effectiveness.

Changes

Cohort / File(s) Change Summary
GitHub Actions Workflow
\.github/workflows/coderabbit-feedback\.yml
New workflow file that triggers on pull_request_review_comment events and appends a feedback collection message to comments authored by coderabbitai[bot] if not already present

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

  • Workflow trigger logic: Verify the pull_request_review_comment event and coderabbitai[bot] author filter are configured correctly
  • Feedback message duplication check: Confirm the conditional logic properly prevents duplicate message appends
  • GitHub API permissions: Ensure the workflow has necessary permissions to read and update comments

Suggested reviewers

  • ps48
  • kavithacm
  • derek-ho
  • joshuali925
  • penghuo
  • qianheng-aws

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title 'Add feedback reminder for CodeRabbit' directly and clearly summarizes the main change in the pull request—adding a GitHub Actions workflow that appends feedback prompts to CodeRabbit's review comments.
Description check ✅ Passed The description is directly related to the changeset, explaining the purpose of the feedback reminder workflow and the intended use case for collecting effectiveness metrics via emoji reactions.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (1)
.github/workflows/coderabbit-feedback.yml (1)

12-28: Add error handling and logging to the script.

The script lacks error handling for the API call, and there's no logging to aid debugging if the update fails. If the comment update encounters an error (e.g., rate limit, permission issue), the workflow will silently fail without useful feedback.

Apply this diff to add error handling and logging:

       - uses: actions/github-script@v8
         with:
           github-token: ${{ secrets.GITHUB_TOKEN }}
           script: |
             const comment = context.payload.comment;
             const originalBody = comment.body;
             const feedbackMessage = "\n\n<sub>👋 Leave emoji reaction (👍/👎) to track effectiveness of CodeRabbit.</sub>";
 
             // Only add the message if it's not already there
             if (!originalBody.includes(feedbackMessage)) {
+              try {
                 await github.rest.pulls.updateReviewComment({
                   owner: context.repo.owner,
                   repo: context.repo.repo,
                   comment_id: comment.id,
                   body: originalBody + feedbackMessage
                 });
+                console.log(`Successfully appended feedback message to comment ${comment.id}`);
+              } catch (error) {
+                console.error(`Failed to append feedback message: ${error.message}`);
+                throw error;
+              }
+            } else {
+              console.log(`Feedback message already present in comment ${comment.id}`);
             }

This ensures visibility into both successes and failures.

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 71813bf and bc95eb5.

📒 Files selected for processing (1)
  • .github/workflows/coderabbit-feedback.yml (1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (28)
  • GitHub Check: security-it-linux (25)
  • GitHub Check: security-it-linux (21)
  • GitHub Check: build-linux (21, doc)
  • GitHub Check: build-linux (25, integration)
  • GitHub Check: build-linux (21, unit)
  • GitHub Check: build-linux (25, unit)
  • GitHub Check: build-linux (25, doc)
  • GitHub Check: build-linux (21, integration)
  • GitHub Check: bwc-tests-rolling-upgrade (25)
  • GitHub Check: bwc-tests-full-restart (21)
  • GitHub Check: bwc-tests-rolling-upgrade (21)
  • GitHub Check: bwc-tests-full-restart (25)
  • GitHub Check: security-it-windows-macos (windows-latest, 21)
  • GitHub Check: security-it-windows-macos (macos-14, 21)
  • GitHub Check: security-it-windows-macos (windows-latest, 25)
  • GitHub Check: security-it-windows-macos (macos-14, 25)
  • GitHub Check: build-windows-macos (macos-14, 21, integration)
  • GitHub Check: build-windows-macos (windows-latest, 25, -PbuildPlatform=windows, integration)
  • GitHub Check: build-windows-macos (windows-latest, 21, -PbuildPlatform=windows, integration)
  • GitHub Check: build-windows-macos (macos-14, 25, integration)
  • GitHub Check: build-windows-macos (windows-latest, 21, -PbuildPlatform=windows, unit)
  • GitHub Check: build-windows-macos (macos-14, 25, unit)
  • GitHub Check: build-windows-macos (macos-14, 21, unit)
  • GitHub Check: build-windows-macos (macos-14, 21, doc)
  • GitHub Check: build-windows-macos (macos-14, 25, doc)
  • GitHub Check: build-windows-macos (windows-latest, 25, -PbuildPlatform=windows, unit)
  • GitHub Check: Update draft release notes
  • GitHub Check: CodeQL-Scan (java)
🔇 Additional comments (1)
.github/workflows/coderabbit-feedback.yml (1)

1-10: LGTM on the workflow structure.

The trigger configuration, permissions, and job filtering look appropriate for this use case.

Copy link
Copy Markdown
Collaborator

@dai-chen dai-chen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just wonder is this doable in Coderabbit config? Just feel this workflow is very specific for this small task.

@dai-chen
Copy link
Copy Markdown
Collaborator

@ykmr1224 Also wondering is this knowledge base feature available to us? Instead of manual analysis, can we give feedback to Coderabbit directly? https://docs.coderabbit.ai/integrations/knowledge-base

@ykmr1224
Copy link
Copy Markdown
Collaborator Author

Just wonder is this doable in Coderabbit config? Just feel this workflow is very specific for this small task.

Coderabbit don't have configuration for this kind of customization so far.

@ykmr1224 Also wondering is this knowledge base feature available to us? Instead of manual analysis, can we give feedback to Coderabbit directly? https://docs.coderabbit.ai/integrations/knowledge-base

It is enabled and doable, but that relies on each developer giving appropriate feedback.
We should track overall effectiveness and improve that continuously.

@ykmr1224 ykmr1224 merged commit 5d24945 into opensearch-project:main Dec 11, 2025
38 checks passed
@dai-chen
Copy link
Copy Markdown
Collaborator

dai-chen commented Dec 11, 2025

Just wonder is this doable in Coderabbit config? Just feel this workflow is very specific for this small task.

Coderabbit don't have configuration for this kind of customization so far.

@ykmr1224 Also wondering is this knowledge base feature available to us? Instead of manual analysis, can we give feedback to Coderabbit directly? https://docs.coderabbit.ai/integrations/knowledge-base

It is enabled and doable, but that relies on each developer giving appropriate feedback. We should track overall effectiveness and improve that continuously.

I agree. Either the knowledge base can separate team and personal preference, or we give permission to a small group. We can discuss offline. I'm just thinking we can maintain a single source of coding guidance, team knowledge between local Dev agent and this Review agent. Thanks! cc: @penghuo

@ykmr1224 ykmr1224 deleted the coderabbit-feedback branch January 27, 2026 20:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

infrastructure Changes to infrastructure, testing, CI/CD, pipelines, etc.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants