Discuss using AI to help in the review process of issues and PRs

[ocelotl]

Is your feature request related to a problem?

This repo has been receiving more issues and pull requests than maintainers can handle. As the project grows, keeping up with the review queue is increasingly difficult with the current maintainer bandwidth.

Describe the solution you'd like

I would like to explore ways to use AI responsibly to help triage and review incoming issues and PRs, so maintainers can focus their attention where it matters most.

Describe alternatives you've considered

Automated triage and labeling

A GitHub Action triggered on new issues and PRs could use a model to read the submission and suggest labels, assign a category (bug report, feature request, question, documentation gap), and route it to the right area of the codebase. This would reduce the time maintainers spend on the mechanical parts of triage and help new submissions surface in the right queue faster.

AI-assisted first review

Rather than waiting for a maintainer to open a PR, a bot could post an initial structured review: does the description explain the motivation? Is there a linked issue? Does the diff match the stated intent? Are there obvious gaps like missing tests or changelog entries? This gives the contributor immediate, actionable feedback and means that by the time a maintainer looks at it, the submission is already in better shape.

Clarifying questions on incomplete submissions

Many issues stall because they are missing reproduction steps, environment details, or version information. A model could detect when a submission is likely incomplete and post a friendly comment asking for the missing context before a maintainer ever sees it. This keeps the queue from filling up with issues that cannot be acted on yet.

Duplicate and related issue detection

When a new issue is opened, a model could search the existing issue tracker for semantically similar open or closed issues and surface them in a comment. This helps contributors find prior discussion before a maintainer has to do it manually, and reduces duplicate work across the board.

Contribution policy

Beyond tooling, we could adopt lightweight policies that raise the quality floor before automation or a maintainer ever gets involved:

• Require a linked issue for every PR. A PR without a corresponding issue cannot be reviewed because there is no agreed-upon problem statement to evaluate the change against. Requiring this upfront ensures that the motivation for a change has been discussed and accepted before implementation work begins.
• Require the PR author to be assigned to the linked issue. This establishes that the contributor has signaled intent and received at least implicit acknowledgment from the project before opening a PR. It prevents duplicate work and gives maintainers a natural checkpoint to redirect effort before code is written.
• Automatically close submissions that do not meet requirements. PRs or issues that are missing required information would be closed automatically after a grace period, with an AI-generated comment explaining exactly what is missing and what the author needs to do to have the submission reconsidered. The close is not a rejection — it is an invitation to return once the requirements are met. This keeps the queue actionable without requiring maintainer time on submissions that are not yet ready for review.

Additional Context

Whatever approach we adopt should be transparent — if automation acts on a submission, it should say so clearly — and should keep a human in the loop for any consequential decisions. I am interested in hearing from the community: if you have seen this handled well in other projects, or have ideas around tooling or process, please share them here.

Would you like to implement a fix?

Yes

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[xrmx]

Some notes and some behaviors I'm seeing around here:

• the fact that there is an issue does not mean there's an agreement :)
• people throwing agents at already filed issues, agents don't check if there's a PR already open
• people throwing agents at years old issues, issues are legit
• We should make our AGENTS.md be more explicit on making agents report themselves explicitly, ghostty AGENTS.md seems effective
• https://github.blog/changelog/2026-06-17-limit-open-pull-requests-for-users-without-write-access/ looks like a good start (need to add approvers to bypass list I guess)
• we shouldn't forget to look at this problem from another angle and continue asking: what we can do to have people around willing to review and generally taking care of this community?

[emdneto]

On top of what @xrmx said:

• For Duplicate and related issue detection: I just noticed GitHub launched this https://github.blog/changelog/2026-06-18-duplicate-detection-and-issue-fields-mcp-support-for-github-issues/
• +1 for the contribution policy. Besides what you included in the policy, I think we can improve our Guidelines for using AI tools based on https://devguide.python.org/getting-started/ai-tools/.
• +1 For the automated triage. I found this: https://github.com/marketplace/actions/ai-assessment-comment-labeler, which can be a good starting point (i didn't try it before) - Clarifying questions on incomplete submissions can be also part of this workflow.

Regarding the AI-assisted first review, I think it can work fine after the contributor does the first contribution, so workflows can run automatically.

[Kludex]

I'm currently using https://www.cubic.dev/, https://chatgpt.com/codex/ and https://www.coderabbit.ai/, depending on the project for reviews.

I've also used https://devin.ai/, but they started charging a lot on the close source side, so I stopped using on the open source side.

All of them feel very similar to be honest.

[herin049]

We should make our AGENTS.md be more explicit on making agents report themselves explicitly, ghostty AGENTS.md seems effective

+1 to all of @xrmx's points. I think we should try to add a small section to the AGENT.md/CLAUDE.md which directs agents to leave a small fingerprint on PRs so we can filter out/ban accounts that are blindly submitting PRs with little/no thought/effort.

Personally, I've been seeing a lot of people who appear to be engaging in "Resume farming" where they use their Claude Max subscription to generate a large number of PRs for various open source projects in the hopes that contributors will take the time to review their code so they can add "Open Source Contributor to XYZ" on their Resume. We should start by limiting the number of open PRs to non-members/first-time contributors as @xrmx suggested.