Skip to content

[docs] Strengthen our quality standards and connect AI contribution policy to it #154441

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rnk wants to merge 16 commits into
base: main
Choose a base branch
from
Open

[docs] Strengthen our quality standards and connect AI contribution policy to it #154441

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
b9aaea9
[docs] Strengthen our quality standards and connect AI contribution p…
rnk Jul 18, 2025
8f861a0
Updating text to center it around extractive/non-extractive contribut…
rnk Aug 20, 2025
c52546a
Address feedback on nightly testing and add copy-pasted message
rnk Aug 25, 2025
dd6e729
Update wording: We comment on first-time PRs, we do not label them (yet)
rnk Aug 25, 2025
4590f61
Merge branch 'main' into tool-policy
rnk Sep 2, 2025
89a6d2f
Move all the text into a new policy doc
rnk Sep 2, 2025
fed0096
Restore original Developer Policy text with minimal edits
rnk Sep 2, 2025
9fc163d
Merge branch 'main' into tool-policy
rnk Sep 15, 2025
3683932
Merge branch 'main' into tool-policy
rnk Oct 1, 2025
77071ed
Incorporate the Fedora policy, indicate it carries a CCA license, ret…
rnk Oct 1, 2025
a8ed845
More minor edits
rnk Oct 1, 2025
ed524ff
Add objective guideline of 150 additional lines of code
rnk Oct 1, 2025
5394a3c
Merge branch 'main' into tool-policy
rnk Nov 19, 2025
1a942f0
Try again, focusing on the idea of having a human in the loop.
rnk Nov 19, 2025
2203c63
Merge branch 'main' into tool-policy
rnk Dec 16, 2025
97bd981
December editting pass, make another attmept
rnk Dec 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
162 changes: 162 additions & 0 deletions llvm/docs/AIToolPolicy.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
# LLVM AI Tool Use Policy

## Policy

LLVM's policy is that contributors can use whatever tools they would like to
craft their contributions, but there must be a **human in the loop**.
**Contributors must read and review all LLM-generated code or text before they
ask other project members to review it.** The contributor is always the author
and is fully accountable for their contributions. Contributors should be
sufficiently confident that the contribution is high enough quality that asking
for a review is a good use of scarce maintainer time, and they should be **able
to answer quesions about their work** during review.
Copy link
Collaborator

@hubert-reinterpretcast hubert-reinterpretcast Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo:

Suggested change
to answer quesions about their work** during review.
to answer questions about their work** during review.


We expect that new contributors will be less confident in their contributions,
and our guidance to them is to **start with small contributions** that they can
fully understand to build confidence. We aspire to be a welcoming community
that helps new contributors grow their expertise, but learning involves taking
small steps, getting feedback, and iterating. Passing maintainer feedback to an
LLM doesn't help anyone grow, and does not sustain our community.

The most important question in shared, foundational open source infrastructure
is often not whether something can be done (many things are possible), but
whether a change advances the many competing interests of the project
contributors.

An important implication of this policy is that it bans agents that take action
in our digital spaces without human approval, such as the [GitHub @claude
agent](https://github.com/claude/). Similarly, automated review tools that
publish comments without human review are not allowed. However, an opt-in
review tool that **keeps a human in the loop** is acceptable under this policy.
As another example, using an LLM to generate documentation, which a contributor
manually reviews for correctness, edits, and then posts as a PR, is an approved
use of tools under this policy.

This policy extends beyond code contributions and includes, but is not limited
to, the following kinds of contributions:
Comment on lines +35 to +36
Copy link
Collaborator

@hubert-reinterpretcast hubert-reinterpretcast Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove duplicate mention of code contributions.

Suggested change
This policy extends beyond code contributions and includes, but is not limited
to, the following kinds of contributions:
This policy includes, but is not limited to, the following kinds of
contributions:


- Code, usually in the form of a pull request
- RFCs or design proposals
- Issues or security vulnerabilities
- Comments and feedback on pull requests

**Be transparent about the tools used to produce a change.** When a
contribution has been substantially generated by an AI tool, contributors
should note this in your pull request description, commit message, or wherever
authorship is normally indicated for the work. For instance, use a commit
message trailer like Assisted-by: <name of code assistant>. This transparency
helps the community develop best practices and understand the role of these new
tools.

## Extractive Changes

The reason for our "human-in-the-loop" contribution policy is that sending
patches, PRs, RFCs, and comments to LLVM, is not free -- it takes a lot of
Comment on lines +53 to +54
Copy link
Collaborator

@hubert-reinterpretcast hubert-reinterpretcast Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The reason for our "human-in-the-loop" contribution policy is that sending
patches, PRs, RFCs, and comments to LLVM, is not free -- it takes a lot of
The reason for our "human-in-the-loop" contribution policy is that processing
patches, PRs, RFCs, and comments to LLVM is not free -- it takes a lot of

maintainer time and energy to review those contributions! Sending the
unreviewed output of an LLM to open source project maintainers *extracts* work
from them in the form of code and design review, so we call this kind of
contribution an *"extractive change"*.

Our **golden rule** is that a contribution should be worth more to the project
than the time it takes to review it. These ideas are captured by this quote
from the book [Working in Public][public] by Nadia Eghbal:

[public]: https://press.stripe.com/working-in-public

> \"When attention is being appropriated, producers need to weigh the costs and
> benefits of the transaction. To assess whether the appropriation of attention
> is net-positive, it's useful to distinguish between *extractive* and
> *non-extractive* contributions. Extractive contributions are those where the
> marginal cost of reviewing and merging that contribution is greater than the
> marginal benefit to the project's producers. In the case of a code
> contribution, it might be a pull request that's too complex or unwieldy to
> review, given the potential upside.\" \-- Nadia Eghbal
Prior to the advent of LLMs, open source project maintainers would often review
any and all changes send to the project simply because posting a change for
Copy link
Collaborator

@hubert-reinterpretcast hubert-reinterpretcast Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
any and all changes send to the project simply because posting a change for
any and all changes sent to the project simply because posting a change for

review was a sign of interest from a potential long-term contributor. Reviewing
changes from new contributors is part of growing the next generation of
contributors and sustaining the project. While new tools enable more
development, it shifts effort from the implementor to the reviewer, and our
policy exists to ensure that we value and do not squander maintainer time.

We encourage contributions that help sustain the project. We want the LLVM
project to be welcoming and open to aspiring compiler engineers who are willing
to invest time and effort to learn and grow, because growing our contributor
base and recruiting new maintainers helps sustain the project over the long
term. We therefore automatically post a greeting comment to pull requests from
new contributors and encourage maintainers to spend their time to help new
contributors learn.

## Handling Violations

If a maintainer judges that a contribution is *extractive* (i.e. it is
generated with tool-assistance or simply requires significant revision), they
Copy link
Contributor

@asb asb Oct 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe these lines don't capture the intent of the policy. It suggests any contribution generated with tool assistance is extractive. Perhaps "it is generated with tool-assistance in a way inconsistent with our AI tool usage policy"?

should copy-paste the following response, add the `extractive` label if
Copy link
Member

@JDevlieghere JDevlieghere Oct 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I realize this isn't meant to be exhaustive, but should we also encourage to "request changes". The motivation being that it (1) clears it from my review queue and (2) sends a clear message to other reviewers not to bother with the patch. I can filter based on the extractive label but I think the "changes requested" is more obvious.

applicable, and refrain from further engagement:

This PR appears to be extractive, and requires additional justification for
why it is valuable enough to the project for us to review it. Please see
our developer policy on AI-generated contributions:
http://llvm.org/docs/AIToolPolicy.html

Other reviewers should use the label prioritize their review time.
Copy link
Contributor

@maflcko maflcko Oct 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Other reviewers should use the label prioritize their review time.
Other reviewers should use the label to prioritize their review time.


The best ways to make a change less extractive and more valuable are to reduce
its size or complexity or to increase its usefulness to the community. These
factors are impossible to weigh objectively, and our project policy leaves this
determination up to the maintainers of the project, i.e. those who are doing
the work of sustaining the project.

If a contributor responds but doesn't make their change meaningfully less
extractive, maintainers should escalate to the relevant moderation or admin
team for the space (GitHub, Discourse, Discord, etc) to lock the conversation.

## Copyright

Artificial intelligence systems raise many questions around copyright that have
yet to be answered. Our policy on AI tools is similar to our copyright policy:
Contributors are responsible for ensuring that they have the right to
contribute code under the terms of our license, typically meaning that either
they, their employer, or their collaborators hold the copyright. Using AI tools
to regenerate copyrighted material does not remove the copyright, and
contributors are responsible for ensuring that such material does not appear in
their contributions. Contributions found to violate this policy will be removed
just like any other offending contribution.

## Examples

Here are some examples of contributions that demonstrate how to apply
the principles of this policy:

- [This PR][alive-pr] contains a proof from Alive2, which is a strong signal of
value and correctness.
- This [generated documentation][gsym-docs] was reviewed for correctness by a
human before being posted.

[alive-pr]: https://github.com/llvm/llvm-project/pull/142869
[gsym-docs]: https://discourse.llvm.org/t/searching-for-gsym-documentation/85185/2

## References

Our policy was informed by experiences in other communities:

- [Fedora Council Policy Proposal: Policy on AI-Assisted Contributions (fetched
2025-10-01)][fedora]: Some of the text above was copied from the Fedora
project policy proposal, which is licensed under the [Creative Commons
Attribution 4.0 International License][cca]. This link serves as attribution.
- [Rust draft policy on burdensome PRs][rust-burdensome]
- [Seth Larson's post][security-slop]
on slop security reports in the Python ecosystem
- The METR paper [Measuring the Impact of Early-2025 AI on Experienced
Open-Source Developer Productivity][metr-paper].
- [QEMU bans use of AI content generators][qemu-ban]
- [Slop is the new name for unwanted AI-generated content][ai-slop]

[fedora]: https://communityblog.fedoraproject.org/council-policy-proposal-policy-on-ai-assisted-contributions/
[cca]: https://creativecommons.org/licenses/by/4.0/
[rust-burdensome]: https://github.com/rust-lang/compiler-team/issues/893
[security-slop]: https://sethmlarson.dev/slop-security-reports
[metr-paper]: https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
[qemu-ban]: https://www.qemu.org/docs/master/devel/code-provenance.html#use-of-ai-content-generators
[ai-slop]: https://simonwillison.net/2024/May/8/slop/
21 changes: 1 addition & 20 deletions llvm/docs/DeveloperPolicy.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1480,23 +1480,4 @@ permission.
AI generated contributions
--------------------------

Artificial intelligence systems raise many questions around copyright that have
yet to be answered. Our policy on AI tools is guided by our copyright policy:
Contributors are responsible for ensuring that they have the right to contribute
code under the terms of our license, typically meaning that either they, their
employer, or their collaborators hold the copyright. Using AI tools to
regenerate copyrighted material does not remove the copyright, and contributors
are responsible for ensuring that such material does not appear in their
contributions.

As such, the LLVM policy is that contributors are permitted to use artificial
intelligence tools to produce contributions, provided that they have the right
to license that code under the project license. Contributions found to violate
this policy will be removed just like any other offending contribution.

While the LLVM project has a liberal policy on AI tool use, contributors are
considered responsible for their contributions. We encourage contributors to
review all generated code before sending it for review to verify its
correctness and to understand it so that they can answer questions during code
review. Reviewing and maintaining generated code that the original contributor
does not understand is not a good use of limited project resources.
This section has moved into a :doc:`separate policy document <AIToolPolicy>`.
1 change: 1 addition & 0 deletions llvm/docs/Reference.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ LLVM and API reference documentation.
.. toctree::
:hidden:

AIToolPolicy
Atomics
BitCodeFormat
BlockFrequencyTerminology
Expand Down
Loading