Add *no low-effort contributions* policy

[traviscross]

View all comments

Previously I had proposed a no low-effort contributions policy in rust-lang/leadership-council#273 (comment), where it had gone into FCP. In this commit, I add the text of that proposed policy in RFC format.

Later, I may add other sections -- e.g., motivation, rationale, unresolved questions, etc. In the interest of expediency, I'm starting with just the policy itself and the associated examples.

cc @rust-lang/leadership-council

Important

This is a difficult issue. Remember that we're all colleagues trying to do the best for Rust and that we all need to continue to work together. Please keep in mind that people have many different experiences and earnestly hold many different views. Please focus on assuming good intent, being curious, and acting compassionately.

Note

Keep in mind, too, that on this one, we mostly are interested in hearing from members of the Project. If you're not a member of the Project and have valuable information or a unique and relevant experience to share, e.g., about your experience contributing, then we're probably interested in hearing that. Otherwise, you may want to sit back a bit on this one. See #3936 (comment).

Important

When responding to RFCs, try to use inline review comments (it is possible to leave an inline review comment for the entire file at the top) instead of direct comments for normal comments and keep normal comments for procedural matters like starting FCPs.

This keeps the discussion more organized.

Rendered

[traviscross]

Given that this policy had already gone into FCP over in rust-lang/leadership-council#273 (comment), and given that there is some sense of time sensitivity on this, I'm going to go ahead and repropose FCP merge so that people on the council who wish to do so can proxy their checkboxes. But please don't mistake that for foreclosing discussion. This is being brought here as a request for comment so as to ensure wide visibility and participation, particularly among members of the Project.

@rfcbot fcp merge leadership-council

[rust-rfcbot]

Team member @traviscross has proposed to merge this. The next step is review by the rest of the tagged team members:

• @Kobzol
• @ehuss
• @lqd
• @m-ou-se
• @oli-obk
• @traviscross

Concerns:

• needs-discussion (Add *no low-effort contributions* policy #3936 (comment))

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[traviscross]

We had talked elsewhere about whether we should lock this conversation such that only members of the Project could participate. With my finger now hovering over the lock button and having to select a reason (e.g., "too heated"), I find myself instead wanting to have faith in the ability of our community to have a thoughtful, curious, and compassionate discussion. So let's maybe try that first. Please don't let me down and prove me wrong.

As I put at the top:

Important

This is a difficult issue. Remember that we're all colleagues trying to do the best for Rust and that we all need to continue to work together. Please keep in mind that people have many different experiences and earnestly hold many different views. Please focus on assuming good intent, being curious, and acting compassionately.

[ChayimFriedman2]

Does this policy obligate every subproject in the Project or does it apply only to rust-lang/rust? I don't think any subproject wants slop contributions, but this may be meaningful in other related decisions of the council.

[traviscross]

Does this policy obligate every subproject in the Project or does it apply only to rust-lang/rust? I don't think any subproject wants slop contributions, but this may be meaningful in other related decisions of the council.

As proposed, it's for the Project as a whole.

[steffahn]

I’m not a huge fan of the wording here. It’s talking about compassion, but

Listen carefully. Reflect on it. Reply to us compassionately.

is IMO not the most compassionate way of addressing someone in and by itself.

Perhaps leaning too much into the story of relating everything to AI, but if I was setting up an LLM with a personalized prompt, something like

I want to make you better able to help me. Keep that in mind when we're offering feedback. Listen carefully. Reflect on it. Reply to me compassionately.

can almost be more fitting there. It reads like a list of stern commands towards a subordinate; not exactly the tone for a non-hierarchical, collaborative, largely volunteer-based open-source project like ours.

---

I know I’m a bit nitpicky about wording, but it’s very important in my opinion, since moderators would more easily/happily link to an impeccable document without any second-guessing or need to provide additional context or clarification, and every time we can link to something that already exists for explanation, something that feels suitable and appropriate to many situation well that’s when we are really saving time and effort.

[traviscross]

Interesting. It doesn't strike me negatively in that way, even now as I sit here trying to force it to do so. It reads to me more in the way a meditation might -- a set of maxims, aphorisms, tenets, reflections, affirmations, proverbs, etc. E.g., "treat others as you would want to be treated." That sort of thing. It's set in imperative mood, just as those are.

I wrote it this way to be concise -- that's something people had mentioned as desirable in discussions. That's probably why other maxims tend to be written this way too. I could take it out of imperative mood, but it would make it longer and less crisp.

(And also, this is a policy, so we are actually telling people how we expect them to act. It is, in fact, a command.)

[steffahn]

It doesn’t strike me negatively - personally - either, but I’ve witnessed so many cases of bad reactions to unfortunately worded and/or misunderstood statements, or demands, in online interactions that I’m cautious about such things. (Ultimately the proof would only be in whether or not other people, who weren’t the author, might come in and report on any such negative perceptions.)

---

(And also, this is a policy, so we are actually telling people how we expect them to act. It is, in fact, a command.)

That’s good that you’re bringing this point up, because I hadn’t considered it yet: as far as I understand, “policies” would often rather describe (and hence prescribe for those that want to join and follow along) mainly the behavior of the project or teams on certain matters. A moderation policy for example, would describe how we - the Rust project (specifically moderation team) - is moderating things. A privacy policy would explain to a customer the information on the rules and procedures and such of how the company handles your data. The original policy proposal hence also touched a lot more on the specific behavior of compiler team, moderators, etc. The document now still does it a little bit, but only really in one single line starting Maintainers and moderators are **not** required […].

I’m not sure though whether this detail is relevant, or just my personal limited exposure to things called “policies” but actually being documents of rules for the user.

[traviscross]

The comparable to the ones you mention would be an Acceptable Use Policy.

[steffahn]

I’m comparing this phrasing to the things that already exist in the etiquette section on the “How to start contributing” page of forge.rust-lang.org and I’m seeing similarities. For reference, that section states:

We know that starting contributing in a FOSS¹ project could be confusing at times and we want both contributors and reviewers have the best possible experience when collaborating in our project.

To achieve this goal, we want to build trust and respect of each other’s time and efforts. Our recommendation is to follow these simple guidelines:

• Start small. A big ball of code as first contribution does not help to build trust
• The work you submit is your own, meaning that you fully understand every part of it
• You take care of checking in detail your work before submitting it - ask questions or signal us (with inline comments or todo!()) the parts you’re unsure about
• If you want to fix an issue but have doubts about the design, you’re welcome to join our Zulip server and ask for tips
• Please respect the reviewers’ time: allow some days between reviews, only ask for reviews when your code compiles and tests pass, or give an explanation for why you are asking for a review at that stage (you can keep them in draft state until they’re ready for review)
• Try to keep comments concise, don’t worry about a perfect written communication. Strive for clarity and being to the point

---

I’m not calling this out as thus being redundant or anything, or asking for it to be the same extended length as the linked (and quoted) etiquette is, but only because I’m noticing the other document made things much less ambiguous with the demand to “check your work”.
The demand there read more clearly: the “checking” is a step on top of making sure that it’s your work. You’re supposed to create some work, and then also check it.

---

The current phrasing here instead accountability requirements for “everything you send to us” and says we “expect you to understand it” and “expect you to check it carefully”.

It can be easily mis-read to also imply “feel free to send us something AI generated” as long as you “understand it” and “check it”. IMHO in this reading, one would easily be missing the crucial step of making someone that’s truly yours in the first place.

Footnotes

1. Free-Open Source Project, see: https://en.wikipedia.org/wiki/Free_and_open-source_software ↩

[traviscross]

It is interesting the similarities. I had not consulted the other document. But it doesn't surprise me -- I had intended to put forward a set of principles that I believed to already be widely accepted within the Project.

The document on the Forge reads to me as a set of guidelines -- a good set. What I think we want here are a set of policies. That's a bit different of a thing, and we do have to state policies more narrowly and crisply than we would state guidelines. Not following all of our guidelines might be OK -- they're to be read as suggestions, even if strong ones. Not following even one of our policies, though, is a violation.

IMHO in this reading, one would easily be missing the crucial step of making someone that’s truly yours in the first place.

Yes, I agree that's a risk. And I agree that we want people to make the contributions their own regardless of the tools used. But as a policy matter, I feel that we need to break this into parts. The parts that occur to me are:

• Put real effort into it.
• Understand it and be able to explain it.
• Check it carefully.
• Take responsibility for it.
• Respect our time.
• Treat us well.

That's what I tried to encode.

[steffahn]

For context: When I had first come to reading this text in the context of the original LC-repo PR, where we were explicitly talking about AI policy, I first read this sentence actually as being predominantly about “carefully check the things you submit” as an extra step for AI-powered contributions specifically and my initial reaction was kind-of along the lines of “Why is this in the generally applicable section if it’s assuming use of AI, from the ‘check it carefully’ wording?” though now I see that this isn’t necessarily actually what’s meant here.

(I’m saying “this isn’t actually what’s meant here” here, or that it would be a “mis-reading”, since I’m convinced - and I have previously argued in Zulip discussions - that ‘check it carefully’ cannot possibly be sufficient for a requirement to make AI-generated things truly your own.¹ Whereas checking things that are already your own work should be useful, since if you made it you ought to be able to actually be able to effectively check it.)

Footnotes

1. There certainly are a lot of people where these requirements can be sufficient, but there’s also a lot of people who would be honestly convinced to follow the requirement of “understanding” and “checking”, but would still produce contributions not worth our time. And due to the latter, it’s not sufficient as a requirement. ↩

[steffahn]

Maybe the “own work” could just be added 🤔

Suggested change

	- **Accountability**: We hold you responsible for everything you send to us. We expect you to understand it and be able to explain it. We expect you to check it carefully. Respect our time.
	- **Accountability**: We expect that everything you send to us is your own work, and we hold you responsible to it. We expect you to understand it and be able to explain it. We expect you to check your work carefully. Please respect our time.

[steffahn]

I just remembered, I’m also not happy with the phrasing of

Fully AI-generated and not carefully self-reviewed contributions are low-effort

or

Code, PR descriptions, bug reports, proposals, and comments are all low-effort if fully-generated and not carefully self-reviewed

further down in the document then. I overlooked these earlier, I dislike how does clearly pick up the previous “check it carefully” wording, even though that wording came up in a section of requirements for all contributions.

---

It’s probably pretty hard to pull off, but I think it’s very important that the requirement to check your work carefully should IMHO not be made (or too strongly implied to be), in any way a defining feature or sufficient condition for “acceptable AI use”.

I know the document “This list is nonexhaustive.” does aim to make it technically so that this isn’t the case; but I believe we could manage to represent the degree of current consensus much more clearly than this current text. (I have no concrete ideas for (ideally minimal) changes that could improve this yet, but I’ll try to think about this.)

[nikomatsakis]

So...I don't quite agree @traviscross with the way you characterized the stats I gathered. For reference, I found that moderation action was taking place on about 2-4 PRs per week. I agree that's not like an avalanche but it's also a consistent stream and I can imagine that the emotional impact from that is higher than the raw number makes it sound. But also, I think that I'm only counting a slice of the problem, for example, it doesn't include issue reports like @lqd just mentioned.

My honest opinion here? I think that the problem is real. I think that as a total percentage of activity it's probably lower than it feels like it is -- but in part that's because the frustration of dealing with some of these issues is so disproportionately high, particularly given the ethical, societal, and climate concerns that are on peoples' minds.

I should say that my goal in gathering stats wasn't to try and assess whether the problem was real or not, I think it's enough to say that it's a source of burnout, however large it is-- but rather to try and figure out whether we could take some automated steps to reduce the amount of manual work required for mods and people triaging issues.

[nikomatsakis]

I do want to say something-- hmm-- I'm not sure how to phrase this. I hope it's not offending people, I'm just trying to be real. I think that part of the work people put into "sniffing out" AI is unnecessary and makes the problem feel worse than it has to.

I see in @apiraino's comment for example, "Don't include the time project members take to report something that smells fishy", I feel like... there's something here that needs further digging. Like, maybe disclosure would help, so there's no guesswork required. Maybe just relaxing a bit about what tools someone is using would help. I think if we worry about people hiding from us what tools they use that's probably on us to make it clear what our rules are and what acceptable use looks like.

In saying this I'm aware I'm not on the front line of doing a lot of reviewing. I'm trying to put my money where my mouth is for a-mir-formality, maybe I ought to try and pick up more of the compiler review queue, for that matter.

[oli-obk]

I like that, not having to analyze deeply, but instead figuring out how we can avoid that effort entirely.

I think we can do two things maybe:

• Disclosure is needed, both for not being sidelined later, and for "mode-switching" when analyzing the contrib

• requiring (where there is no policy for an area like the reference and formality have or are getting) that you "ask for permission" first. Basically "I wanna do an AI assisted contrib in foo, who can be the reviewer".

This doesn't mean we allow AI contribs or forbid them. This is not a path to getting your AI contrib merged. This is a way to not get ppl annoyed at you for various reasons ppl would get annoyed.

[nikomatsakis]

I like those ideas as well, @oli-obk. I've been thinking about how to deal with the idea of "not everybody wants to review AI-generated output". It seems clear from the rust project perspectives that there is a segment that would feel morally compromised by doing so and I want to respect that without forcing everyone to agree with it.

[traviscross]

A charitable interpretation makes me think that you're just not aware of how much work we collectively put into this stuff, so please allow me to expand a little bit here.

Thanks for taking a charitable interpretation, @apiraino, and for expanding. Filling in more details, as you did, is exactly the kind of engagement I was hoping to see here. The fuller picture you paint — the time spent separating wheat from chaff, the internal discussions, the appeal emails, the back and forth with GitHub — helps build the kind of shared understanding I think we need.

So...I don't quite agree @traviscross with the way you characterized the stats I gathered. For reference, I found that moderation action was taking place on about 2-4 PRs per week. I agree that's not like an avalanche but it's also a consistent stream and I can imagine that the emotional impact from that is higher than the raw number makes it sound.

For what it's worth, @nikomatsakis, I think we're in agreement. You said it wasn't an avalanche — I said it wasn't an avalanche (though, to all, please read on — in saying that, I don't mean to diminish it one bit). You called it a stream — I described it as an unrelenting drip (by which, in context, I mean the kind that destroys property, or water torture). You said it may have emotional impact higher than the raw numbers — I agree — I had said:

Now — I understand — even small numbers per week can wear on people since it's a drip and goes on week after week. I get that.

In that, I'm suggesting that regardless of what the numbers are, the experience is wearing on people. Clearly we must take that seriously.

I understand that people are angry. I understand that many find dealing with the garbage that's been coming in triggering. I understand why, in this context, questioning a characterization on a quantitative basis might seem beside the point. With hindsight, I can see how asking questions directly might have been more effective. That is the benefit of hindsight.

But whether or not I raised it as well as I might have, I did and do feel this is an important point to understand. People have been characterizing it similarly in other places. The numbers that Niko found — 2-4/week — were therefore smaller than what I was expecting.

I don't think the exact words matter, but I think our shared understanding does. Seeing the characterization, and then seeing the numbers, I get curious. Maybe the numbers are wrong. Maybe, as @apiraino suggested, they don't tell the whole story. Maybe the numbers don't matter. In any case, I believe we do well by digging into it and having that discussion to build the shared understanding we need.

---

Many of us had the opportunity to talk in a more relaxed setting today, and that proved healthy. I'm grateful we were able to do that, and I'm grateful to those who were able to join us. This situation is creating a lot of understandable raw anger, frustration, exhaustion, and more, and it seems clear that's leaking into these threads. The more we can see each other as people who are all trying to do our best for the Rust Project, the better that will be for all of us and for the Project. That's the only way, I think, that we'll have some hope of coming out of this crisis stronger.

[clarfonthey]

Overall, this policy disappoints me. Not because the general idea isn't good, but because it feels… spineless. It's trying to generalise something that can't easily be generalised.

Compare to an existing moderation team policy which has also been accepted: https://github.com/rust-lang/moderation-team/blob/main/policies/spam.md/#spam-policy

The reason why I label this as spineless is because it seems desperate to try and classify "slop" contributions as something that isn't descriptive enough: "low effort."

Here's the reality: LLM contributions are uniquely positioned as something that is resource-intensive, requires minimal input from a contributor, and creates maximum work for maintainers. They cannot be generalised without losing specificity and should not be.

Overall, the team has had an overwhelmingly negative response toward LLMs and despite this, a few extremely LLM-positive folks have been resisting the urge to create a proper LLM policy, with the fear that we might reach too far. I think that choosing to just label this as "anti-low-effort" is doing a massive disservice to those of us who have already spoken in favour of a stricter LLM policy.

I'd been holding off on working more into this but the presence of this PR seems to be forcing the issue a little bit. We already have moderation policies to stem the flow of spam, and so, it seems more apt to hold off in favour of a stricter, more robust policy.

Right now, ironically, the "no low-effort contributions" policy seems to be, itself, a low-effort contribution. It doesn't follow the RFC template at all and has a number of paragraphs that can almost be counted on a singular hand. Compared to all the stuff folks have written on existing "no slop" contribution policies, it seems to completely ignore those in favour of a spineless policy.

Additionally, the desire to close the discussion as "too heated" even before a discussion has began, while also pushing to vote on it immediately, is incredibly telling. It gives the opinion that you don't trust the community at all and are expecting backlash… from whom? I am glad you chose to not close this, but am very disappointed.

I have more specific comments I'll add on specific points.

[clarfonthey]

Also putting this here for lack of a better place: I've opened a thread with some polls on Zulip to get a basic "vibe check" for how people feel about various AI policy implementations, and whether they feel they should apply. This is 100% just to get a feel for what kinds of policies people would be willing to accept, and is not meant to be a place to suggest what should be done, just to get a feel for what the shape of an ultimate policy might be. Although Zulip doesn't let you make votes private, I figure I should explicitly add that this thread is not a place to discuss anyone's views on the matter, and is purely for getting an aggregate view of the community's opinion specifically when it comes to policy.

#general > Vibecoding vibecheck

There is also the already-existing thread on project perspectives on AI, which has been orthogonal to any discussion of policy: #council > Project perspectives on AI

Also worth adding that, although the poll explicitly includes a policy on slop for completeness, it's unlikely that the final policy will allow "slop contributions" in any form, considering how much work so far has been done to quell spam against the rust-lang repository and how the community consensus, even among the most diehard LLM fans, has been that you shouldn't just spam the repo with changes that you haven't reviewed.

[traviscross]

I'd been holding off on working more into this but the presence of this PR seems to be forcing the issue a little bit. We already have moderation policies to stem the flow of spam, and so, it seems more apt to hold off in favour of a stricter, more robust policy.

It's a reasonable point, I think, that maybe we should take it more slowly. Maybe it's OK to leave things, for now, to the moderators who "know it when they see it". Maybe we don't need the council to adopt a policy immediately. In other places I've suggested similar things may be true.

On the other hand, it's also possible that adopting some policy may help us in avoiding or moving on from the kind of upheavals that happened over the weekend. Reasonable people have made that point too.

This proposal is acting on that second theory: that adopting some policy -- that encodes the intersection of the principles we share -- might help us to move forward.

(As also mentioned in #3936 (comment).)

The reason why I label this as spineless is because it seems desperate to try and classify "slop" contributions as something that isn't descriptive enough: "low effort."

While I think your characterization to have unnecessary bite, it's true that my goal in drafting this policy was to write down the principles on which, I believe, we already agree. The idea is, by starting with that intersection and writing down in plain language what we want and don't want -- things we may have taken for granted before -- that maybe that helps us to move forward.

You've made a number of harsh criticisms -- in this subthread and others -- but I'm curious. Given that there are a wide spectrum of views on this, and given that there are many practical considerations (see, e.g., the various Zulip threads where these have been enumerated), how do you see this ending? You would propose to start, I imagine, with a less "spineless" policy proposal, but many of the options for that would struggle to gain consensus. So what do we do then? Do you think there's not some risk in not adopting some policy now, given what occurred over the weekend?

On this:

Right now, ironically, the "no low-effort contributions" policy seems to be, itself, a low-effort contribution. It doesn't follow the RFC template at all and has a number of paragraphs that can almost be counted on a singular hand. Compared to all the stuff folks have written on existing "no slop" contribution policies, it seems to completely ignore those in favour of a spineless policy.

Additionally, the desire to close the discussion as "too heated" even before a discussion has began, while also pushing to vote on it immediately, is incredibly telling. It gives the opinion that you don't trust the community at all and are expecting backlash… from whom? I am glad you chose to not close this, but am very disappointed.

(See also #3936 (comment).)

I'd encourage you again to reflect on the request at the top of this thread:

Important

This is a difficult issue. Remember that we're all colleagues trying to do the best for Rust and that we all need to continue to work together. Please keep in mind that people have many different experiences and earnestly hold many different views. Please focus on assuming good intent, being curious, and acting compassionately.

I didn't lock the PR (see #3936 (comment)), as had been proposed (and people had proposed even more drastic measures), because I do want to have faith in the ability of our community to have a thoughtful, curious, and compassionate discussion. But we each must be careful -- we must demonstrate this ourselves -- or there is real risk that this faith will have proved misplaced.

[clarfonthey]

Travis, I'm going to be honest, you repeatedly telling me to reread the request at the top of the thread is feeling less polite and more… aggressive, and I'd like you to consider your actions as well. Telling someone to watch what they say is not a neutral act, and in your case, it feels almost like a threat. I know very well what the rules are, and have even been repeatedly reminded because I have overstepped in cases, and should not here. I'm trying to be as careful as possible in my words here without compromising on what I'd like to say.

As far as endgame goes, genuinely, I would rather have a discussion first. Like, if nothing else, that's the point I want to emphasise first.

So far, a lot of discussion has gone into how people feel about AI, and summarising what people think about AI, but absolutely none of it has gone into policy discussion, effectively, minus the stopgap spam policies. (Which, for the record, have been widely received with minimal disagreements. Again, I think that stopgap policies can exist without the RFC process.)

Right now, this feels like an attempt to end discussion, even though I know that's not the intent. Having discussions about policy is inherently exhausting and I sort of treat the RFC process as a sort of endgame status: this is the last time where we have a community discussion on the matter unless there are major unaddressed points. To me, this RFC says: there is no LLM policy, only a "low-effort contribution" policy, and that's final, regardless of whether that's actually what you're trying to say.

Like, I hope that we can all agree that discussions about policy, and politics, (fun fact, in French, these are the same word), are exhausting. My goal would be to have as many informal discussions about policy as possible before the formal policy discussion (that is, an RFC) to minimise the amount of public political discussions need to be had about the subject. Because, ultimately, those are exhausting.

Like, to me, it's exhausting to even have to clarify to you that I'm not here to fight anyone; I'm here to have a discussion. And it feels like at every step of the way, discussion is being strongly discouraged. Hell, the reason why I felt the need to comment on your repeated insistence on quoting this one bit from the thread is because it feels like you're explicitly telling me to stop talking here.

Ultimately, I feel that LLMs are toxic to the entire world, and that any usage of them is a net negative on the world, regardless of how useful they are. But I do not think that simply stating this is helpful here because all it does is make people turn off their brain and want to argue about it. The reason why I clarify this here now is only to point out how, despite my ethical opinion being dead-set on absolutely no LLM usage whatsoever, I would rather have a policy discussion to see what is reasonable to ask for and what's not, rather than to quell any discussion on the matter.

Right now, starting an RFC without having that discussion feels like further attempts to silence discussion, even though the RFC process is ironically all about discussion.

[traviscross]

Travis, I'm going to be honest, you repeatedly telling me to reread the request at the top of the thread is feeling less polite and more… aggressive, and I'd like you to consider your actions as well. Telling someone to watch what they say is not a neutral act, and in your case, it feels almost like a threat.

Yes, I understand how it could feel this way. But you've opened many threads. Each has a separate risk of spiraling if we're not careful. While you're seeing this message that we should take care repeated, the others I'm worried about (new people coming into this PR and responding to remarks that can be read as inflammatory) may not see this message in time unless it's repeated where such remarks are happening.

[traviscross]

As far as endgame goes, genuinely, I would rather have a discussion first. Like, if nothing else, that's the point I want to emphasise first. So far, a lot of discussion has gone into how people feel about AI, and summarising what people think about AI, but absolutely none of it has gone into policy discussion... My goal would be to have as many informal discussions about policy as possible before the formal policy discussion (that is, an RFC) to minimise the amount of public political discussions need to be had about the subject

Thanks for this reply. This is helpful in understanding where you're coming from on this.

I had originally proposed to do this policy in a non-RFC form (in rust-lang/leadership-council#273 (comment)); it momentarily moved into FCP. If I hear you correctly, you interpret the fact of us doing this now in an RFC form as making it more final. I can see why it would seem that way. But I don't think it does, and that wasn't the goal in doing this as an RFC. There was simply a worry that, procedurally, we should not a adopt any policy of this nature for the Project without it being in the form of an RFC, because the RFC process is how we ensure the proposal gets sufficient visibility within the Project and that we're able to collect wide feedback from team members. So I took the same policy that had been proposed in non-RFC form and proposed it here as an RFC instead.

[clarfonthey]

It's equally fair to try and include the message in each thread to ensure that others see it as well. I think it's worth clarifying that you're more than welcome to clarify this in your comments themselves; saying "for everyone reading this thread, I just want to reiterate this" would go a long way, but in all the comments, you were directly addressing me.

Making many threads was probably a bad idea. Especially considering how, when one thread was deleted for mentioning an off-limits topic (I wasn't aware this was officially decided beforehand), I essentially realised that no points of value were really lost.

I'm mostly just repeating myself here, and I wouldn't mind if you ultimately decided to mark my other threads as resolved for that reason. I'm just very defensive, and rightfully so, for reasons that if I repeated here, well, I'd be repeating myself. Bureaucratic nuances aren't really something to get into passionate debate about, and that's effectively what's happening here.

To be clear, I would be much more relieved if all this discussion were so boring. Boring discussions don't have high stakes.

[burdges]

Here's the reality: LLM contributions are uniquely positioned as something that is resource-intensive, requires minimal input from a contributor, and creates maximum work for maintainers. They cannot be generalised without losing specificity and should not be.

Amen! It's fine if folks use LLMs on their own time, but even if an LLM edited document winds up reasonable, then it'll still be improved by another human rewrite pass.

[clarfonthey]

I think that the focus on effort is incredibly misplaced, because it tries to encapsulate the nuance I mentioned earlier on LLMs (resource-intensive, minimal input from a contributor, maximum work for maintainers) as "effort" when this is simply not the case.

I would argue that fixing a typo in the docs, or removing ableist words like "stupid" and "crazy" from the docs are contributions that require more work from maintainers than contributors. That doesn't mean they shouldn't be made.

Ultimately, the goal is to ensure that contributors and maintainers work together, and the desire is to make sure that someone is ultimately willing to do the work on their own part. For example:

• If there are unforeseen consequences of a change, the author is responsible for evaluating them and fixing them. If they are unaware of them, they should be able to receive minimal guidance from a maintainer to fix them.
• If there is an undue burden placed on maintainers, authors are responsible for dealing with that, not maintainers.

Here's another example I'll show, again from one of my own contributions. I tried to push forward stabilising the #[coverage(...)] attribute without being aware that I was pressuring a burnt out team member to do extra work. I owned up and stopped. That is what this level of cooperation means.

None of this nuance is conveyed in this description at all. In fact, the lack of description feels almost intentional: trying to own up what "effort" means, means actually limiting LLM contributions in a meaningful way, which a lot of folks in the project have been trying to avoid.

I'm tired of avoiding the issue. I'd been meaning to push harder for an actual policy to be written, and it feels like the force at which this RFC has been push is pushing my hand, and the hands of other people who have wanted a policy, to rush such a policy harder.

I doubt this is your intent. But again, it goes against everything this policy should stand for: working together. The fact that this policy seemingly was not done in that way feels telling.

[traviscross]

I'd been meaning to push harder for an actual policy to be written, and it feels like the force at which this RFC has been push is pushing my hand, and the hands of other people who have wanted a policy, to rush such a policy harder.

It's a reasonable point, I think, that maybe we should take it more slowly. Maybe it's OK to leave things, for now, to the moderators who "know it when they see it". Maybe we don't need the council to adopt a policy immediately. In other places I've suggested similar things may be true.

On the other hand, it's also possible that adopting some policy may help us in avoiding or moving on from the kind of upheavals that happened over the weekend. Reasonable people have made that point too.

This proposal is acting on that second theory: that adopting some policy -- that encodes the intersection of the principles we share -- might help us to move forward.

(As also mentioned in #3936 (comment).)

Here's another example I'll show, again from one of my own contributions. I tried to push forward stabilising the #[coverage(...)] attribute without being aware that I was pressuring a burnt out team member to do extra work. I owned up and stopped. That is what this level of cooperation means.

I think this is covered by the compassion point in the policy. Seeking to understand the situation and then stopping if appropriate was showing compassion. That's what we're asking.

I would argue that fixing a typo in the docs, or removing ableist words like "stupid" and "crazy" from the docs are contributions that require more work from maintainers than contributors. That doesn't mean they shouldn't be made.

In my view, it's easy to underestimate the amount of effort it takes a contributor to make even a seemingly-small change in a helpful way. On the Reference, e.g., if someone puts in the effort to fix minor errors correctly -- e.g., by first reading our dev guide, ensuring carefully that each change is actually correct, writing a clear PR description, etc., then the effort we need to spend as maintainers is small to merge it. But if someone does a low-effort job, e.g., by having commits that mix some correct changes with others that aren't correct, "fixing" things in a way that doesn't follow our conventions, not explaining what the person was trying to do, etc., then these PRs can take us as maintainers a lot of effort to figure out.

Even for small changes, in my experience, effort (as defined below) by the contributor is what makes the difference between whether or not the contribution is valuable.

• Effort: If you're not putting in the same level of effort a maintainer will have to put in to review it, then you're not adding value. This level varies by the task and by the cost of review, but it's never less than being careful and thoughtful.

[clarfonthey]

Honestly, I feel like my main reply to what you said is reiterating what I said in a separate thread:

Overall, this policy disappoints me. Not because the general idea isn't good, but because it feels… spineless. It's trying to generalise something that can't easily be generalised.

As you've mentioned, there's a possible way to interpret these policies where we both completely agree on what's been said. My main issue is that this level of interpretation isn't really suitable for an RFC, and the goal should be making something that does not require this much extra interpretation. Although I guess that the main point is the disagreement on whether a stopgap RFC is needed or not, which is kind of a separate argument.

[traviscross]

Although I guess that the main point is the disagreement on whether a stopgap RFC is needed or not, which is kind of a separate argument.

This is, indeed, a key point.

[yaahc]

On the other hand, it's also possible that adopting some policy may help us in avoiding or moving on from the kind of upheavals that happened over the weekend. Reasonable people have made that point too.

FWIW I do not believe this policy as written would have done anything to prevent the situation with Jack's blog post. The issue with that post was not whether or not his blog post was a low effort contribution. Jack and the vision doc team members spent many hours on that blog post and the underlying work. I have not seen any complaints about the content itself, only with how it was presented as a result of LLM usage.

The issue is that the usage of an LLM, which he later confirmed, was perceivable at all despite attempts to ensure the blog post came across with his own voice. Knowing that the blog post was written with an LLM caused an immediate negative emotional response for many readers and more than a few assumed the blog post was entirely LLM slop and avoided reading it entirely.

I can't fault people for this reaction, its an extremely reasonable heuristic in the age of AI slop where there are massive effort disparities between creating slop and determining whether some content is slop. Sometimes the only defense against this DDoS of attention is to make a quick risk assessment and refuse to engage when the risk passes a certain threshold. In this case many of the project members who reacted negatively to the post and wanted it retracted failed to include the source of the blog post in their risk assessment, even with LLM involvement I think the risk of Jack producing a low effort blog post was low.

This policy is not going to address situations when the assessment of whether or not something is low effort are wrong. Its just going to shift the blame onto the reviewer for not putting in the effort of engaging with potential slop when they get it wrong. This risks putting reviewers in a situation where they must deeply review all content that seems like slop so they can prove exactly why its slop, a job that nobody wants. The original policy by @jieyouxu did a much better job at making it clear that even looking like slop is a sufficient reason for reviewers to refuse to engage.

Without clearer guidelines on where and when LLM usage is acceptable and broad trust across the project that other project members will and are adhering to this policy we're going to keep having problems like this.

[tmandry]

This policy is not going to address situations when the assessment of whether or not something is low effort are wrong. Its just going to shift the blame onto the reviewer for not putting in the effort of engaging with potential slop when they get it wrong. This risks putting reviewers in a situation where they must deeply review all content that seems like slop so they can prove exactly why its slop, a job that nobody wants. The original policy by @jieyouxu did a much better job at making it clear that even looking like slop is a sufficient reason for reviewers to refuse to engage.

I think that's right. At the same time I see the wording in the RFC as serving two other needs:

1. External guidance so that anyone acting in good faith knows how to contribute well.
2. A framing that explains how we arrive at the decisions and policies we have, both externally and to ourselves.

As a practical matter it is still important to close the gap between "here's how we think about effort in contributions" and "here's how we concretely protect ourselves against DDoS" (to use your term). Having a big-picture framing is useful as we establish and update more tactical policies.

[Kobzol]

@rfcbot concern needs-discussion

Since we turned this into a Project-wide policy RFC, this really needs more discussion to happen. The original idea was to do a tightly scoped rust-lang/rust stopgap to block slop, but then it was expanded to something else, a much broader policy. I agree that it's time to have that policy, but the way to do that is not post a 30 line text and insta-FCP it.

We should do this properly.

[traviscross]

Yes, as mentioned on the FCP proposal:

But please don't mistake that for foreclosing discussion. This is being brought here as a request for comment so as to ensure wide visibility and participation, particularly among members of the Project.

I'm happy for us to take our time too. It didn't seem to me there was much risk of this going immediately into FCP; otherwise I might have filed such a concern myself.

[m-ou-se]

We don't call for an FCP when it's clear we still need a lot of discussion before consensus. This is clearly controversial and nowhere near the stage for the "final comments". FCP is the tool to confirm once we have a reasonable expectation there is consensus.

@rust-rfcbot fcp cancel

[rust-rfcbot]

@m-ou-se proposal cancelled.

[nikomatsakis]

@rfcbot concern needs-discussion

Since we turned this into a Project-wide policy RFC, this really needs more discussion to happen. The original idea was to do a tightly scoped rust-lang/rust stopgap to block slop, but then it was expanded to something else, a much broader policy. I agree that it's time to have that policy, but the way to do that is not post a 30 line text and insta-FCP it.

We should do this properly.

My two cents: I'm not commenting on the text of this RFC per se (in fact, I've not read it, though I saw the policy issue it's derived from) but rather on what "do this properly" might mean.

I think the reality is that we are nowhere close to anything resembling a project consensus around LLM usage. One thing I think we can all agree on is that LLMs are having a huge impact on our industry and I don't think we're going to be able to "set a policy and be done" any time soon. The relationship of Rust and the Rust project to LLMs is going to be evolving for quite some time.

I've come around to the idea that the best way forward is to find areas of consensus and act on them in a narrow fashion, accumulating knowledge as we go, and not being afraid to revisit those areas when the facts change (which is happening quite rapidly, in my view). I commend @jieyouxu and @traviscross for finding some language that they're both satisfied with, even if it's not what each of them would want. And I commend @ehuss for bringing it to broader project visibility.

I think when we try to reserve RFCs to be some major thing that always requires a lot of discussion, time, and effort, what winds up happening is that we make impactful, important decisions in issues. And then people are surprised when they hear about them. So I don't think we should always do that.

EDITED: cut to be shorter, remove text I felt wasn't conveying what I intended

[nikomatsakis]

I'm generally in favor of this RFC. I have a few suggestions for how it could better set the stage.

View changes since this review

[nikomatsakis]

Suggested change

	We adopt a *no low-effort contributions* policy for the Rust Project.
	**This RFC lays out a minimal policy intended to codify existing practice when it comes to "low-effort PRs" (e.g., videcoded PRs).**
	*What this RFC is not:* This RFC is not intended to be the final word when it comes to LLM usage. There is no consensus presently as to what a more holistic LLM policy might look like. There is ongoing conversation amongst project members on the topic. As areas of consensus emerge, we expect to author further RFCs, or amend this RFC, with policies in those specific areas.

[joshtriplett]

While I think this needs a further change to move fully away from the "low-effort" framing, I agree with the second paragraph of this suggestion, if we find we need an interim policy for the most egregious cases.

[nikomatsakis]

Some basic background would probably be useful here. Basically laying out the increasing trend in automated PRs, the project's shared goals, and the fact that there is no overall consensus on the best way to approach LLM usage.

I can take a stab later.

[jieyouxu]

@nikomatsakis you might be able to recycle some of the text/links from the opening issue comment of rust-lang/leadership-council#273 (comment) (see the Motivation, Prior Art, etc. sections)

[nikomatsakis]

We don't call for an FCP when it's clear we still need a lot of discussion before consensus. This is clearly controversial and nowhere near the stage for the "final comments". FCP is the tool to confirm once we have a reasonable expectation there is consensus.

Just a brief clarification -- consensus amongst the decision-makers, not necessarily amongst the people commenting on the thread, which is frequently unachievable. (And it seems like there isn't the former, that's fine, but I think this is a point where there is often confusion.)

[Kobzol]

@rfcbot concern needs-discussion
My two cents: I'm not commenting on the text of this RFC per se (in fact, I've not read it, though I saw the policy issue it's derived from) but rather on what "do this properly" might mean.

I've come around to the idea that the best way forward is to find areas of consensus and act on them in a narrow fashion, accumulating knowledge as we go, and not being afraid to revisit those areas when the facts change (which is happening quite rapidly, in my view). I commend @jieyouxu and @traviscross for finding some language that they're both satisfied with, even if it's not what each of them would want. And I commend @ehuss for bringing it to broader project visibility.

As you know, I am all for creating smaller targeted policies ^^ Rather than trying to create some uber AI policy - I definitely do not want to do that here!

To clarify, by "doing this properly", I meant that we should properly discuss this scoped policy about AI contributions, which did not really happen. On the original issue, there was some proposed text of the policy that I was ok with. Then the text was changed in a significant way, and an FCP was started immediately after that, so there was not opportunity to discuss those edits. I only checked my box in case we did go forward with just the FCP, to have at least something to stop the bleeding, and also because it was explicitly stated by TC to be just a vibe-check. But since we now moved to an RFC, I want it to be an actual RFC that we can discuss, before jumping to FCP proposal.

[Kobzol]

In terms of prior art, the RFC should link to:

• Policy proposal: No low-effort contributions leadership-council#273
• Policy: Empower reviewers to reject burdensome PRs compiler-team#893
• https://github.com/rust-lang/moderation-team/blob/main/policies/spam.md/#fully-or-partially-automated-contribs
• https://forge.rust-lang.org/how-to-start-contributing.html#etiquette

Because essentially this RFC would subsume all of these and create something official that we could link to.

[Kobzol]

I'm not a big fan of the low-effort framing, because some genuine contributions are low-effort, and it's fine - fixing a typo or a CI config can be low-effort to do, but when it's also low-effort to review, and the contribution is valuable, it's not an issue on its own.

I think that we really don't have to reinvent the wheel here. rust-lang/leadership-council#273 and rust-lang/leadership-council#273 (comment) very helpfully categorize existing AI contribution policies.

I think that the golden standard is the LLVM framing (https://llvm.org/docs/AIToolPolicy.html#extractive-contributions) about extractive contributions - if it takes way more time to review/understand something than the time you put into it, that is the problem.

We should still call out that everyone should own and understand what they submit, and that there has to be a human in the loop, but I'd suggest renaming the whole thing to be something like "No extractive contributions".

[traviscross]

The RFC defines what we mean by effort:

• Effort: If you're not putting in the same level of effort a maintainer will have to put in to review it, then you're not adding value. This level varies by the task and by the cost of review, but it's never less than being careful and thoughtful.

In my reading, this is similar to the LLVM standard. We're saying that the level of effort 1) must be no less than that required for review, 2) will vary therefore based on the cost of review and the task, and 3) is never less than being careful and thoughtful.

(It then defines low-effort contributions as those that fail to satisfy that prong or the other two, accountability or compassion. This makes sense, to me, because, e.g., interacting with others without compassion is also low-effort.)

While I do also like the "extractive" framing that LLVM uses, that theory too must be explained in order to make sense -- and I think it takes more to explain that theory than it does this one. So I'd probably ask, is your concern with this definition or with the term we're defining? I.e., if we said, "Contributions that fail to satisfy these criteria are $PLACEHOLDER contributions", then setting aside the value we choose for $PLACEHOLDER, does this otherwise sound workable to you?

[Kobzol]

"Carefully self-reviewed" can be seen by some people by simply reading the output and nodding their head "yes, this looks like Rust code".

We should be more explicit here and ban fully AI-generated contributions. It is very unlikely that even a trusted and skilled Rust contributor could create a contribution that is fully AI generated without wanting to change at least something in it, and the sign of understanding something is that you can modify it in a way that it improves, or at least still works, so being able to at least modify the output of AI in a reasonable way is something that partially proves that you understand it, and can explain it to others, which is what we want.

[traviscross]

Here's my thinking. Two axes:

1. The solution might only be a few lines. If the model gets it right, and you carefully verify that, then other than due to arguments that would tend toward banning model use on other grounds, this seems OK. It would seem strange to say that you've violated our expectations on this just because it was in fact fully generated.
2. You iterate with the model. Let's say that for a larger PR that the model does fully generate it. Then you review and work with the model on detailed changes. It generates those changes too according to your notes. You do many rounds of this until you're satisfied. In some sense, the PR is still fully AI-generated. But this too seems OK (other than by accepting the premises of the arguments that would tend toward a blanket ban).

It would spill a lot more words to try to distinguish these from other cases, and then it really would make this into a policy that is describing acceptable model use, so I went with a consistent framing that the key distinction, for now, is that the person is taking responsibility for what is submitted, as the rules say. To do that means at least carefully reviewing it, so that's what I put in the examples.

(I previously gave this reply in Zulip in #council > Project perspectives on AI. I note this simply for purposes of textual provenance.)

[nikomatsakis]

@Kobzol

To clarify, by "doing this properly", I meant that we should properly discuss this scoped policy about AI contributions, which did not really happen.

OK, so...let's discuss? Isn't that what the RFC is for? I think it's fine to raise a concern on this basis, that you would like more discussion. I just don't understand what I perceive to be the undertone of "outrage".

Let me lay out what I'm reacting to. Frankly, I think @traviscross gets an undeservedly bad rap from a lot of the project in some areas and has been put into a bit of a bind here.

On the one hand, I've heard more than one person complain, at times publicly and at times privately, that TC is putting up procedural hurdles to progress. And in fact, in the last LC call, TC was saying that they felt that arriving at a policy we could all agree on was going to take more time, and @jackh726, @oli-obk, and I (and possibly others) were pushing back saying "we don't want to wait forever, let's move forward with something targeted".

To his credit, TC put his money where my mouth was. He talked to @jieyouxu and they found something they could both live with. You say it's watered down, sure, that's one perspective, I don't happen to share it and I suspect others don't. Regardless, they put it forward.

And now people are saying "TC is moving too fast! There's no time for conversation!"

So what's the alternative, we set the policy quietly on a github issue where there are fewer people looking? How is that better than having an RFC that acknowledges that this is a complex area and that we are documenting current practice at the request of the mod team while we work out more details.

And let's not forget, it wasn't TC that proposed the RFC, it was @ehuss who (and I agree with him) felt that setting this kind of policy on an LC issue was not going to allow for enough conversation.

All that said, I agree the RFC needs a bit more context, it was... quite minimal. But I can understand why TC just put up the easiest thing he could on short notice.

[yaahc]

We want to make you better able to help us.

I don't agree with this framing and object to the current phrasing. I think this is true but its not the core goal and it comes off as self serving. I personally do not expect every contributor to come back and take the investment of effort we put into them and reapply those skills to the project. I am perfectly happy if they go and use those skills in their personal projects or professional environment. What matters to me is that people learn and grow and that our investment in our contributors contributes to the health and development of the open source community at large.

[traviscross]

What matters to me is that people learn and grow and that our investment in our contributors contributes to the health and development of the open source community at large.

Yes, understand. Thanks for raising that. For my part, if it contributes to the health and development of the OSS community at large, I still consider that helping us. This is one place where a longer document could be more explicit about saying this sort of thing.

[apiraino]

What matters to me is that people learn and grow and that our investment in our contributors contributes to the health and development of the open source community at large.

^ This 💯

Though I am personally reading the "investment" part that Travis suggests not with a capitalistic framing (i.e. a mere "ROE") but more with the "no extractive contributions" LLVM framing.

We can wax philosophical but I think this hints at the core of our social contract: we prefer contributors learning and internalizing how the Rust project work rather than someone learning how to orchestrate an LLM to send us patches. The difference may seem subtle but in the former case we have a person growing up, in the latter we have a ... uh ... dog trainer teaching Spotty how to jump through circles.