Instilling “GitHub-native” conventions, especially for LLM usage

[cassidyjames]

Some things I've started to notice with several—especially LLM-assisted—contributions to Coop and Osprey:

• Overly-verbose descriptions: Oftentimes it is okay or even preferred for an issue or PR description to be brief and to the point; contributors should not be presented with a wall of bullet points with difficult-to-parse code tags that they have to comb through to understand what the issue is, or what a PR is doing—especially if that text is what they'd get straight out of an LLM when reviewing the issue or PR.

• Mixing multiple concerns: Just because two concerns are related doesn't mean they should be filed as part of one issue or fixed in one PR. Similarly, comments on an issue or PR should stay focused on that specific scope rather than introducing other related topics; it's much harder to follow a conversation and perform triage if the original issue or PR is about one thing, then the comments drift to discussing a separate but related concern.

• Lack of GitHub-native linking: GitHub has a ton of niceties for linking to issues/PRs/discussions, showing cross-references in timelines, automatically closing issues, and even previewing code blocks. Manually crafting links to these things in Markdown makes the issue/PR/comment/discussion harder to read, and also misses out on some of those automatic behaviors.

To be entirely clear, I'm not criticizing any contributors for these things! Rather, I see it as a gap in our documentation, processes, and culture; as a relatively young open source project and organization—especially starting up in a time of LLM coding agents—it's our responsibility to set the expectations and norms we want to see. I also realize that I'm someone who has been a heavy user of code forges across large open source projects for two decades now, learning a ton as I went; most people are not me, and many may not be used to working in the open or using platforms like GitHub to their full potential (or at all!).

In a perfect world, we would have a culture where regardless of tools used to contribute, issues and PRs would follow common formats and conventions. To get to that place, I think we need to do more work to:

1. Decide what those conventions are
2. Document them
3. Follow/enforce them

I was going to start by documenting things, but I realized: we need to decide on things first, and then we can document, build the culture, and figure out how best to follow/enforce things. Here's a rough draft/straw man proposal of what I think I'd like to establish; I'm sharing it explicitly for it to be debated/commented on/picked apart—or if everyone shrugs and says, “Sounds good!” that’s helpful, too. 😂

Conventions

Issue and PR descriptions

Issues and PRs should ideally focus on what the human contributor thinks/understands about the issue or PR over all. PR descriptions should focus on the why, whereas the code itself is the what/how. Commit messages should tell the story of the iterative process, including why certain decisions were made along the way.

If using an LLM to assist in filing an issue or write a PR description, avoid submitting text directly from the LLM; instead, use that context to write up your own understanding/summary. If something came from an LLM but you're not able to fully verify it independently, treat it as if a colleague over your shoulder paired with you or pointed it out; e.g. "I did some digging with Claude, and it pointed me to…" or "I am not sure what's going on here, but according to Claude, it might be…". This sets the correct expectations for other contributors when commenting on or reviewing the issue or PR.

The same guidelines should be followed for comments on issues and PRs.

Breaking down tasks

Issues should generally be individually-actionable tasks or problems to be solved. Issues are not ideal for starting a discussion, posting the results of a broad audit, or sharing broad feedback; instead, break the feedback down into individual concerns and/or open a GitHub Discussion from which individual issues can be filed.

It's tempting to file one issue covering two distinct but somewhat related concerns, but it can make triage and planning harder. It's also easier to close a small redundant issue than it is to split up one too-broadly-scoped issue. If an issue is over-scoped, either reduce its scope by editing the title and description for clarity, or file new sub-issues that break it down if the broader issue is still helpful to track individually.

Keep comments on an issue or PR focused on the original scope rather than introducing other related topics. If there are related concerns, feel free to file a new issue that links to the original; the original issue or PR timeline will display the mention to aid with context.

GitHub-native conventions

Issues, PRs, and discussions on a repo can be linked to with just their number (which is a shared pool across all three): i.e. typing #123 links to issue, PR, or discussion number 123 on the same repo. When used in a list (whether unordered/bulleted, numbered, or checklist), GitHub will also expand the link preview to include the state and title. When linking to an issue in these ways, GitHub will also show the cross-reference in the linked item's timeline.

Similarly, avoid using #123 formatting in an issue, PR description, or comment if it does not refer to an item in the same GitHub repo with that number; for example, avoid writing about concern #1, and be careful to wrap pasted content from logs in backticks (```) to prevent accidental linking.

When a PR fully-addresses an issue, it should include fixes #123 in the description; GitHub will automatically close the linked issue when the PR is merged into the main branch (there are more keywords: forms of close, fix, and resolve). If a PR partially addresses an issue, do not use fixes or another keyword; instead, cross-reference by writing something like related to #123.

When referencing a specific line or section of code, link to a permalink of the line(s); you can get a permalink when browsing the code in the GitHub UI by selecting the line(s) of code, then from the ⋯ menu, select Copy permalink. Permalinks include the specific commit so the reference doesn't drift if the line changes in future commits. In issues and PRs, GitHub front-end will nicely render the selected line(s) in a code block with a link to browse the original file.

Following/enforcing

How can we help each other and new contributors follow these conventions (or whatever conventions we settle on) without requiring them to read a wall of text like this discussion?

First, I do know that existing contributors will need to lead by example; lurkers and new contributors will learn more from following what is commonly done than they will by reading documentation.

Are there other effective ways to help enforce/remind people of project conventions? Will better issue and PR templates help? For LLM-assisted issues and PRs, can we expect to address it via AGENTS.md? Can/should we use some sort of bot to flag potential issues via a comment? I'm open to ideas here!

[ThisIsMissEm]

I'd maybe push back on "Mixing multiple concerns" — you can always decide "nope, not in scope, create issue from comment", that is a valid thing to do as a receiver of a comment on a PR.

"PR descriptions should focus on the why, whereas the code itself is the what/how." — yes, unless the code is doing something unobvious, in which case calling it out in the PR description is okay and/or self-reviewing your own PR to make others aware of aspects you think they may want to pay special attention to.

For AI assisted issues, a strong recommend on always using a prompt like "Summarise the core aspects of this issue, do not give an overtly verbose description unless it is critically needed, try to keep the summary under 600 characters."

As for following, a CONTRIBUTING.md is a good place to start, you could even include gentle instructions for AI agents in there. Ideally an AI will read that that file before interacting and filing an issue/pr.

As for enforcing.. this gets tricky and tbh could negatively affect the project in just the same way as not following the contributing guides can.

[Dcallies]

LLM use in GitHub projects is certainly an area of debate at the moment.

@cassidyjames - one thing I'm having trouble pulling out from your discussion is quantifying negative impact from the highlighted concerns. Preference for the use of the "Discussion" feature over the "Issue" feature for certain types of tasks seems like the inverse might also be true in other repos and may not be universal.

Can you give some more context on what the proportionality of the harm is today, and who is paying the cost? Is this your own code reviews on Coop on Osprey or are you speaking on behalf of someone else?

One principle I've been trying to hold for https://github.com/facebook/ThreatExchange/ is to focus on the final result vs how we get there - maintainers can help contributors (especially first-time contributors) get the rest of the way without getting hung up on the forms. A PR to some extent only matters to give context to the reviewer. An issue only matters until it produces a fix. Being too strict on the "how" can end up being a tax if it's not in proportion to coordination needs proportional to the size of the project. A project like https://github.com/django/django has a huge contributor base, and coordinating all those people depends on those forms.

Coop appears to be primarily 1 contributor - Osprey seems to be primarily 3 (overlapping with Coop).

[taobojlen]

I opened a separate discussion here to propose outright banning LLM-generated discussion on ROOST's GitHub, but I guess it makes most sense to fold that into this topic!

@cassidyjames I think your conventions are generally very good ideas, but I wonder if they're treating symptoms. The issues you point out seem to be primarily associated with LLMs! So I wonder, is it too harsh to ask that everyone writes their own words, and make sure to cite/quote LLMs explicitly if copy-pasting their output?

I'm not sure how to do that in like a non-punitive sense. If there's a written policy somewhere, then at least we can say "thank you so much for the contribution, we ask that PR descriptions/discussions/etc. are written in your own words, would you mind writing it yourself before I review" or similar. That seems reasonable to me, but you'll know better than I do how that sort of thing affects community + contributions!

I do think this is an issue and it certainly affects my experience, so I'd suggest that we introduce a minimal policy pretty soon and iterate if needed.