Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
149 changes: 149 additions & 0 deletions culture/review_culture.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
> If you want to go fast, go alone. If you want to go far, go together.

As a community, we value going far over going fast. To go far, we need to stay
aligned in our values, goals, and practices. One of the ways that we do this is
through cultivating review culture.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a hugely annoying detail (and something I don't tend to do for the repos I maintain) but I've seen it in various communities so I want to mention. Should there be standards around how to write the markdown? For example, I've worked with folks that like to have one sentence per line, because it makes editing / commenting / seeing changes easier on GitHub.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to have standards around markdown, and I think they are out-of-scope for a guide on review culture in general. They would, however, be very relevant as contributing guidelines for this repository (sourcecred/docs) in particular. I made an issue on this repo to discuss: #3


# On Review Culture

Review culture came to us from the practices of software engineers. Large
codebases are incredibly complex: they contain the ideas and work of hundreds
of people, manifested in millions of lines of code. They need to remain
logically consistent, even as they grow and change. A big part of how engineers
manage is through code review: every single change to the system is reviewed by
at least one other person before becoming official, or "merging".
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would add some context around merging, since it's a GitHub term. Maybe like:

at least one other person before becoming official, or "merging," meaning that requested changes are integrated into the main codebase.

This has me thinking that we need an entire "Welcome to GitHub, here are some useful terms" section, but maybe a link (later in the document when it's appropriate) would do the trick - those that are scratching their heads would click, others would continue reading.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, since we're using GH as source-of-truth for docs, and we want less-technically-experienced contributors to be able to help write docs, we need some guides for them.


Ensuring that all work is reviewed brings several benefits. Most obviously, it
leads to higher quality work, because the final output includes feedback and
suggestions from others. Just as importantly, reviews enable the exchange of
knowledge, context, and perspective within the team. In a good review, both the
contributor and the reviewer can teach each other about how work gets done on
the project. Finally, review ensures that every part of the project is
understood by at least two people.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably the number of reviewers required for merge would vary based on the repository? Later we should figure out the standard for deciding this. Intuitively, small trivial things == one review, if it takes more than 5 minutes, 2 reviewers, and more robust changes (code or docs) should have 3-4.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that it depends on project and context, but in my experience it
has been far more effective to bias toward fewer reviewers than more.
Pulling in two reviewers for a non-trivial change is a huge productivity
hit, both because the extra reviewer takes the time to review the code
and because the presence of more people encourages bikeshedding and
nitpicking. In fact, all projects that I’ve been on that started out
with a many-reviewer culture eventually adopted a guideline of “prefer
one reviewer; if you request multiple, explicitly call out what you want
from each person and why one reviewer doesn’t suffice”, or similar.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you've experienced this multiple times, I'm all for fewer reviewers - this feels a bit like (what are they called, anti-patterns?) because intuitively you'd think more is better, but... (insert metaphor with cooks and soup broth). I like the idea of erring toward fewer, and leaving it up to the requester (and possibly the first reviewer) to decide if more are appropriate.

I wasn't familiar with the term bikeshedding - here is a helpful link for others that might not be.


# The Review Process

## Changes
The review process orients around _changes_, which are the units of work being
reviewed.

A change can take many forms, for example:
- Adding or changing official SourceCred documentation (in [sourcecred/docs](https://github.com/sourcecred/docs))
- Adding to or changing the SourceCred codebase
- Proposing a re-organization of the [SourceCred Discourse](https://discourse.sourcecred.io/)

## Participants in the Review Process

### Contributor

The contributor is the person proposing a change. Anyone can decide to be a
contributor: the only requirements are that they have a change to propose, and
that they agree to be a respectful participant in the community.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something like:

The review process starts an implicit agreement that you will follow the points specific in the community code of conduct, and the guide for contributing.

I was trying to think of a better word than "rules" and came up with "points" and it still isn't very good.

### Reviewer

The reviewers are the people who review the change, suggest changes to it, and
eventually approve it. Anyone in the community can act as a reviewer: the only
requirements are that they have some insight to share (or questions to ask)
about the change, and that they agree to be a respectful participant in the
community.

### Maintainer
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Above you say that “every single change to the system is reviewed by at
least one other person before becoming official”. Should it be noted
somewhere that at least one reviewer must be a maintainer? I think that
this is a helpful distinction, because one common point of confusion
from those unfamiliar with free software is, “if anyone can change it,
how doesn’t it just get broken from poor contributions, much less overt
vandalism?” Indicating that at least one reviewer must be explicitly
trusted rather than “anyone in the community” addresses this.

May also want to mention something about this in “Approval and merging”
below.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually don't think it's a requirement that every approved change was approved by at least one maintainer. Rather, I think every change needs to accumulate "sufficient review" for its level of complexity and potential controversy.

For example, suppose there's a change which fixes a typo in the README. Because the complexity and potential for controversy is minimal, a review by any reasonably-trusted member of the community is sufficient. On the other hand, a massively invasive cross-cutting change may need approval from two maintainers before being approved (in practice this might look like the first maintainer commenting with "this makes sense to me, but because it's so invasive, i'll let (other_maintainer) give canonical approval".

im open to an improvement to the guide that elaborates on this... weak preference to merge the guide as is. (as a general principle, i don't want merging docs to block on every potential improvements--that's what future pull requests are for.)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, to add a section about it. But also suggest a new PR for this.

(In my experience this point varies wildly between communities. So this document seems like the right place to touch on it.)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think (at the end of the day) someone with write access has to be the one to merge the PR, so a sense the final say always comes down to someone that is technically a maintainer. But a maintainer being pinged to merge a PR that has accumulated sufficient review without reviewing him/herself is different than a maintainer doing the actual review. But if I put myself in those shoes, even if we had a rule about accumulated sufficient review, if I were a maintained pinged to merge it, I'd still want to take a look.

If a tree falls in the woods does it make a sound... if a maintainer browses to a pull request... something something :)


Maintainers are responsible for the health and maintenance of a portion of the
project. They are the stewards of the review process, and have the privileges
needed to move forward or block a change. Not everyone can be a maintainer; it
requires deep knowledge of the project, high trust from the community,
experience as both a contributor and reviewer, and substantial commitment.
Since we invest maintainers with special privileges and trust, we have high
expectations of them. Those acting as maintainers must be respectful,
considerate, and judicious in the way they exercise their powers.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe something here about (general) kinds of review? Or if that is out of scope to be too much detail, we can link to something. I'm putting myself in the shoes of someone that might be new, and what I'm hungry for at this point is to be able to take this abstraction and see a concrete, written out example. E.g., "There is a repository with documentation on running software. Sigmund realizes there is an undocumented command and opens an issue on the software repository to ask if it would be useful for him to write documentation for it. He gets the green flag and checks out a branch, writes some instructions in markdown, pushes to his branch, and opens a pull request to request review for his changes. In the words of the three bears, the changes aren't too small, but aren't too big, so two reviewers is just right, so when Henny and Hunny receive a GitHub notification for the pull request, they feel like they can help out, and give Sigmund feedback on his markdown file. He is pointed to the contributing.md file, where it mentions some small details like putting one sentence per line in the file, and adding languages to code blocks for syntax highlighting. When Sigmund fixes all the suggested changes, H&H approve, and a maintainer with write access is pinged to click on the "Merge" button. After merge, the documentation is rendered live in the software web interface, and the world is right again!

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think as a followon we should have some sort of "scope of review in SourceCred" document where we talk about what the domains of review are. For example, review of code and official docs happens on GitHub, review of weight changes (currently) happens on Discourse, review of plans to change how the Discourse is organized take place on Discourse, etc. Happy to see a pull request adding this. :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tracking suggestion here #6

## Lifecycle of a Review

### Proposal
The first step is for a contributor to come up with and propose a _change_.
Before starting work on the change, they may want to run it by the community.
Often times community members will have great feedback on how to approach the
change, and whether it's valuable.

Once the contributor wants to move forward with the change, they will formally
propose it. They need to do two things at this stage:
- Flesh out the change (e.g. writing the code), so that there is something to review
- Explain the change (e.g. writing a pull request description), so that reviewers can easily understand it

Both steps are vital, and contributors should put real effort into explaining
the change well. Doing so is respectful of reviewers' time, and makes it easier
for others in the community to follow along with the project and get involved
themselves. Finally, the change's explanation acts as documentation for future
contributors hoping to understand the project.

Finally, before requesting a review, the contributor should _self-review_ their
change. It's amazing how many issues can be spotted and fixed during
self-review. This is respectful of reviewers' time, and makes the process a lot
smoother.

### Review

Once a change is ready and has been self-reviewed, it's ready for review. Since
most SourceCred artifacts are tracked via repositories on GitHub, this will
usually mean creating a pull request.

Once the change is proposed, anyone in the project can add their own review.
The person proposing the change can also request specific reviewers, if certain
individiuals have context or knowledge thet will make their reivew particularly
valuable.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we have various platforms for discussion, it might be useful here to suggest a standard way to alert people on those platforms that "Hey this PR needs some love!"

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a really good idea ☝️ . New contributors will probably have less confidence communicating with folks, and err on the side of being less assertive in communicating the state of their requests. It's easy to feel naggy when you're requesting somebodies time/attention.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To some extent we can also automate this with actions, given that platforms involved have APIs / programmatic access.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like the idea, let's continue at #4


On finishing their review, the reviewer can:
- Ask questions to better understand the change
- Suggest changes to it
- Approve the change
- Suggest closing the change as 'wontfix'

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we don't want to consider the reality that some changes don't get accepted, and get a wont-fix?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. Added a line mentioning this case.

Many changes go through a few rounds of reviews before being approved.

### Approval and Merging

During the review process, a successful change will accumulate "approvals" from
satisfied reviewers. At some point, the change will have "enough approvals" and
will be ready to merge. There's no hard-and-fast rule to determine when a
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While this makes sense for code changes, changes to official docs, I think relying on GitHub as a governance mechanism (as is done in Bitcoin, Ethereum, etc.) leads to problems of centralization when using it for more broad governance proposal. For instance, if someone proposes a change to a document artifact adding a new principle to a values statement, does the maintainer get a voto due to their monopoly on merge access? Something like this seems better decided with cred-weighted voting or other mechanism.

Copy link
Contributor

@Beanow Beanow Jan 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely share this view. While GitHub may have an open culture, from a technical perspective it uses authorities (the maintainers). To protect you from those authorities, you have this button
Fork One-click to be president of your own corner of the internet!

And this works very well for things that are inexpensive to copy. Like documentation and code.

But when something won't be easily copied:

  • Like the people in a community
  • Or the domain name it has
  • Or the funding it received

That Fork button gives you no protection at all. Nice people might try to reduce the 'cost of forking'. But it's not when you agree with nice people that you need protections 😄.

So using only GitHub as your governance model, is the equivalent of giving a sitting president the power to veto the outcome of an election... There are better ways to do it.


That said. Reviewing, maintaining and governing are very different things so I think it's unrelated to "Review culture". Let's make a document in this repo to share some information about how the SourceCred community does governance.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's make a document in this repo to share some information about how the SourceCred community does governance.

This is a good idea. Though the process is in enough flux right now I'm not sure it's worth the effort yet. Once we settle on some more formal governance mechanisms (e.g. get cred weighted voting going), and those are fairly stable, it makes sense to document them well. For now, linking to the TBD post on Discourse post as we've been doing seems to suffice. Though as we build out user guides for newcomers, we should be on the lookout for opportunities to convey all of this stuff better.

change has enough approvals. Generally, simple and small changes are easy to
approve, whereas changes that are breaking new ground, or have wide-ranging
consequences, will require more reviews. If contributors are unsure as to
whether a change is ready to merge, they should ask a maintainer.

### Changes that never merge

Some changes don't ever merge. A few good reasons why this may happen include:
- Reviewers may find that the change isn't actually needed, is basically
unsound, or isn't aligned with the project's values or goals
- Over the course of review, the contributor or reviewers may find a different
and better way to achieve the same goals
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would add something along the lines of "In these cases, maintainers should openly communicate how their thinking or the situation has changed from the initial opening of the pull request, and make an effort to both thank the contributor, and look for opportunities to still involve him or her with future development."

The general idea is that you wouldn't want to say "OH sorry we don't need / want this anymore" but something more like "Our thinking has changed and we think this [other thing] might be a better strategy to pursue, would you be interested in helping with well scoped task]?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This ☝️ is a really good idea. Concrete examples are really great IMO, especially to those unfamiliar with the process.

A bit of a tangent, but: It can be really nerve-wracking to submit a pull request; one thing that is not at all nerve-wracking about submitting a PR to SourceCred is formatting your PR in terms of commit structure and message, because instructions and rationale are laid out clearly and comprehensively in the Contributing Guide and there's no grey area. We could aspire to something similar in terms of etiquette; providing concrete examples about things to do in non-ideal circumstances feels like a way to remove grey area.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's probably implied, but pull request templates can help with a nice checklist of TODOs when opening the pull request, and specific groups to mention if the requester needs help with a particular issue. For example, I've used GitHub since 2011 but I still have cases when I'm not sure about a particular bullet point in a pull request template. The pull request template should have a catch all for this, because we just can't predict all situations / questions that might have a contributor hesitating and not sure about something. For example, if the requester is in this situation, he/she might be encouraged to open a draft pull request pinging some particular team that can give support before it's ready for review.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. Following on from this comment above, maybe we should have a "best practices for maintainers" section too. I'd rather see that in a followon PR so that this one can merge. :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tracking suggestion here #8


# Best Practices for Reviewers
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest adding a section on “nit-and-approve” as an encouraged practice,
calling out the trust considerations. (Or, I can draft such an addition
and submit my own PR, if you’d prefer?)

I have a couple of other best practices for proposers and reviewers in
my mental model, and will try to think about which ones are worth
surfacing in a document like this.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would love to have a section on nit-and-approve. Please draft the addition as a separate PR.

Copy link
Contributor

@wchargin wchargin Jan 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure: will do. (edit: #2)


## Be respectful and appreciative

As a reviewer, you should be respectful and appreciative towards the person
proposing the change. This is true especially when you quite disagree with the
change itself. It's fine to express strong beliefs that a change is misguided,
but you shouldn't make the contributor feel belittled.

## Limit nit-picking

Nit-picking is when a review focuses more on small, superficial details (e.g.
precise formatting or word choice) rather than on the substance of the change.
Nit-picking is a lot easier than substantive review, which can make it
tempting. It's ok to nit pick in moderation; when nit-picking, it's polite to
preface your suggestion with "nit", as in: "nit: this comma should actually be
a semicolon". Avoid holding up a review over nits.

## Avoid "not how I would have done it"

As a reviewer, you'll often notice that someone has made a change differently
from how you would have made it. It can be tempting to think that your way is
_the_ right way, and to ask the contributor to change their approach. Before
doing this, you should ask: in what ways is doing it my way materially better?
What benefits come from the changed approach? If you don't have clear answers
to these questions, don't ask for the change.