[EPIC] Add Peril automations for common tasks

[jlengstorf]

NOTE: This issue is an epic, which bundles up lots of related issues into a single goal. If you have the free ZenHub extension, you’ll be able to see overall progress and related issues, as well as a "board view" that shows this epic from a bird's-eye view.

Who will own this?

Role	Contributor(s)
AoR owner	@jlengstorf
Domain owner	@jlengstorf
Project manager	TBD
Tech lead	@jlengstorf
Contributors	@jlengstorf, @SD10, @piotrkwiecinski

Summary

We're currently using Peril to automate inviting contributors to the Gatsby org and letting them know about our free swag.

However, we could be doing a lot more with Peril.

For example, we could:

• Validate site showcase submissions (proposed in [perilbot] validate updates to the site showcase #6538)
• Validate other YAML docs (proposed in [perilbot] validate yaml for different yaml files + required fields #6634)
• Check bug submissions for required info, such as env info and a reproduction link
• Automatically label issues based on the issue template or other factors
• Post relevant docs and support channels on question/support issues

And that's just what I could come up with off the top of my head. I believe @m-allanson, @KyleAMathews, and @marisamorby all have additional ideas for ways to leverage Peril to make the OSS process more seamless, expedient, and helpful.

This proposal will also pull in automatable steps based on #7583 and create Peril rules to implement them.

How will this impact Gatsby?

Domains

• OSS process

Components

• Peril

Goals

What are the top 3 goals you want to accomplish with this epic? All goals should be specific, measurable, actionable, realistic, and timebound.

1. Eliminate chore and admin overhead for OSS maintainers
2. Improve consistency for issue and pull request handling
3. Increase discoverability of open issues for contributors

How will we know this epic is a success?

What changes must we see, or what must be created for us to know the project was a success. How will we know when the project is done? How will we measure success?

We'll know this epic is a success if contributors are:

• Receiving useful feedback on their issues and PRs that helps get the resolved/merged
• Able to find issues that they can take on
• Confident that the issues they're taking on are valuable, clearly defined, and not already in progress

In addition, the repo will:

• See fewer stale/outdated issues
• Decrease the time to response for issues and pull requests
• Increase the quality of submitted issues (via template enforcement, etc.)

Metrics to Measure Success

• Decreased response and resolution times for issues and pull requests
• Increased engagement from first-time contributors
• Decreased number of stale and/or abandoned issues

What are the risks to the epic?

In a few sentences, describe what high-level questions we still need to answer about the project. How could this go wrong? What are the trade-offs? Do we need to close a door to go through this one?

If we over-automate, we can make the repo to finicky, which would deter contributors. If we overcomplicate things, the additional hoops will prevent people from following processes at all.

What questions do we still need to answer, or what resources do we need?

Is there research to be done? Are there things we don’t know? Are there documents we need access to? Is there contact info we need? Add those questions as bullet points here.

We need the results of #7583 to identify key opportunities for automation.

How will we complete the epic?

What are the steps involved in taking this from idea through to reality?

1. Identify automations
2. Write the Peril rules/tasks and tests
3. Run them under close supervision for a week or so
4. Let ’em loose

How else could we accomplish the same goal?

Are there other ways to accomplish the goals you listed above? How else could we do the same thing?

The checklist that #7583 will produce also enables contributors to follow all the steps that the automations will follow. Many of these steps will be tedious (e.g. labeling, assigning) and likely to be skipped, which is why it's ideal to strike a balance between automation for repetetive, predictable tasks and human review for nuanced, empathetic support in issues and PRs.

Next Steps

• Under Pipeline select Proposed Epics (only if you are NOT the AoR owner)
• Under Assignees select the AoR Owner
• Under Labels select Epic
• Select Create Epic

[m-allanson]

Extending on your Post relevant docs and support channels on question/support issues, here's a great example discussing that: Use of bots - answers based on keywords

[calcsam]

Love this idea.

As more of the community is assisting with open-source issue management we should articulate a long-term vision of open-source issue management & code contribution.

Overview

Roughly speaking there are three things we can do to make issue & PR management better and easier:

1. Define issue processes & automate them when possible
2. Create dashboards to give visibility into issue flow, both overall and which issues are stuck in queue.
3. Prioritize / plan improving issue flow (SLAs, process documents, hiring, community rewards...)

There are also two core audiences to address:

A. Anyone part of the core Gatsby team -- whose responsibilities include responding to issues & PRs or creating them.
B. Contributor community.

Right now the low-hanging fruits // well-defined problems are 1A, 1B, 1C, and 2A.

2B and 2C are tricky because there are many ways to encourage the broader community of contributors. We’ll want to do this, but that’s the next step.

Project Stages

Defined Issue Journey (v1):

This looks something like:

(1) a well-defined state machine using tags, documented in the Contributing section
(2) Peril automation to move things from one stage to another

Maintainer Metrics (v2):

This looks something like:

(3) A dashboard on how long an issue stays at each stage, — granular tracking of time-to-response and time-to-resolution.
(4) SLA of how long an issue stays at each stage (for relevant stages)
(5) measurement & SLA for auxiliary metrics, such as: (1) new contributors per week (2) pull requests merged per week (3) community pull requests merged per week
(6) Figure out how available levers (# of core members, training, onboarding time) affect high-level metrics. Understand how high-level OSS metrics affect organizational value adds (code contribution, community members, etc).

Eventually....

Community Metrics AKA 2A and 2B, aka things we want to do when we feel like the problem is better defined, is a pretty big bucket that could include things like the following:

(7) probably a hierarchy of bug severity
(8) A contributor dashboard.

• It could include the following metrics, filtered by contributor:
  • Issues labeled as bugs resolved via PR
  • Issues labeled as features resolved via PR
• It should focus on this week / month and have some sort of exponential decay algorithm to highlight recent contributions, new contributors, etc.
  (9) A Top Feature Requests page. It can include all issues ranked by 👍 and whether they are currently being worked on
  (10) Levels of PR complexity assigned by the person who merges them (& enforced by automation)
  (11) Levels of community (if possible flair) based on PR complexity, etc
  (12) Incorporating community members of specific levels into MTTR responses

Other notes:

The best source of quantifiable open-source metrics I’ve been able to find is here:

https://github.com/chaoss/metrics

There are potential off-the-shelf options and analytics packages that claim to computer these numbers automatically, but they look incredibly difficult to use.

https://github.com/chaoss/grimoirelab-tutorial/
http://dev.augurlabs.io/?repo=gatsbyjs+gatsby

It would be really interesting if we were able to put this together as a series of peril scripts & a Gatsby site.

Finally, some notes on what’s useful or helpful here:

https://docs.google.com/document/d/1tgDGWLJ3dazHeSw3QcpK5khlNsa8IJdMRaTss--7Qj0/edit#heading=h.imxmj9ieyytp

[jlengstorf]

Enforce conventional commits via Peril. https://conventionalcommits.org/

[shannonbux]

Wanted to capture a conversation from this morning with @m-allanson and @pieh about creating automatic notifications when someone edits the docs --> notify me and a team of relevant people. Might make sense to assign teams to certain files, e.g. if someone makes a PR that changes the blog, notify a certain team, if someone makes a PR that changes gatsby config file, notify a certain team.

Might not be the only solution. The overall problem to solve is just to get the right people involved in reviewing PRs, which can't be labelled. We could also require that a PR is always connected to an issue, since issues can be labelled.

[m-allanson]

GitHub has some built in functionality for this sort of thing, see https://help.github.com/articles/about-codeowners/

[jlengstorf]

This is a good idea, @shannonbux. Like @m-allanson said, I think we don't need a Peril automation for this, but I think we could look into GitHub Teams + code owners to manage it really well. Peril could theoretically mention the relevant people and/or assign the PR to them if the teams/code owners isn't enough, though.

[marisamorby]

Agree @jlengstorf, @shannonbux, and @m-allanson that looking into GitHub Teams is a great idea. We want the lowest level of effort for the highest impact, so setting up teams so that people understand who owns what will probably help a lot.

What else do we need before we can create GitHub teams and try it out?

[jlengstorf]

@marisamorby there might be more that's needed, but from where I'm sitting:

1. We need to identify what teams should exist (e.g. docs, core, maybe plugin-specific teams)
2. We need to identify who should get notified as part of those teams
3. We need to make those teams the code owners for the relevant sections of code

Step 1 I think should be reasonably straightforward in an MVP sense. Step 2 is a little harder, because ideally code owners aren't only Inkteam members, so we need to do a little digging to decide who has the most context for a given section of code and ask if they'd like to be a code owner.

We should also probably figure out how someone goes from unaffiliated > contributor > maintainer > code owner. We should maybe tackle that as part of #7583.

Anything I'm missing, @m-allanson and @shannonbux?

[marisamorby]

@jlengstorf That line of reasoning makes sense to me. I'm putting together some of the OSS pain points that I took note of and so that might help us identify who should get notified on these teams by identifying where the gaps are.

[marisamorby]

@jlengstorf: Should this be labeled under Community instead of Open Source?

[m-allanson]

Another idea for Peril thanks to @papandreou at gatsbyjs/rfcs#12 (comment) - detect (and correct?) common mis-spellings of Gatsby. Gastby and Gatbsy are the usual culprits.

[gatsbot]

Old issues will be closed after 30 days of inactivity. This issue has been quiet for 20 days and is being marked as stale. Reply here or add the label "not stale" to keep this issue open!

[gatsbot]

This issue is being closed due to inactivity. Is this a mistake? Please re-open this issue or create a new issue.