Team nagboard

[brson]

This is to help the teams manage their workflows.

Each team member (rust-lang/Core, compiler, Lang, Libs, tools) gets their own page, like rusty-dash.com/nag/brson. This page shows issues/PRs that require a response.

It supports two types of 'nags':

First, issues written by a team member, with comments containing f? @user @team etc., require feedback. When these comments are created they go onto the mentioned users' nag board. When subsequently the user leaves any comment, the nag is satisfied and it leaves the nag board.

Second, issues written by a team member, containing both the final-comment-period and T-* tag, require a vote. When these comments are created they go onto the nag board of the team members associated with the T-* tag. That user must subsequently leave a comment containing either fcp r+, fcp r- or fcp abstain, after which the nag is cleared.

@dikaiosune This one is higher priority than the other new dashboards we've discussed because it will allow for dev process changes.

ISTM that new nags should also trigger some kind of notification, not just the passive addition to the nag board, but that may be the role of another bot.

cc @nikomatsakis @nrc

[anp]

Sounds great!

1. Is there a machine-readable location of the team hierarchy? It'd be great if every once in a while I could just scrape a source instead of including a configuration file that has to be in sync with Rust governance processes. Could also be useful for other things (maybe?) if it doesn't already exist.
2. Re: notifications -- I think if there's going to be a bot which tallies votes, that'd be a logical place to host notifications. (edit: also, setting up an MTA is a pain and integrating with 3rd party email services is also a pain)

[anp]

@brson Are there any existing examples of these comments from team members? It'd help to have some actual GitHub comments to use for testing as I build this.

[nikomatsakis]

cc @aturon

[brson]

Is there a machine-readable location of the team hierarchy? It'd be great if every once in a while I could just scrape a source instead of including a configuration file that has to be in sync with Rust governance processes. Could also be useful for other things (maybe?) if it doesn't already exist.

I mentioned this on IRC but it's at https://raw.githubusercontent.com/rust-lang/rust-www/master/team.md

@brson Are there any existing examples of these comments from team members? It'd help to have some actual GitHub comments to use for testing as I build this.

No, the f? comments are a new scheme, but syntactically they are the same as the r? comments we already use on pull requests.

[brson]

If you need to test this stuff we can open demo issues and people can leave comments.

[nikomatsakis]

Note that I had not envisioned the bot tallying "votes" -- just tracking who has given feedback and who hasn't. (I don't even think of these as votes per se, since we operate on a consensus plan typically -- more just tracking who has given their opinion and is satisfied with status quo).

[anp]

@nikomatsakis Sure thing. So are the fcp r+ comments not meant for voting? And should I also include in this feature a summary of recent fcp-based nags?

If there's no voting or tallying needed beyond a simple total on the dashboard, I can look at implementing notifications as simply as possible. Email?

[anp]

@brson Some demo issues w/ comments would be fantastic. In my dev database I've got a bunch of issues and comments built, and it's a pretty simple set of criteria for display, but I always prefer to test on "real" data before deploying.

[brson]

@dikaiosune I created some test issues and pinged some people to fill in some comments

[anp]

Great! Hopefully with a quick look at some test data I'll be able to
commit/deploy a first pass tonight.

Adam

On Wed, Jun 22, 2016 at 1:59 PM, Brian Anderson notifications@github.com
wrote:

@dikaiosune https://github.com/dikaiosune I created some test issues
and pinged some people to fill in some comments

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#61 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AGfyeXyNeqOxKgr1n-Ukh8spNJptfy0Pks5qOaIxgaJpZM4I4fSg
.

[anp]

Just a heads up that I've been holding on this since @aturon pinged me on #rust-internals on Thursday:

3:29 PM <•aturon> dikaiosune: hi there -- i've been trying to reply on the issue about the "nagboard" but have been crazy busy -- just wanted to say that i think there's a bit more design the teams need to do around exactly how the process should work before we're ready for implementation
3:29 PM <•aturon> dikaiosune: i don't want you to sink too much time into it only to have the requirements change from underneath you

[anp]

@brson @aturon I've managed to carve out some time to work on this, so in the next few days I'll be implementing the behavior specified at https://internals.rust-lang.org/t/refining-rfcs-part-3-async-decisions/3658/33. If you're feeling antsy (I know I am), you can follow along in the rfcbot branch of this repo. I'd like to get something minimal online ASAP.

(oh, I know @ubsan was also very interested about the progress of this)

[anp]

@brson @aturon @nikomatsakis

I've got most of the backend code together for the bot (although not nearly as tested as I'd like). Last piece for the actual bot is to have it comment on the issues to notify users of the current status of issues. I'm not quite sure how chatty it should be -- but I figure I can start out verbose and scale it back until only information which is useful in practice is communicated.

Next up are some JSON endpoints to fetch the current state for a user or an issue/RFC, and then to render it on the front-end.

[brson]

Thanks @dikaiosune!

[anp]

@brson @aturon @nikomatsakis

Still not 100% on all the bot interactions, but it's a start: anp/rfcbottest#1

More testing to do re: reviews & feedback requests, and then I need to add followup notifications and "everyone's reviewed" notifications.

[nikomatsakis]

@dikaiosune very cool!

One nit: when listing the people from whom reviews are requested, could we list them with GH checkboxes? i.e., - [ ]...

[anp]

@nikomatsakis I don't see why not. Are you suggesting that reviewers check the box to indicate they've reviewed? Or that the bot edit the comment to indicate they've reviewed once they leave a "reviewed" comment per https://internals.rust-lang.org/t/refining-rfcs-part-3-async-decisions/3658/33 ?

[aturon]

@dikaiosune This looks fantastic! 🏆

re: reviewing, I think just having people check off their box would work fine as well.

One other issue: rather than the bot posting responses, is it possible for the bot to edit its initial FCP comment? For example, active concerns raised by team members could be linked under their name, and the links removed (or struck out) when the concern is resolved?

To keep noise to a minimum, I think the bot should generally post roughly four kinds of comments:

• The initial FCP comment, when a move to FCP is made
• A comment saying "All team members have reviewed and no objections remain" (basically a ping to actually go into FCP)
• A comment saying "A week has passed since FCP"
• Possibly a comment for FCP being canceled

I think all other activity can take the form of updates to the FCP comment, which acts as a kind of status.

[nikomatsakis]

@dikaiosune ah, hmm, I was hoping people could just check the box in lieu of a comment, but I forgot that we had said everyone would leave a comment. If that's too hard though a comment is fine. Is it possible (hard?) for the bot to check-off a check-box?

Ah, I see @aturon posted in the meantime -- like him, I'd like the bots "motion to do X" comment to serve as a kind of quick status board where you can see what is current state of play.

[anp]

I think just having people check off their box would work fine as well.

I don't believe GitHub's API tells the client who edited a comment -- would it be OK for any person with edit access to the issue/comment/repo be able to mark any reviewer as "reviewed"? My gut reaction is that reviews should be restricted to the person being marked as having reviewed the issue. But perhaps that isn't as important as I'm thinking.

If that's too hard though a comment is fine. Is it possible (hard?) for the bot to check-off a check-box?

It's not too hard to register checking the checkboxes, as AFAIK it's just an edit to the comment markdown which is easily parsed. However it's still not possible to authenticate those edits beyond "any person who has edit permissions to the issue comments."

is it possible for the bot to edit its initial FCP comment?

Is it possible (hard?) for the bot to check-off a check-box?

Definitely possible, it's just an edit to the existing comment.

Note to self: I need to come up with a solution for when command comments are edited or deleted in a way which changes the command.

[aturon]

Note to self: I need to come up with a solution for when command comments are edited or deleted in a way which changes the command.

That'd make your bot smarter than @bors! :)

[nikomatsakis]

@dikaiosune I am not worried about people checking names that are not their own.

[anp]

@nikomatsakis Cool. Thinking about it now, the checkbox solution also provides a convenient escape hatch for the vacation/extended absence problem without having to build a whole interface for registering those absences.

[nikomatsakis]

@dikaiosune ah that's a good point :)

[anp]

@aturon @nikomatsakis

Alright, after modifications to support tracking the overall status in a single comment, here's the most recent test I did:

anp/rfcbottest#2

I don't yet have a good solution to edited command comments, except to ignore them if they're already in the database. I suspect that for now that's actually a fairly low-impact thing to fix.

I haven't yet tested the "one week after FCP starts, post another comment," but it's pretty straightforward code so hopefully won't be an issue.

This is running locally on my machine, but if everyone thinks that the level of noise and comment format(s) are OK, I'll do some cleanup, put it live, and point it at all the Rust repos.

EDIT: Note that I haven't built the web interface to this yet, just in case the data model ended up needing to change.

[aturon]

@dikaiosune 💯 this pretty much made my day. Ship it!

[aturon]

@dikaiosune Looks like it went through now, interesting...

[aturon]

@dikaiosune Any chance the concern list can link to the comment raising the concern?

I noticed that you can only have one concern per comment, but I think that's a good thing -- just something people need to be aware of.

[anp]

@aturon Definitely.

There are several things people need to be aware of that have yet to be documented :). BTW, where do you think a good location for those docs would be? I could stick it in a markdown doc in the repo, but that seems maybe a bit out of the way.

[aturon]

@dikaiosune Seems like a great use for The Forge.

[anp]

I shall fetch my hammer!

[anp]

@aturon @nikomatsakis

OK, the barebones web view is live:

http://rusty-dash.com/fcp/

As of right now, it only shows review requests that haven't been completed yet, so they disappear once the review has been checked off on the rfcbot comment.

Also, @brson if you want to register those concerns on rust-lang/rfcs#1636 separately you may want to edit that original comment to only be one concern and submit the others as separate comments. Unless we want to support multiple concerns per comment?

[anp]

@aturon The concern list now links to the initiating comment, rather than pinging the author: rust-lang/rfcs#1636 (comment).

I will try to find some time for docs in the next little bit, although the next 2 weeks are pretty bonkers for me.

[anp]

@aturon @brson PR in to the forge with docs for rfcbot: rust-lang/rust-forge#15

I've written a quick document here on the commands it accepts and the current constraints on what it'll do.

[nrc]

@dikaiosune re notifications - I think it would be useful and important to get notification set up. Posting notifications to irc is really easy and low cost (to #rust-bots, so as not to pollute other channels), although I'm not sure exactly how valuable it is.

Email would be better, but I sympathise with the pain of getting it working. For triage bot I ended up sending via gmail using nodemailer.

[anp]

@nrc is github not sending notifications from the initial status update comments? Or are you referring to other intermediate items (like so-and-so has checked a box or registered a concern)?

[nrc]

@dikaiosune it probably is, but I get so many GH notifications that they get lost - I have no way to mark pings as special, afaik (I could just get notifications for pings, I guess, but that is still a lot of GH mail).

[aturon]

@dikaiosune Hey there! We've been loving the bot!

Quick question/request: when the bot posts comments like rust-lang/rfcs#1682 (comment), AFAICT it's "going into FCP" in its own head (in that, according to the docs, it'll post a follow-up comment one week later). One problem is, it's not actually flagging the RFC as final-comment-period, and the comment it leaves is a little unclear.

After discussion with @nikomatsakis, we feel comfortable with the bot fully triggering FCP, given our experience so far. That would mean:

• Tagging the issue as final-comment-period
• Posting a very clear comment saying: "This RFC is now entering its final comment period, as per the review above", with a link back to the original checklist.

A couple other details:

• The bot should post another comment, "The final comment period is now complete." after one "business week" has passed since going into FCP. Actually merging the RFC requires human intervention, since we want to review intervening comments.
• Is it possible to set a different FCP duration for tracking issues on the rust repo? There, FCP lasts roughly for a release cycle -- we'd want the comment noting the end of FCP to come about 1.5 weeks prior to the release.

[aturon]

Bug report: the bot doesn't seem to've caught the request here

[aturon]

Nevermind the last bug report -- just had a very long turnaround time.

[aturon]

Feature request: the person who moves to FCP should automatically have their checkbox ticked.

I often find I forget to check my own box.

[aturon]

Request: I'd like to change the text on the initial comment. Today it says:

FCP proposed with disposition to merge. Review requested from:

To make this more clear, can we say:

Team member {user} has proposed merging the RFC. The next step is review by the rest of the {team-names} team:

• {reviewer list}

Once these reviewers reach consensus, the RFC will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

[aturon]

Feature request: global dashboard.

In particular, it'd be great for http://rusty-dash.com/fcp to list each subteam with its open FCP proposals underneath -- ideally, with a "days since proposed" counter which acts as the sort order. That will make it easier for teams to keep on top of stragglers.

Conversely, the list of team members is not so important to have here.

[nikomatsakis]

Feature request: global dashboard.

What I would like is a list of the FCPs for each team, along with a matrix showing who has yet to respond, so I can nag them. =)

[alexcrichton]

Bug report: It looks like @rfcbot reviewed wasn't caught here?

[nrc]

Feature request: work with https://github.com/rust-lang-nursery/fmt-rfcs I think all that would need would be using the style team for all PRs, rather than looking for a T- label. (Which is probably useful functionality to have for other repos too).

[anp]

So exciting! I'm really glad to see the amount of use the bot is getting. There are a few more requests/bugs than I anticipated landing in this thread when I suggested to continue using it, so I'm going to break these out into separate issues under the rfcbot label, and close this since I'm now counting 7 items up for discussion/investigation here :).

The new issues should show up as "referencing" this one right underneath my comment.