Feature pipeline dashboard

[brson]

This is something we've desperately wanted for a long time. Guiding features to stabilazition is the major concern of several of the Rust teams, and requires a major coordination. We have quite a rich and reasonbly well defined lifecycle for introducing new features to the platform, but nowhere a complete view of the entire pipeline.

There's lots of data though, and I think we can piece together most of the picture now, the full picture with incremental process improvements.

Features generally have names and can be in a number of stages:

• roadmap: topics on the roadmap, the team wants to talk about and consider for RFCs
• pre-RFC: pre-RFCs posted to irlo
• proposed: RFC PRs
• proposed-FCP: RFCs in final call
• accepted: These have RFCs in the repo, and may have tracking issues and in-tree metadata
• stable-FCP: Features are in final call
• stable: usually along with a version number

Each of these stages may have a date associated with it.

The data comes from a variety of sources: irlo, RFC issues, RFC PRs, RFCs, tracking issues, the in-tree featureck, crates.io.

So we need to scrape lots of stuff to try to associate feature names with pipeline stages and dates. Also likely want a short description, all the links, other goodies.

We might put them in a list, ordered by "progress", where each line contains a the feature name, description, most relevant link, and a bar indicating the progress as richly as the data allows.

As extensions we might also:

• indicate who is assigned to it (with avatar)
• indicate links/dislikes
• indicate scheduled release number
• indicate 'hot' features - features on the official roadmap that don't have accepted RFCs
• show lib/library/tools/etc distinction
• show the number of times the feature is used by crates.io

The data:

• RFCs/RFC PRs. Feature names and descriptions, PR numbers, tracking issue numbers, final-comment-period tag
• Tracking issues : B-RFC-approved tag, descriptions, final-comment-period tag. scape the link to the RFC and get the feature name there.
• rust-lang/rust tree: featureck collects information including the version stabilized, lang/lib distinction
• irlo pre-RFCs: Haven't thought about it. Not critical

[anp]

Off the top of my head, TODO:

• Database migration to add a "repository_fk" column to all github tables
• Parameter for instantiating GitHub scraper across multiple repositories
• Parameters for reporting methods to support specifiying repository
• Figure out how to scrape github reactions
• Clone RFC repo and scrape RFC text (some preprocessing on ingestion will be needing to make database entries machine readable -- can't do NLP/parsing on the fly every time someone hits an HTTP endpoint)
• Clone rust repo and scrape featureck (perhaps one of the libsyntax-based tools could be a starting point here?)
• Set up multiple-tab interface (in progress elsewhere)
• Create controller + template in ember for feature dashboard
• Scrape crates on crates.io and check for feature

[anp]

Boilerplate: I'm going to refocus this project on rfcbot, since that's the primary usage rusty-dash.com is seeing.