Show GitHub metrics within pub.dev

[jayoung-lee]

When evaluating a package that needs regular maintenance, users visit GitHub to check the package’s recent activities and maintenance status. We can consider showing key GitHub metrics within pub.dev to save users’ trip to GitHub.

Key metrics that users mentioned in a recent user study were:

• Last commit date/content
• Number of open issues
• Number of open PRs

[YazeedAlKhalaf]

That's a really great and useful idea!
I believe we could use the GitHub API to do this. npm already does this and even VSCode marketplace.

[elbeicktalat]

Very nice idea in this way developers safe time, also having this data will make easier choices between the similar package.

[stuartmorgan-g]

One complicating factor here is packages from monorepos (flutter/plugins and flutter/packages being prominent examples, but certainly not the only ones); repo-level metrics will be misleading for per-package evaluation.

[jayoung-lee]

@stuartmorgan That's a good point. I think we shouldn't display the metrics when there's no 1:1 match. Do you know how many packages are from monorepos?

[stuartmorgan-g]

I don't, although we could probably cobble together a script to make an educated guess based on crawling pub.dev links. The problem with not showing anything would be that it would likely be confusing to have a lot of the most popular packages (from a quick manual check, it looks like about 50% of Flutter Favorites, for instance) not have the quality metrics.

[leighajarett]

+1 This came up during research sessions for Flutter Vikings. Developers were also interested in seeing security vulnerabilities broken out as their own category if possible (perhaps through the use of Github tags), and # of tests.

We also talked to developers about being too intertwined with Github, but everyone we spoke with was happy to only use Github for hosting their packages as opposed to other git providers.

@stuartmorgan in the case of monorepos: could we also try to look only for commits in directories that match the package name, and issue tags that match as well (or contain the name so it would catch our p: )? I know there would be some errors here, but maybe we just make that clear to both package consumers and developers so they're aware

[stuartmorgan-g]

could we also try to look only for commits in directories that match the package name

We'd want to spot-check some of the other monorepos, but that could likely work. (There are some subtle questions to resolve first about what exactly we even want to count when it comes to federated plugins.)

issue tags that match as well

Maybe? I just checked the Plus Plugins repo, and they don't appear to be labeling consistently.

I actually have significant reservations about issue counts as a metric we actively promote though. In practice, issue counts are measures of five things:

1. popularity - Flutter has 11k open issues not because it's orders of magnitude less well-maintained than most packages, but because orders of magnitude more people are using it and filing issues. Obviously that's an extreme example, but the principle applies within the package ecosystem.
2. size/scope - Same argument here.
3. age - (maybe; see next item) Flutter has more than 2x the issues it had when I started working on it; that's not because it's become more poorly maintained, we just get more reports than we can fix (which is common, and not inherently problematic) so accumulation is a function of time.
4. issue handling policy - I'm aware of some packages that run a bot in their repo that closes every bug that hasn't been updated in several weeks. That doesn't make them better maintained than packages with a policy of leaving all unresolved bugs or feature requests open (like Flutter mostly does), but changes issue counts dramatically.
5. bug reports and feature requests - there's no reliable way to distinguish between them.

So of five things that have major influence on issue count, only one (really, only part of one) is really any kind of indication of package quality/maintenance quality. There's so much noise that at best I think the number is meaningless. At worst, we would incentivize behavior that I would argue is undesirable, like running bots that close almost everything (which results in things like 20 one-off closed issues about the same thing, rather than one issue that collects discussion, votes, etc.)

[sigurdm]

I agree it is hard to tell anything for certain from many of these numbers, and many of them can be gamed to some extent.

Still many people use the github link to estimate how well-maintained a package is. So we should be able to extract something useful if we're careful.

• Maybe rate of commits (how many commits per week during the last year) could be a useful indication of project liveliness. github has an api for getting recent commit activity,
• Github stars: indicating how many people are interested.

[stuartmorgan-g]

If we are willing to do not-trivial queries, an interesting metric could be something based on the percentage of recent issue that a project member has interacted with in some way (comment, label, etc.) That would give some indication of whether someone is at least looking at issues.

[mhamri]

bringing from dart-lang/pana#1472

The scoring system doesn't adequately account for package maintenance status over time. As illustrated in the example above, the top-ranked packages are often outdated, despite their high scores. This creates a misleading impression of quality and reliability. Numerous developers have attempted to fix and fork the same outdated packages, leading to a scattered landscape of similar solutions (3rd option is another random fork). It's unclear which forks are actively maintained, have better functionality, or respond to issues promptly

Finding a reliable package now requires extensive manual investigation, checking each repository individually to assess:

• Active maintenance status
• Issue resolution frequency
• Pull request merge activity
• Developer contribution patterns

The scoring system should evolve to better serve the developer community by:

• Package scores should decline/decay over time without updates
• Recent activity should be heavily weighted in scoring algorithms
• While packages may have been excellent in the past, their current relevance should be the primary factor

Add metrics

• Include metrics for issue response time, pull request frequency, and contributor activity and contributor counts
• Clear indicators of whether a package is actively maintained
• Metrics showing community engagement and support (active PR, active issue, recent issue, recent PR, different contributors count in a time range, ...)
• Support for latest gradle (which always is a pain) or Java version or minimum NDK or minimum SDK that is enforced by google and apple yearly

[stuartmorgan-g]

• Package scores should decline/decay over time without updates

This would be actively harmful for some packages. For instance, Dart math libraries can be fine for a very long time without updates, because they often don't have dependencies and math itself doesn't change. Similarly, mustache_template hasn't been updated in 4 years, and it's still actively used in the current version of the flutter tool with no issues.

It is easy to describe general heuristics that would apply well to specific examples, but don't generalize well.

[sigurdm]

Yeah, I agree that we should not conflate liveness with the pana scoring.

But presenting liveness somehow could still be useful.