Package Quality Metrics

[daveverwer]

This will probably be the major push of the site post launch. Each package should have package quality metrics like the old CocoaPods site had.

Metrics should include things like:

• Does it have an open source license?
• Does it have a README of an appropriate length?
• How many releases have been made, and when was the last release?
• Does this project have tests?
• If the project has CI, and if so extract the:
  • Is it building?
  • What's the code coverage and are tests passing?
• If there are docs, where are they?

Please suggest additional metrics below.

[daveverwer]

Adding comment [from here] by @heckj:

To additional pieces of information I'd love to have:

• if it has CI, is it building (build status) - and a link to the information
• if there are tests, what's the code coverage and are they all passing?
• if there are docs, where to find them - how to use the library

I also integrated these requests into issue above.

[skyylex]

Hi @daveverwer,

I'd like to suggest few more things to touch in the metrics:

• community trust. It's difficult to measure trust, however there are correlations between popularity and trust. So it's possible to use number of stars on Github that shows that a lot of people somehow recognize a repo (package). In addition or as an alternative a number of forks on Github. In addition this could be checked by number of referencing packages from other repos and what is their rank.
• quality. It's even harder, but it could be implicitly checked by analyzing how an author & contributors work with problems that appear. More concretely, number of open / closed issues or bugs (ratio?), number of issues without comments, average time of reaction to the issues.

[daveverwer]

community trust. It's difficult to measure trust, however there are correlations between popularity and trust. So it's possible to use number of stars on Github that shows that a lot of people somehow recognize a repo (package). In addition or as an alternative a number of forks on Github. In addition this could be checked by number of referencing packages from other repos and what is their rank.

Absolutely 👍

quality. It's even harder, but it could be implicitly checked by analyzing how an author & contributors work with problems that appear. More concretely, number of open / closed issues or bugs (ratio?), number of issues without comments, average time of reaction to the issues.

Ratio of closed/open bugs is a very interesting metric. I like this 👍

Thank you!

[heckj]

@daveverwer while it's relatively easy to capture, and in a closed environment with consistent mechanisms can be a very useful number, the ratio can often be absurdly off - either intentionally, or as a side effect of automation to help with "management of the project". The specific thing that I'm aware of is a recent tendency to "close unactioned bugs", which ends up transforming the "bug list" from an action bug list to more of a "what's getting attention" metric - which often as not is completely independent of any form of quality.

I have found something like the time to get a pull request merged (and more specifically it's trend) to be a good indicator of health and coordination within a project, kind of a secondary indicator to actual project quality, but perhaps quite a bit more interesting to someone wanting to know "do these guys even react to PRs", which you might have if you want to use a library.

With the diversity of how people run projects, it's remarkably hard to know how to effectively measure quality.

[daveverwer]

Absolutely. I’ll be very careful adding anything like this. Thanks Joe.

_{Sent with GitHawk}