Reassessing our Metrics of Popularity

[PeterShaggyNoble]

Since my return, I've been primarily focussing on triaging old & stale issues and one thing that has become increasingly clear, and increasingly urgent that we address, while doing so is that GitHub stars are actually a terrible metric of popularity! I've come across numerous requests where, at the time, the traffic rank for the website wasn't too far outside our 500k cut-off and had over 5k stars but, since then, the rank has plummeted and/or the GitHub stars haven't increased significantly over time and/or development on the project looks to have slowed or stalled completely. So, I'm proposing that we throw stars out the window completely as a measure of popularity and introduce some other metrics & cut-off points instead.

Rather than go into one of my usual, long-winded & overly wordy proposals this time(!), I'll just bullet point my suggestions for now and we can discuss them further in the comments. All cut-offs below are merely my suggestions and open to discussion - in most cases I'm just using our own data and rounding it up or down as a starting point.

With this, I am also proposing the creation of a label called "assessing" that we can add to issues or PRs while assessing their popularity. Unlike "in discussion", it would highlight to people browsing our repo that we are looking for someone to attest to popularity. I would also suggest that that label be added to all open issues & PRs where the only current qualifying metric is GitHub stars, while we work through and implement these changes.

1. The brand is in existence for at least one year.
   • Where applicable, one year will be measured from the date of the first stable release, rather than the project start date.
2. Its website has a Similarweb global rank of 500k or higher
   • As Similarweb only update their data once a month, there will be a monitoring window for website's ranked between 450k & 550k until the next update, unless the brand is within scope on any other metric.
   • For existing icons in our library, the threshold is dropped from 500k to 750k.
   • A rank of 2m or lower will result in the brand being declared outside our scope, regardless of any other metrics, but may be appealed.
3. The website's Similarweb rank in any one country is either:
   • In the top 100, or,
   • 50k or higher, with a global rank of 1m or higher,
4. The website's Similarweb rank in any one category is either:
   • In the top 100, or,
   • 10k or higher, with a global rank of 1m or higher,
5. Its packages meet one of the following minimum requirements:
   • npm: 100k weekly downloads
   • JSDelivr: 1m daily or 35m monthly requests
   • crates.io: 100k weekly downloads
   • pypistats.org: 100k weekly downloads
   • Packagist: 200 average daily installs in a month
6. The brand's popularity can be illustrated by other publicly available & verifiable statistic (e.g., downloads, usage)
   • Stats should preferably also include data on one of our existing brands so we can make a direct comparison
7. The brand's popularity can be illustrated through a Google Trends comparison
   • Must be with a similar brand that is already in our library, or that would qualify under any metric.
   • Must be unambiguous (i.e., it's not a suitable metric for brands with generic words for names).
   • Trending equal to or higher than the compared brand will be considered in scope.
   • Trending lower than the compared brand will require the consensus of the person providing the comparison and at least 2 project maintainers
8. Where applicable, the primary repository for the brand's GitHub project meets the following requirements:
   • A minimum of 5k GitHub stars will be required for consideration, and,
   • An additional 1k will be required for each year the project is in existence beyond its initial year, providing,
   • The repository is still active, and,
   • its star history is on a consistenly upward trajectory.
9. If all else fails, though, feel free to make a good case for the popularity of the brand you're requesting on any other grounds, provided it can be backed up with verifiable data.
   • Example: a car manufacturer's own website falls outside our scope but a major dealership dealing exclusively or primarily in that brand falls within our scope - in that case we'd accept the manufacturer as being popular.
   • If you can provide a particularly good metric that can be applied to other brands then it will be added to this list.

[service-paradis]

Some minor comments I have with the above proposal:

• We need to be cautious with with npm/cdn stats since they are easily manipulable and may not be reliable. We rejected an icon in the past even if the weekly download looks good (Add rovel.js logo #4519).
• If we use Google Trends comparison, the comparison should be made with a similar brand.
• Github stats: this is really difficult to evaluate the current popularity of a project... We may probably use the star history to see if there is a downward trend, but it is quite subjective. We may use repository activities to make sure the project is still active and maintained, but some older stable packages might slip through this...

[PeterShaggyNoble]

• We need to be cautious with with npm/cdn stats since they are easily manipulable and may not be reliable. We rejected an icon in the past even if the weekly download looks good (Add rovel.js logo #4519).

Agreed 👍 I had given that some thought already and it can partly be taken care of by only considering brands that have been existence for at least a year - it's usually newer projects that have their stats gamed like that in order to boost their visibility. We should also, though, look at historical data to ensure the current data doesn't appear in any way anomalous.

• If we use Google Trends comparison, the comparison should be made with a similar brand.

An oversight on my part, fixed now.

• Github stats: this is really difficult to evaluate the current popularity of a project... We may probably use the star history to see if there is a downward trend, but it is quite subjective. We may use repository activities to make sure the project is still active and maintained, but some older stable packages might slip through this...

Yeah, this is going to be the trickiest one; we may need to look at a combination of factors in each case to make an accurate assessment.

[PeterShaggyNoble]

Updated the proposal to reword it in preparation to be added to our contributing guidelines, and made the following changes:

• Anything within the top 100 in a country or category will be deemed within scope
• I removed GitHub stats/activity/etc. until we can come concrete way of using that data to assess popularity
• I removed the point about allowing brands that are being used by other major brands as that's just inviting people to temporarily add the big boys' logos to their client list; that can now fall under point 9
• I added GH stars back in as the last qualifying metric, at least until we can come up with a better way of assessing popularity through GH. But I added an additional requirement to our current 5k base cut-off

I'd still like to see some discussion around the cut-off numbers before we proceed, particularly with npm, JSDelivr & Packagist, as I kinda just pulled those out of the air based on our own current data.

Also, once SW get around to adding global rankings for apps, I think we should add that in immediately after website rankings.

[PeterShaggyNoble]

Added crates.io to bullet point 5, with an intial suggestion of 100k weekly downloads.