Define different threshold for removing a brand

[fbernhart]

As suggested in #3465, it would be helpful in my opinion to define a different threshold for removing a brand than for adding it.

Problem

As we've probably all seen in the past, the Alexa rank isn't a very "stable" measurement of popularity sometimes. Once in a while it happens that we're adding a brand that is within scope, just to see it's Alexa rank dropping in the next few months or even weeks to out of scope.

Example: #3465

• Alexa rank when the PR was created and merged in August 2020: 219,404
• Alexa rank now (January 2021): 683,075

So according to our contribution guidelines right now, we would already have to remove it - throwing all the work that has been done 5 months ago away.

And probably as soon as we've removed it, the Alexa rank will climb up again to 450k and someone will request the same icon again.

Suggestion

I'd suggest defining a different threshold for removing a brand. Something like:

• Alexa rank of 500k and below in order to be added to Simple Icons
• If the icon is already part of Simple Icons and if it ever drops beyond an Alexa rank of 1 million, we'll remove it.
  This would give us a bit of "free space".

1 million is just a number here. I'm open to discuss different thresholds for removing a brand. 750k? 1.5 million?

@simple-icons/maintainers any opinions or other ideas how to solve this problem?

[adamrusted]

I think it may be worth mentioning #4725 here - as the icon was proposed, and initial work put into it, before the Alexa rank plummeted. Should this fall into the same rank as 'icons already added', as similar amounts of work have gone into constructing the icon for the collection?

[PeterShaggyNoble]

Here were my thoughts on it over the weekend, all assuming that the Alexa rank is the only thing we have to go on.

1. A brand dipping below 500k should not be grounds for immediate consideration for removal.
2. Only when it reaches a threshold of 600k* should it be considered for removal and an issue opened for it.
3. If, after 30* days, there is no sign of recovery (rank is stable or still declining) then a PR should be opened for its removal in the next major release.
4. If, after those initial 30* days, its rank is instead increasing but not yet at 500k then we give it another 30* days before reviewing it on both points 3 and 4.
5. On the week of the major release, the rank should be reviewed with points 3 and 4 in mind prior to merging the PR.

On the subject of brands that have yet to be added but have open PRs then I suggest we add the "pending" label to them and review their rank at the end of the 30* day grace period to see if they've hit 500k. If not then we reject the PR as "out of scope" as it's easier to add the brand if it does recover in the future than it is to remove it if it continues to show no signs of recovery.

In addition to that, there will be occasions where we'll need to make judgement calls such as a site's reduced ranking being as a result of a recent domain name change or, as I suspect is the case with the Hack Club example mentioned by @fbernhart, simply a seasonal fluctuation or otherwise as a result of anomalous and extenuating circumstances.

*Both the 600k threshold and 30 days grace are merely suggestions at this stage and will need to be refined.

[fbernhart]

1. A brand dipping below 500k should not be grounds for immediate consideration for removal.
2. Only when it reaches a threshold of 600k* should it be considered for removal and an issue opened for it.
3. If, after 30* days, there is no sign of recovery (rank is stable or still declining) then a PR should be opened for its removal in the next major release.
4. If, after those initial 30* days, its rank is instead increasing but not yet at 500k then we give it another 30* days before reviewing it on both points 3 and 4.
5. On the week of the major release, the rank should be reviewed with points 3 and 4 in mind prior to merging the PR.

In my opinion this seems to be too complex and too much work, as it would require reviewing and checking in on the Alexa rank on a regular basis.

I would instead go for a bigger threshold (1 million sounds reasonable to me) and just remove it, once it drops beyond this line.

simply a seasonal fluctuation or otherwise as a result of anomalous and extenuating circumstances.

This would as well be covered by a bigger threshold.

[ericcornelissen]

I think @PeterShaggyNoble's proposal can be simplified to something along the lines of (sticking with the Alexa rank as popularity measure): If a brand's Alexa rank drops below [INSERT EXACT NUMBER HERE], the brand can/should be marked for removal by opening a Pull Request where the brand is removed. At the latest with the next major release the Alexa rank is checked again, if a) the Alexa rank is too low it is removed and if b) the Alexa rank has (mostly) recovered it is kept. I hope this adjustment reduces the amount of work to mostly just verifying the Alexa rank before a major release. Additionally, the Pull Request that removes the brand may prompt the community to check in on the Alexa rank for us. Of course, as @PeterShaggyNoble pointed out, there are some edge cases including that the brand popularity may have declined to close to the major release. In this case I think we can just leave the Pull Request open for that major release and check in later.

On the subject of brands that have yet to be added but have open PRs then I suggest we add the "pending" label to them and review their rank at the end of the 30* day grace period to see if they've hit 500k. If not then we reject the PR as "out of scope" as it's easier to add the brand if it does recover in the future than it is to remove it if it continues to show no signs of recovery.

I agree with this. But it is probably important to communicate this clearly with the community as it can be demotivating for contributors.

[PeterShaggyNoble]

Thinking on this some more over the weekend in light of #4759, I think we should hold off on removing any icons based on decreased popularity until the world gets back to normal. There are a lot of brands and companies taking a hit these past few months and, while the reasons for some, like the IRFU, may be obvious to us, others may be less so. Of course, we'd hope that, by the time v5 comes around in May, things should be back to normal in a lot of countries.

On the subject of criteria for removal, I like @ericcornelissen suggestion of combining both @fbernhart's and my suggestions - I'm coming at it from the angle of us endeavouring to find any possible reason not to remove a brand that has already been added but I agree that there are going to be cases where that's not possible. As to the threshold a rank should fall to before opening a PR, I'd like to initially suggest 800k. It's only a personal thing but, when considering icons for addition, that's around the point where my kneejerk reaction usually goes from trying to find an alternative metric to wanting to reject it outright.

[atao60]

According to What’s going on with my Alexa Rank?, “every day, Alexa estimates the average daily visitors and pageviews to every site over the past 3 months. ” Using the duration of this sliding sample window as grace period for each of steps 3 and 4 as specified by @PeterShaggyNoble would seem sound to me.

As stated by again What’s going on with my Alexa Rank?, above 100 k “a very small change in the number of visitors to a site on the long tail results in a large change in [Alexa] rank. ”. Therefore an hysteresis with thresholds at 500 k (in) and 1 M or even 1.5 (out) as proposed by @fbernhart would avoid to many back and forths.

[adamrusted]

If a brand's Alexa rank drops below [INSERT EXACT NUMBER HERE], the brand can/should be marked for removal by opening a Pull Request where the brand is removed. At the latest with the next major release the Alexa rank is checked again, if a) the Alexa rank is too low it is removed and if b) the Alexa rank has (mostly) recovered it is kept.

I agree with the wording of this, and think all we need to decide now is the threshold at which we remove icons. I would be for adding another 50-100k to the Alexa rank, so if it's rank drops below 550/600k, then it's considered for removal. What do others think?

[ericcornelissen]

I think we should just pick some threshold and see how it works, it can always be adjusted later if it is found to be insufficient. Based on earlier comments in this thread it seems 50-100k is on the low end. To err on the safe side I would follow @PeterShaggyNoble suggestion and suggest somewhere between 200k-300k instead.

[adamrusted]

I'd be happy to go with 300k, given that I'm in a similar position to @PeterShaggyNoble, in that 800k is roughly where I go "Well that definitely can't stay!".

[PeterShaggyNoble]

suggest somewhere between 200k-300k instead

Split the difference and say 750k? Which would be a nice, round allowance of 50% over what we require for inclusion in the first place?

[PeterShaggyNoble]

Should we include a note in the contributing guidelines about this in an effort to avoid people questioning how 700k ranked Brand X is within our scope but 501k ranked Brand Y isn't? Something along the lines of:

* Is of a _popular_ brand:
    - For websites, the [Alexa rank](https://www.alexa.com/siteinfo) should be less than 500k.
        - Note that for brands that have already been added the threshold for continued inclusion rises to 750k.
    - For GitHub projects, the amount of "stars" should be above 5k.
    - For anything else, popularity will be judged on a case-by-case basis.