Add download counts to API

[Rexios80]

I know the download counts are experimental right now, but it would be good to track this for when the feature is stable

[Rexios80]

@isoos it looks like this feature is now stable. Do you have an ETA for implementing this?

[gbassisp]

Has this already replaced the popularity metric? My experimental flag doesn't seem to do anything anymore.

It's a shame to completely replace it, both are useful metrics.

[isoos]

@Rexios80: #8397 (or some similar, as name for it is pending) may expose it, with the disclaimer that we don't treat these as stable APIs (yet).

@gbassisp: yes, this is a replacement, and the experiment has been permanently enabled, the flag is ignored.

Personally I've liked the popularity metrics because it provided a way to compare packages that had very different scale of downloads, but it was always hard to explain what it is and why we prefer it.

Also note: the popularity metrics will be removed from the API in a release probably very soon.

[sigurdm]

It's a shame to completely replace it, both are useful metrics.

While there is some value in the old metric, it will be a hard to explain message why we have two different scores here.
Also we don't want to maintain both pipelines of data.

[Rexios80]

Is the popularity score calculation code public? If not could it be open sourced?

[isoos]

Is the popularity score calculation code public? If not could it be open sourced?

The calculation itself is public, but the underlying data is not. The input data is roughly the following: daily unique IP download counts, filtering out CI systems and bot traffic, aggregated over 60 days. Take that number for each package, order them, and the lowest gets 0.0, highest gets 1.0, and if total is N, every nth package gets n/N.

You could calculate it similarly using the download counts, but download counts are unfiltered, and over 30 days, so it wouldn't be the same at all.

[gbassisp]

Also we don't want to maintain both pipelines of data.

I completely understand that and, if we have to pick one, then download counts is better.

But from a package maintainer perspective, popularity is more motivating than count. It is exciting to see a package reaching the upper quartile, but it's frustrating to see it means only 1k downloads 😅

Still, great work on this feature! 🎉

[sigurdm]

Small nits:

daily unique IP download counts, filtering out CI systems and bot traffic, aggregated over 60 days.

It was not only downloads, but also version listings that were counted. Also we would score direct dependencies higher than transitive dependencies.

[Rexios80]

Thanks for the addition to the API! I added support for download counts to pubstats.dev, which you can see an example of here: https://pubstats.dev/packages/http

I now calculate my own popularity scores using the raw download counts which has resulted in some interesting switch-ups (see the get package). Of course you can still see the historical data for the old popularity scores, and they will continue to be collected as long as the pub.dev API provides them.