clean up npm downloads data for graphs

[patak-dev]

removing outliers

See for example npmx.dev/package/vite

Probably a rogue CI created the spike you see here in the graph. This isn't significant information, and destroys the scale of the graph.

We should do a clean step so we can remove these kind of spikes. There are several packages that are ruined by them, more when doing full-history graphs. It may require some tweaking, and we may need to reach to some data experts here to help us but the big majority of these spikes are clearly outliers that should be easy to remove.

We could do a few things after the cleanup:

1. remove that data points and not show them
2. extrapolate in the missing region so we get a full graph
3. extrapolate using a dotted line so users know that this part of the graph is estimated

I think we should go with the last option.

remove zeroed data

Another issue in npm data is that sometimes it gets borked for a few days. During these periods, sometimes lasting a few days, npm returns zero. See for example the Nov 9, 2024 incident. We can extrapolate when the data is zerod out.

[patak-dev]

asking for help on atproto

[QuietImCoding]

I'm not a super-expert but you can do a sliding window over the data and it should smooth the graph and make outliers look like little bumps instead of crazy spikes

https://en.wikipedia.org/wiki/Savitzky–Golay_filter overly complicated
https://en.wikipedia.org/wiki/Moving_average an exponential moving average would probably crush that spike pretty easily

[korompaiistvan]

answering the bat signal.

@patak-dev do you know of anything that could cause negative non-zero spikes? as in, we know a rogue CI can cause a significant bump in the line, but is there something else that can depress the line without pushing it to zero completely?

in general, i think using the simplest working algo is advisable. in my opinion it should follow the following steps:

1. calculate a moving average and a deviation metric (e.g. median absolute deviation) around each point looking backwards and forwards for the window.
2. compare the value of each point with the moving average and deviation metric at the given point.
3. if the comparison suggests the value is an outlier, filter the value out. if the value is not an outlier, display the original point value without modification

notably, this filtering approach would not squish expected undulations in the line (e.g. weekends). simply taking a moving average and displaying that would however.

[webpro]

Just to add another recent outlier (so far looks much like a "monthly Christmas dip"): https://npmtrends.com/knip

[gameroman]

Yeah, I think all packages have a dip during christmas

[webpro]

In January? 😬

[gameroman]

Oh I see what you mean

[korompaiistvan]

the "nice" thing about dips is that they don't distort the scale of the chart though, so you might want to just not worry about them

[webpro]

Not worrying at all, just a data point that might perhaps be useful in the pursuit of smoothening lines.

[jycouet]

Usually, in process industry we use a lot of first order filtering to filter properly noise / distrubance (that's great to keep dynamics). But depending on the granularity of data and size of the distrubance it can be challenging!

I played with "Smoothing" (first order filter)
Then with Hample & Outlier threshold
And combination to see what would be good.

MAD = Median Absolute Deviation. it's like standard deviation but robust to outliers. For each window, we compute the median, then the median of all distances from that median. So 4 MADs ≈ "replace a point if it's more than ~4 standard deviations from the local median" -> catches truly abnormal
(green in the calculated trend)
https://github.com/user-attachments/assets/f969556a-5d72-4eee-8e56-2fed2faf0918

Maybe each chart can be displayed with these defaults and people can tune them to see more raw values when needed?

[jycouet]

Here is the playground https://svelte.dev/playground/298ad5f21eb84fda95c27453a43701fa?version=5.50.1