ENH: include Graph.describe() to describe neighbourhood values

[u3ks]

This PR adds a method to the graph api which takes an array of values and calculates descriptive statistics within each neighborhood.
Optionally, some neighbors can be filtered out based on the percentiles of the passed values.
The supported stats are - "count", "mean", "median", "std", "min", "max", "sum", "nunique" and "mode".

The method similar to .apply, but all values are calculated in one grouping operation and all functions are jitted.

[martinfleis]

Just to add some context to this. As we are refactoring momepy, we realised that we rely very often on this internal function, which is fairly generic and shall be tied directly to Graph.

The idea behind the q limiting the range is coming from morphology. We often want to get some sort of a spatial average but given the high likelihood of outliers (think of a church in the middle of a neighborhood), we can't include all the values within each neighborhood.

[ljwolf]

I think, for generality, this should be called a truncated or trimmed reduction/lag?

This is very useful generally... @weikang9009 and I have been working on related concepts recently, so it'd be very nice to have something core here!

[martinfleis]

I think, for generality, this should be called a truncated or trimmed reduction/lag?

Only if q is not None. Otherwise it is just a generic lag. I am also not sure what can be called a lag (nunique?). The describe terminology comes from pandas. It felt close enough to what we're doing here.

[codecov]

Codecov Report

Attention: Patch coverage is 97.82609% with 2 lines in your changes missing coverage. Please review.

Project coverage is 85.1%. Comparing base (bcabdbc) to head (879f3f5).
Report is 18 commits behind head on main.

Additional details and impacted files

@@          Coverage Diff           @@
##            main    #717    +/-   ##
======================================
  Coverage   85.0%   85.1%            
======================================
  Files        141     145     +4     
  Lines      15203   15483   +280     
======================================
+ Hits       12924   13169   +245     
- Misses      2279    2314    +35     

Files	Coverage Δ
libpysal/graph/tests/test_base.py	100.0% <100.0%> (ø)
libpysal/graph/_utils.py	97.1% <97.6%> (+2.2%)	⬆️
libpysal/graph/base.py	96.8% <92.9%> (-1.1%)	⬇️

... and 6 files with indirect coverage changes

[jGaboardi]

Should this be pandas.Grouper?

[u3ks]

I think the pandas.Grouper is another type of object, that is used for filtering columns , i used the name grouper since its used in other functions and the type is groupby since, pandas groupy returns a groupby object

[jGaboardi]

Maybe 'nan' is OK here, but also maybe NaN or numpy.nan (or something else?)

Probably not a big deal either way.

[u3ks]

changed to numpy.nan

[martinfleis]

Looking at this, we may want to check that the y.index matches self.unique_ids in case of a custom Series is passed. I suppose that non-matching index may break this.

[u3ks]

added a check

[knaaptime]

i think i would call this describe_cardinalities or something because "Graph.describe() to describe neighbourhood values" implies we're looking at the neighbor values

[martinfleis]

But this is not describing cardinalities, no? Where cardinality is a number of elements in a set. It is describing distribution of values within a neighbourhood.

[knaaptime]

oh i see. It was this note on line 2014 that tripped me up:

'Weight values do not affect the calculations, only adjacency does.'