REF: refactor Graph.to_W to avoid perf bottleneck

[martinfleis]

Closes #672

For adjacency of 1,404,080 edges, the conversion time is 42.7s on main and 4.5s in this PR.

@jGaboardi not sure how to properly label this...

[jGaboardi]

I'd say this is an "enhancement" due to the performance improvement.

[martinfleis]

Thinking about this more, I think that we shall use the same logic in weights and neighbors properties that are unbearably slow for large graphs. Hard to catch these things when developing on small graphs...

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.0%. Comparing base (018f1e2) to head (bcabdbc).
Report is 4 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##            main    #691     +/-   ##
=======================================
+ Coverage   84.7%   85.0%   +0.3%     
=======================================
  Files        141     141             
  Lines      15200   15203      +3     
=======================================
+ Hits       12880   12924     +44     
+ Misses      2320    2279     -41     

Files	Coverage Δ
libpysal/graph/base.py	97.9% <100.0%> (+<0.1%)	⬆️

... and 11 files with indirect coverage changes

[knaaptime]

i got a question about the performance of the weights builders in the workshop yesterday, and i had to proudly scoff that they're the fastest you can find... 'Martin uses these routinely on datasets in the millions' :D

[knaaptime]

looks nice. The idea is the groupby already gives you all the info you need, so just bypass the apply and build the lists cheaply?
Agreed, lets adopt this same style elsewhere

[jGaboardi]

Do we know what's causing the failures on macOS & ubuntu-dev?

[knaaptime]

the mac test looks like its probably a fluke. the ubuntu stuff is all over the place

[martinfleis]

Builders are fast but some compatibility bits seem to be worse :). I'm noticing it only now when I regularly use Graph on large data. There's a reason it is still experimental :)

[ljwolf]

This looks fine to me! I think we may want to consider some benchmarks with asv for these kinds of things... I think that construction, serialization, conversion, lag, and standardization are the big targets?

[jGaboardi]

Looks like all failures will be fixed by #692. Going ahead with that merge now and see if we can get this green.

[jGaboardi]

All green (for now --> the macOS failure just before had to do with a connection issue for geodatasets).

Not sure if we wanted to see about implementing @ljwolf's idea for asv benchmarking here, or that's something from the future. Thinking probably it's future, but I'll wait on merging until confirmed.

[martinfleis]

Not going to do asv here. I'm generally a bit skeptic about it given we have it in geopandas and no one is running it or anything.