Geotable plot

[darribas]

This is a PR to bring into pysal/dev work to implement easy plotting for geotables. Not to be merged yet but rather to create a place to centralized the discussion around design details and so on. My idea is that, once we're happy with it we merge and, ideally, we have it in by the time @ljwolf's GSoC project gets merged as well so we can have a big splash on simplifying workflows with spatial data.

So far, this implements basic machinery to plot geotables with both a matplotlib and a bokeh backend and contains a color.py module with palette support that depends on bewer2mpl. There's a notebook (geotable_plot) explaining the new functionality.

[coveralls]

Coverage increased (+0.0005%) to 85.28% when pulling dc0ae2a on darribas:geotable_plot into 336cfd0 on pysal:dev.

[darribas]

brewer2mpl mentions that it's superseded by paletteable, should we switch?

@sjsrey @ljwolf

[ljwolf]

Ahh, good call!

I think it's smart to switch to palettable since it involves much more than just brewer, but am not sure of how much would need to be refactored. Probably just colors.py? And, maybe changes in the ipywidgets colorpicker?

I could probably author the necessary changes over the weekend, but I can review any stuff this week.

[darribas]

yes, I think all of the refactoring should happen in color.py. I really like the modular approach you took in there. At the end of the day, the API should be: pass in an indexed series of classes and a palette, receive an equally indexed series of colors. I also really like the option to select the style of the colors (hex, rgb, etc.). bokeh only seems to take hex, so the default currently set is also great.

[ljwolf]

I like the colors.py stuff too, but that's 💯 @sjsrey

I agree on the API. Not sure how the style of colors matters in the backend, other than what each package expects naturally. Maybe that'll be more apparent as I dig in.

[darribas]

re. the color style (or unit), I think matplotlib takes essentially anything, but bokeh only hex. I like leaving it open because we might end up using more backends and it seems to be pretty straightforward.

[darribas]

Right, but right now pandas and matplotlib will fail at import.

[coveralls]

Coverage increased (+0.6%) to 83.073% when pulling 85b4db7 on darribas:geotable_plot into b18c2cf on pysal:dev.

[ljwolf]

Cool,

[ljwolf]

Looking into the issue with the brexit map under bokeh that @sjsrey noted, I think the bokeh backend will be limited for complex geometries.

To reprise, the issue was that a call to plot the brexit data in bokeh finishes computing, but crashes before display due to something somewhere in the Javascript rendering.

#this will crash your notebook
maps.geoplot(db, 'Pct_Leave', k=7, alpha=0.5, classi='fisher_jenks', backend='bk')

This is surprising, since there are only ~380 constituencies. In contrast, geotable_plot can show NAT in bokeh just fine ☹️

NAT = ps.pdio.read_files(ps.examples.get_path('NAT.shp'))
maps.geoplot(NAT, 'HR90', alpha=.5, k=7, classi='fisher_jenks', backend='bk')

This led me to look at the vertex counts.

brexit_verts = db.geometry.apply(lambda x: len(x.vertices))
brexit_verts.unique()

The british constituency shapefile has some pretty amazingly detailed polygons, with the Highlands constituency using ~360k points alone. The worst in NAT is under 200 vertices. Thanks mostly to the intense detail with which the Scottish part of the dataset is stored, it takes around 3.5 million points to plot all constituencies, whereas NAT is only 80k points to plot. Extrapolating a bit, it looks like BokehJS or the notebook Javascript chokes at around 2 million points in the bokeh plot.

[ljwolf]

To help the user, we could warn if the plot "size" is too large and the rendering likely to fail, but I'm not sure if the point limit you can find in the brexit data will hold for other datasets.

[darribas]

Thanks very much for this @ljwolf. I think this is not necessarily a bug in our side but just a characteristic or rendering on the browser these days at least. I wonder also if this is bokeh only or any renderer that directly attempts to put that many points on a browser will choke.

I'd be inclined not to set any message because: a) it's probably hard to come up with a number as it might depend on browser, machine specs, etc. and b) it's likely to change over time as bokeh and web technologies improve. What do you think?

[ljwolf]

I agree, looks like a JS issue. I'm not sure when matplotlib will fail, but I'm sure if you try to plot some massive number of points, it'll start to seize up in the image display.

You're also right that it's not worth it to find that limit & display when we think it might fail. I'm game to consider the concerns we discussed from the dev call answered and this to get merged!

[darribas]

Yes, I think at some point any visualization approach is bound to fail if it naively tries to plot all of the data. At that point, approaches like datashader are probably more useful.

+1 on merging as far as I'm concerned, provided executing the guide notebook returns no error.

[darribas]

Just for future reference, this notebook could be an example on how to plot lines with a lot of points and not get bokeh to choke:

https://anaconda.org/jbednar/trajectory/notebook

[sjsrey]

Thanks @ljwolf for digging into this and finding the reason why that particular shapefile was raising the problem. I agree this is not a bug on our side and I'm +1 on merging once the issues you mentioned get resolved. This will be really nice to have in the library!

[coveralls]

Coverage increased (+0.6%) to 83.074% when pulling da44055 on darribas:geotable_plot into b18c2cf on pysal:dev.

[coveralls]

Coverage increased (+0.6%) to 83.074% when pulling da44055 on darribas:geotable_plot into b18c2cf on pysal:dev.

[coveralls]

Coverage increased (+0.6%) to 83.074% when pulling da44055 on darribas:geotable_plot into b18c2cf on pysal:dev.

[coveralls]

Coverage increased (+0.6%) to 83.074% when pulling da44055 on darribas:geotable_plot into b18c2cf on pysal:dev.

[coveralls]

Coverage increased (+0.6%) to 83.074% when pulling da44055 on darribas:geotable_plot into b18c2cf on pysal:dev.

[darribas]

I think the only thing left here is decide whether we want to bring in @ljwolf's work to make qualitative palettes automatically set k if they have a fixed number of colors. There was a PR for that but can't find it now...

[ljwolf]

That PR is here.

I'll take a look at bringing that up to speed.

[ljwolf]

Revised PR on that is here.

[darribas]

OK, that's merged! If tests pass (which they should), are we good to merge? I think so.

[coveralls]

Coverage increased (+0.6%) to 83.074% when pulling 1d19ea3 on darribas:geotable_plot into b18c2cf on pysal:dev.

[coveralls]

Coverage increased (+0.6%) to 83.074% when pulling 1d19ea3 on darribas:geotable_plot into b18c2cf on pysal:dev.

[coveralls]

Coverage increased (+0.6%) to 83.074% when pulling 1d19ea3 on darribas:geotable_plot into b18c2cf on pysal:dev.

[coveralls]

Coverage increased (+0.6%) to 83.074% when pulling 1d19ea3 on darribas:geotable_plot into b18c2cf on pysal:dev.

[coveralls]

Coverage increased (+0.6%) to 83.074% when pulling 1d19ea3 on darribas:geotable_plot into b18c2cf on pysal:dev.