GridInterface bin support for Histogram and QuadMesh #2160

philippjfr · 2017-11-28T22:47:15Z

This PR adds support for binned coordinates to the GridInterface addressing #547. This is one of the final conversions of Elements to a Dataset type.

Get unit tests passing
Add __setstate__ for pickle compatibility
Move Histogram and QuadMesh tests into testbinneddataset.py or similar
Add examples of irregular meshes to reference documentation and gallery

Several nice to haves, which don't necessarily have to hold up merging this PR:

Support for xarray Datasets
Support for irregular meshes (i.e. 2D array coordinates)

I've now managed to also support irregular meshes, which are a commonly requested feature particularly for GeoViews. Support for datashading quadmeshes will come in a later PR (probably after TriMesh is merged). Here are some irregular meshes:

And all the usual machinery also works to explore high-dimensional datasets, e.g. here is an example of a multi-dimensional irregularly gridded dataset (requested in holoviz/geoviews#57 and holoviz/geoviews#73).

lon, lat = np.meshgrid(np.linspace(-20, 20, 5), np.linspace(0, 30, 4))
lon += lat/10
lat += lon/10

da = xr.DataArray(np.arange(40).reshape(2, 4, 5), dims=['z', 'y', 'x'],
                  coords = {'lat': (('y', 'x'), lat),
                            'lon': (('y', 'x'), lon),
                            'z': [0, 1]}, name='A')

ds = hv.Dataset(da, ['lat', 'lon', 'z'])
ds.to(hv.QuadMesh, groupby=['z'])

philippjfr · 2017-11-30T02:51:50Z

Here is an overview of the new functionality.

jbednar · 2017-11-30T03:03:37Z

Nice!

philippjfr · 2017-11-30T03:20:33Z

@jbednar One question I have is how we can properly support datashading a QuadMesh, currently I see two options:

We can treat them like points which means the resulting output cannot have a density higher than the lowest density portion of the mesh because otherwise the output will have sampling artifacts.
We can treat them like rasters, which I think means that the areas will not be appropriately weighted and I haven't figured out how I'd go about generating appropriate coordinates for each pixel after regridding.

Both don't seem quite right so the real solution is getting actual support for quadmeshes into datashader.

jbednar · 2017-11-30T03:31:23Z

What about slicing every bin in half and rendering it as a TriMesh?

philippjfr · 2017-11-30T03:36:29Z

Neat idea!

philippjfr · 2017-12-19T22:48:22Z

Now implemented datashader rasterization of QuadMesh types, based on your suggestion. It's fairly fast once precompute is enabled.

philippjfr · 2017-12-20T16:02:01Z

@jbednar @jlstevens Requesting review.

jlstevens · 2017-12-21T13:55:00Z

examples/reference/elements/matplotlib/QuadMesh.ipynb

+    "Qz = np.sin(Y) + np.sin(X)\n",
+    "Z = np.sqrt(X**2 + Y**2)\n",
+    "\n",
+    "print(Qx.shape, Qz.shape, Z.shape)\n",


Stray print? If not it needs to print a proper message.

Sure, just meant to show how the data differs between unevenly sampled and irregularly sampled QuadMeshes.

Ok. Then I would draw attention to that more explicitly including a proper print message, maybe in a separate cell.

jlstevens · 2017-12-21T13:59:45Z

holoviews/element/raster.py

+        t3s = (js+1)*(s0+1)+t1%s0
+        t4s = t2s
+        t5s = t3s
+        t6s = t3s+1


This is pretty unreadable and the variable names are uninformative.

If this is doing some self-contained computation, it would be better as a utility which can at least have a docstring explaining what this is supposed to be doing.

Sure, a utility is fine.

End up with all kinds of circular import issues if I try to move it to element.util, I've added various comments which should clarify.

jlstevens · 2017-12-21T14:10:46Z

holoviews/element/raster.py


+        Note: Deprecate as part of 2.0


Is this being tested via the current notebook tests?

It was, but once tests are rebuilt just once it won't be.

jlstevens · 2017-12-21T14:13:09Z

holoviews/core/data/grid.py

                raise error('Key dimension values and value array %s '
                            'shapes do not match. Expected shape %s, '
                            'actual shape: %s' % (vdim, expected[::-1], shape), cls)
        return data, {'kdims':kdims, 'vdims':vdims}, {}


+    @classmethod
+    def irregular(cls, dataset, dim):


You've used the word irregular a lot. I am wondering if uneven would have been clearer though it probably isn't worth changing now.

irregular != uneven, they are different concepts.

jlstevens · 2017-12-21T14:14:57Z

holoviews/core/data/xarray.py

+        if edges and not isedges:
+            data = cls._infer_interval_breaks(data)
+        elif not edges and isedges:
+            data = np.convolve(data, [0.5, 0.5], 'valid')


Why is there a convolution here?

Easiest way to compute a rolling mean, i.e. edges -> bin centers.

jlstevens · 2017-12-21T14:15:46Z

Looks fine and happy to merge once you've addressed my questions above.

jlstevens · 2017-12-21T14:29:39Z

After discussing 'uneven' versus 'irregular', I think an irregular mesh example where the sample positions have a small (but obvious) random jitter would make the meaning of 'irregular' a lot clearer.

jlstevens · 2017-12-21T16:48:19Z

Tests have passed. Ready to merge?

philippjfr · 2017-12-21T19:44:47Z

Yes, let's merge, should get as much testing and exercising as possible before the next release.

jlstevens · 2017-12-22T12:04:10Z

Merged!

philippjfr changed the title ~~Added GridInterface bin support for Histogram and QuadMesh~~ GridInterface bin support for Histogram and QuadMesh Nov 28, 2017

philippjfr force-pushed the binned_datasets branch from 1fedff8 to dc1ca4b Compare November 29, 2017 20:12

philippjfr added this to the v1.10 milestone Nov 30, 2017

philippjfr force-pushed the binned_datasets branch from 9270153 to 73696bb Compare November 30, 2017 16:18

philippjfr mentioned this pull request Nov 30, 2017

Image should warn about unevenly spaced dimensions #1869

Closed

philippjfr force-pushed the binned_datasets branch from 5b298b3 to 793ac32 Compare December 5, 2017 01:33

jbednar mentioned this pull request Dec 14, 2017

Log axis when plotting holoviews object through datashader #2195

Open

philippjfr force-pushed the binned_datasets branch from 276dba6 to b41765f Compare December 19, 2017 18:36

philippjfr force-pushed the binned_datasets branch from c0b09b2 to 4e6b5ce Compare December 20, 2017 00:16

philippjfr added 15 commits December 20, 2017 14:56

Added GridInterface bin support for Histogram and QuadMesh

17f3fcb

Fixed various issues for binned data

7b8fa03

Ensure QuadMesh uses GridInterface

dfd0261

Added Histogram values and edges properties for backward compatibility

74fe640

Added support for irregular meshes

5e032fd

Improved support for irregular Datasets

a646342

Improvements and docs for QuadMesh

51dfbed

Fixed bugs introduced by irregular data handling

e9e1edd

Added old Histogram and QuadMesh pickle compatibility

93052bc

Added gallery examples for irregular QuadMesh

84ca707

Fixed interval breaks doctest

84743a5

Cleaned up colormesh utility

2d62b34

Cleanup and fixes

d7d6b3e

Handled scalar coords in xarray and grid interfaces

08fafd0

Allow slicing irregular meshes

c5db6b8

philippjfr added 5 commits December 20, 2017 14:56

Guard array and dictionary interface against wrong dimensionality

e5668ac

Added binned dataset tests

6af46f4

Implemented QuadMesh to TriMesh conversion

af9724b

Implemented QuadMesh datashader rasterization

981a218

Fixed array and dict interface validation

36a2ec4

philippjfr force-pushed the binned_datasets branch from 4e6b5ce to 36a2ec4 Compare December 20, 2017 14:56

Added unit tests for datashading QuadMesh

7aabc20

philippjfr force-pushed the binned_datasets branch from a68631d to 7aabc20 Compare December 20, 2017 16:14

Added strict validation for Image sampling

2d346b2

jlstevens reviewed Dec 21, 2017

View reviewed changes

philippjfr added 3 commits December 21, 2017 15:35

Expanded on irregular meshes

298bbc4

Added comments for QuadMesh -> TriMesh conversion

58676fb

Losened Image sampling validation

ee012a1

jlstevens merged commit 0f5ad5d into master Dec 22, 2017

philippjfr deleted the binned_datasets branch January 13, 2018 13:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GridInterface bin support for Histogram and QuadMesh #2160

GridInterface bin support for Histogram and QuadMesh #2160

philippjfr commented Nov 28, 2017 •

edited

philippjfr commented Nov 30, 2017 •

edited

jbednar commented Nov 30, 2017

philippjfr commented Nov 30, 2017 •

edited

jbednar commented Nov 30, 2017

philippjfr commented Nov 30, 2017

philippjfr commented Dec 19, 2017 •

edited

philippjfr commented Dec 20, 2017

jlstevens Dec 21, 2017

philippjfr Dec 21, 2017

jlstevens Dec 21, 2017

jlstevens Dec 21, 2017

jlstevens Dec 21, 2017

philippjfr Dec 21, 2017

philippjfr Dec 21, 2017

jlstevens Dec 21, 2017

philippjfr Dec 21, 2017

jlstevens Dec 21, 2017

philippjfr Dec 21, 2017

jlstevens Dec 21, 2017

philippjfr Dec 21, 2017

jlstevens commented Dec 21, 2017

jlstevens commented Dec 21, 2017

jlstevens commented Dec 21, 2017

philippjfr commented Dec 21, 2017

jlstevens commented Dec 22, 2017

GridInterface bin support for Histogram and QuadMesh #2160

GridInterface bin support for Histogram and QuadMesh #2160

Conversation

philippjfr commented Nov 28, 2017 • edited

philippjfr commented Nov 30, 2017 • edited

jbednar commented Nov 30, 2017

philippjfr commented Nov 30, 2017 • edited

jbednar commented Nov 30, 2017

philippjfr commented Nov 30, 2017

philippjfr commented Dec 19, 2017 • edited

philippjfr commented Dec 20, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jlstevens commented Dec 21, 2017

jlstevens commented Dec 21, 2017

jlstevens commented Dec 21, 2017

philippjfr commented Dec 21, 2017

jlstevens commented Dec 22, 2017

philippjfr commented Nov 28, 2017 •

edited

philippjfr commented Nov 30, 2017 •

edited

philippjfr commented Nov 30, 2017 •

edited

philippjfr commented Dec 19, 2017 •

edited