Port to latest holoviews #688

fcr · 2018-07-16T14:48:47Z

These changes allows topographica to run with the head version of holoviews. These requires my push request for featuremapper.
The one thing that that does not run correctly is the composition:
CoG_spec = "Image.X CoG * Image.Y CoG * Image.BlueChannel" XYCoG = chain.instance(group='XYCoG', name='XYCoG', operations = [image_overlay.instance(spec=CoG_spec), factory.instance()]) Compositor.register(Compositor("Image.X CoG * Image.Y CoG", XYCoG, 'XYCoG', 'display'))
defined in topo.analysis. As is, it causes an error when trying to display an overlay of CoGs.
If this is deleted overlays appear without an error. Any attempt that I made to fix this avoided the error, but only displayed outlines of the images.

the latest holoviews.

jbednar

Those changes all look good to me, thanks! @jlstevens, can you also review and maybe think about the remaining issue?

jlstevens · 2018-07-16T17:06:28Z

The remaining issue is down to changes in the compositor in holoviews. For now, I would recommend deleting that line: the compositor is just a convenience and you can achieve the equivalent visualization manually. It is better to get the data out and visualize it the way you please than having to deal with an unnecessary error.

Updating that line to work with the current state of the compositor should be possible but that would require some thought and possibly some extra work too.

"other" argument. This change catches this case.

value_dimensions -> vdims Correct arguments passed to Histogram

Fixed integration of Tk with IPython >= 5.x Fixed matplotlib crash with Tk All unit tests pass if nosetests pointed to unit folder

fcr · 2018-07-27T13:04:26Z

I added some more changes that I made. These solve:

Tk no longer crashes in Mac Seirra
Latest Jupyter 5.x for Python 2.7 runs.
Matplotlib correctly uses Tk.
Additional problems with the latest holoviews.
The unit tests all pass with nose if the path topo/tests/unit is explicitly provided.

Unfortunately there are still two serious problems that i do not know if I will have time to address:

The optimized code randomly generates FloatingPoint errors. These do not occur if the non optimized version is run.
All overlays in the GCAL_Tutorial.ipynb crash with a KeyError. This is probably a Holoviews incompatibility but the call stack is so long and convoluted, because of the deeply nested params that I did not have enough familiarity of the architecture to trace the problem.

jbednar

Nearly all of those changes look good, thanks! But there are a few problematic ones...

jbednar · 2018-07-27T17:20:46Z

topo/misc/commandline.py

-    matplotlib_imported=True
-except ImportError:
-    matplotlib_imported=False
-


Ooh, seeing anyone change this code just raises the hairs on the back of my neck! There have been so, so many issues over the years from Matplotlib choosing the wrong backend, that I am very reluctant ever to change this code. Are you certain that when used in batch mode (no Tk), it still works?

jbednar · 2018-07-27T17:23:16Z

topo/submodel/specifications.py

+            # Change in holoviews
+            if type(other) == str:
+                return False
+            raise e


Can you explain the motivation for this change? When will a Specification ever be compared to a string?

jbednar · 2018-07-27T17:24:07Z

topo/tests/unit/testcf.py

+# Need to load TkAgg before any calls to pyplot in the imports
+import matplotlib
+matplotlib.use("TkAgg")
+


This doesn't seem correct; tests cannot assume that Tk is available in that run.

jbednar · 2018-07-27T17:25:23Z

topo/tkgui/topoconsole.py

@@ -531,7 +531,7 @@ def quit_topographica(self,check=True,exit_status=0):
            except:
                pass

-            self.message("Quit selected%s" % ("; exiting" if  self.exit_on_quit else ""))
+#             self.message("Quit selected%s" % ("; exiting" if  self.exit_on_quit else "")) # Removed because does not restore prompt


Not sure what you mean?

jbednar · 2018-07-27T17:28:29Z

The optimized code randomly generates FloatingPoint errors. These do not occur if the non optimized version is run.

That sounds like a serious problem that should be reported in a separate issue with a reproducible test case (even if you have to say "run this code 100 times and it typically has floating point errors 5% of the time", or something like that).

All overlays in the GCAL_Tutorial.ipynb crash with a KeyError. This is probably a Holoviews incompatibility but the call stack is so long and convoluted, because of the deeply nested params that I did not have enough familiarity of the architecture to trace the problem.

I would guess that the plots in the GCAL tutorial have polar orientation map plots that depend on the Compositor that you have commented out. @jlstevens, what do you propose should happen to that type of plot?

jlstevens · 2018-07-27T18:44:38Z

what do you propose should happen to that type of plot?

@fcr I'll probably have a better idea how to resolve this if you could post the output of print around the object that is failing with the keyerrors (presumably during display).

fcr · 2018-07-29T09:14:31Z

@jlstevens.
I simply run the GCAL_Tutorial.ipynb with the model = 'LISSOM'.
The cell, for example:
`pref = data.OrientationPreference.V1 sel = data.OrientationSelectivity.V1 (pref*sel).select(Time=max_time)`
crashes with:

KeyError Traceback (most recent call last)
/Users/fred/anaconda/envs/python2/lib/python2.7/site-packages/IPython/core/formatters.pyc in call(self, obj, include, exclude)
968 d['include'] = include
969 d['exclude'] = exclude
--> 970 return method(**d)
971 return None
972 else:

/Users/fred/Development/topographica/external/holoviews/holoviews/core/dimension.pyc in repr_mimebundle(self, include, exclude)
1263 combined and returned.
1264 """
-> 1265 return Store.render(self)
1266
1267

/Users/fred/Development/topographica/external/holoviews/holoviews/core/options.pyc in render(cls, obj)
1287 data, metadata = {}, {}
1288 for hook in hooks:
-> 1289 ret = hook(obj)
1290 if ret is None:
1291 continue

/Users/fred/Development/topographica/external/holoviews/holoviews/ipython/display_hooks.pyc in pprint_display(obj)
270 if not ip.display_formatter.formatters['text/plain'].pprint:
271 return None
--> 272 return display(obj, raw_output=True)
273
274

/Users/fred/Development/topographica/external/holoviews/holoviews/ipython/display_hooks.pyc in display(obj, raw_output, **kwargs)
243 elif isinstance(obj, (Layout, NdLayout, AdjointLayout)):
244 with option_state(obj):
--> 245 output = layout_display(obj)
246 elif isinstance(obj, (HoloMap, DynamicMap)):
247 with option_state(obj):

/Users/fred/Development/topographica/external/holoviews/holoviews/ipython/display_hooks.pyc in wrapped(element)
140 try:
141 max_frames = OutputSettings.options['max_frames']
--> 142 mimebundle = fn(element, max_frames=max_frames)
143 if mimebundle is None:
144 return {}, {}

/Users/fred/Development/topographica/external/holoviews/holoviews/ipython/display_hooks.pyc in layout_display(layout, max_frames)
213 return None
214
--> 215 return render(layout)
216
217

/Users/fred/Development/topographica/external/holoviews/holoviews/ipython/display_hooks.pyc in render(obj, **kwargs)
63 renderer = renderer.instance(fig='png')
64
---> 65 return renderer.components(obj, **kwargs)
66
67

/Users/fred/Development/topographica/external/holoviews/holoviews/plotting/renderer.pyc in components(self, obj, fmt, comm, **kwargs)
327 plot = obj
328 else:
--> 329 plot, fmt = self._validate(obj, fmt)
330
331 widget_id = None

/Users/fred/Development/topographica/external/holoviews/holoviews/plotting/renderer.pyc in _validate(self, obj, fmt, **kwargs)
226 if isinstance(obj, tuple(self.widgets.values())):
227 return obj, 'html'
--> 228 plot = self.get_plot(obj, renderer=self, **kwargs)
229
230 fig_formats = self.mode_formats['fig'][self.mode]

/Users/fred/Development/topographica/external/holoviews/holoviews/plotting/renderer.pyc in get_plot(self_or_cls, obj, renderer)
209 plot_opts = self_or_cls.plot_options(obj, self_or_cls.size)
210 plot = self_or_cls.plotting_class(obj)(obj, renderer=renderer,
--> 211 **plot_opts)
212 defaults = [kd.default for kd in plot.dimensions]
213 init_key = tuple(v if d is None else d for v, d in

/Users/fred/Development/topographica/external/holoviews/holoviews/plotting/mpl/plot.pyc in init(self, layout, keys, **params)
749 super(LayoutPlot, self).init(layout=layout, keys=keys, **params)
750 with mpl.rc_context(rc=self.fig_rcparams):
--> 751 self.subplots, self.subaxes, self.layout = self._compute_gridspec(layout)
752 if self.top_level:
753 self.comm = self.init_comm()

/Users/fred/Development/topographica/external/holoviews/holoviews/plotting/mpl/plot.pyc in _compute_gridspec(self, layout)
938 layout_dimensions, frame_ranges,
939 dict(zip(positions, subaxes)),
--> 940 num=0 if empty else layout_count)
941 subplots, adjoint_layout, _ = subplot_data
942 layout_axes[(r, c)] = subaxes

/Users/fred/Development/topographica/external/holoviews/holoviews/plotting/mpl/plot.pyc in _create_subplots(self, layout, positions, layout_dimensions, ranges, axes, num, create)
1060 ranges=ranges, subplot=True,
1061 uniform=self.uniform, layout_num=num,
-> 1062 renderer=self.renderer, **plotopts)
1063 if isinstance(view, (Element, HoloMap, Collator, CompositeOverlay)):
1064 adjoint_clone[pos] = subplots[pos].hmap

/Users/fred/Development/topographica/external/holoviews/holoviews/plotting/mpl/element.pyc in init(self, overlay, ranges, **params)
748 if 'projection' not in params:
749 params['projection'] = self._get_projection(overlay)
--> 750 super(OverlayPlot, self).init(overlay, ranges=ranges, **params)
751
752

/Users/fred/Development/topographica/external/holoviews/holoviews/plotting/mpl/element.pyc in init(self, element, **params)
60
61 def init(self, element, **params):
---> 62 super(ElementPlot, self).init(element, **params)
63 check = self.hmap.last
64 if isinstance(check, CompositeOverlay):

/Users/fred/Development/topographica/external/holoviews/holoviews/plotting/plot.pyc in init(self, overlay, ranges, batched, keys, group_counter, **params)
885
886 # Apply data collapse
--> 887 self.hmap = self._apply_compositor(self.hmap, ranges, self.keys)
888 self.map_lengths = Counter()
889 self.group_counter = Counter() if group_counter is None else group_counter

/Users/fred/Development/topographica/external/holoviews/holoviews/plotting/plot.pyc in _apply_compositor(self, holomap, ranges, keys, dimensions)
919 ranges = frame_ranges.values()
920
--> 921 return Compositor.collapse(holomap, (ranges, frame_ranges.keys()), mode='display')
922
923

/Users/fred/Development/topographica/external/holoviews/holoviews/core/options.pyc in collapse(cls, holomap, ranges, mode)
874 data = zip(ranges[1], holomap.data.values()) if ranges else holomap.data.items()
875 for key, overlay in data:
--> 876 clone[key] = cls.collapse_element(overlay, ranges, mode)
877 return clone
878

/Users/fred/Development/topographica/external/holoviews/holoviews/core/options.pyc in collapse_element(cls, overlay, ranges, mode, backend)
843 if applicable_op and all(el in processed[applicable_op] for el in items):
844 return overlay
--> 845 result = applicable_op.apply(sliced, ranges, backend)
846 if applicable_op.group:
847 result = result.relabel(group=applicable_op.group)

/Users/fred/Development/topographica/external/holoviews/holoviews/core/options.pyc in apply(self, value, input_ranges, backend)
1009 if k in self.operation.params()})
1010
-> 1011 transformed = self.operation(value, input_ranges=input_ranges, **kwargs)
1012 if self.transfer_options:
1013 Store.transfer_options(value, transformed, backend)

/Users/fred/Development/topographica/external/param/param/parameterized.pyc in new(class_, *args, **params)
2047 inst = class_.instance()
2048 inst.param.set_name(class.name)
-> 2049 return inst.call(*args,**params)
2050
2051 def call(self,*args,**kw):

/Users/fred/Development/topographica/external/holoviews/holoviews/core/operation.pyc in call(self, element, **params)
161 operation=self, kwargs=params)
162 elif isinstance(element, ViewableElement):
--> 163 processed = self._apply(element)
164 elif isinstance(element, DynamicMap):
165 if any((not d.values) for d in element.kdims):

/Users/fred/Development/topographica/external/holoviews/holoviews/core/operation.pyc in _apply(self, element, key)
119 for hook in self._preprocess_hooks:
120 kwargs.update(hook(self, element))
--> 121 ret = self._process(element, key)
122 for hook in self._postprocess_hooks:
123 ret = hook(self, ret, **kwargs)

/Users/fred/Development/topographica/external/featuremapper/featuremapper/analysis/init.pyc in _process(self, overlay, key)
277 normfn = raster_normalization.instance()
278 if self.p.input_ranges:
--> 279 overlay = normfn.process_element(overlay, key, *self.p.input_ranges)
280 else:
281 overlay = normfn.process_element(overlay, key)

/Users/fred/Development/topographica/external/holoviews/holoviews/operation/normalization.pyc in process_element(self, element, key, ranges, keys, **params)
88 params = dict(params,ranges=ranges, keys=keys)
89 self.p = param.ParamOverrides(self, params)
---> 90 return self._process(element, key)
91
92

/Users/fred/Development/topographica/external/holoviews/holoviews/operation/normalization.pyc in _process(self, raster, key)
145 overlay_clone = raster.clone(shared_data=False)
146 for k, el in raster.items():
--> 147 overlay_clone[k] = self._normalize_raster(el, key)
148 return overlay_clone
149 else:

/Users/fred/Development/topographica/external/holoviews/holoviews/operation/normalization.pyc in _normalize_raster(self, raster, key)
154 if not isinstance(raster, Raster): return raster
155 norm_raster = raster.clone(raster.data.copy())
--> 156 ranges = self.get_ranges(raster, key)
157
158 for depth, name in enumerate(d.name for d in raster.vdims):

/Users/fred/Development/topographica/external/holoviews/holoviews/operation/normalization.pyc in get_ranges(self, element, key)
113 specs = ranges[index]
114 except:
--> 115 raise KeyError("Could not match element key to defined keys")
116 else:
117 raise ValueError("Key list length must match length of supplied ranges")

KeyError: 'Could not match element key to defined keys'

:Layout
.Orientation_Preference.V1.I :HoloMap [Time,Duration]
:Image [x,y] (Orientation Preference)
.Orientation_Selectivity.V1 :HoloMap [Time,Duration]
:Image [x,y] (Orientation Selectivity)
.Orientation_Preference.V1.II :HoloMap [Time,Duration]
:Overlay
.Orientation_Preference.V1 :Image [x,y] (Orientation Preference)
.Orientation_Selectivity.V1 :Image [x,y] (Orientation Selectivity)

Looking at the data with a breakpoint at the exception I had:
str: _fig_rcparams_param_value

element: Image: :Image [x,y] (Orientation Preference)
key: None
keys: <type 'list'>: [(mpq(100,1), mpq(7,40))]
ranges: <type 'list'>: [{('Image',): OrderedDict([('x', (-0.5, 0.5)), ('y', (-0.5, 0.5)), ('Orientation Preference', (0.0, 3.141592653589793)), ('Orientation Selectivity', (0.006849852636067116, 1.3778337397815732))])}]
self: raster_normalization: raster_normalization()

The key gets its None value as the default argument in the call to _apply(element) in line 163 of /Users/fred/anaconda/envs/python2/lib/python2.7/site-packages/holoviews/core/operation.pyc.
In all cases of Overlays, where there was a crash, the key was None.

Hope that this helps

Fred

fcr · 2018-07-29T09:20:27Z

BTW the problem does not rely on Compositor that was commented out.

jbednar · 2018-07-29T21:35:49Z

I didn't study the entire traceback, but it does mention Compositor, so I would guess this error is just to do with Compositor not being defined (as it's commented out).

fcr · 2018-07-30T06:40:16Z

I made my previous comment after I restored the Compositor. Made no difference. Not surprizing since the Compositor is keyed to cog and this is a general problem.

jlstevens · 2018-07-31T14:48:55Z

I think this is too trick to debug from the traceback alone so I'll have a go at reproducing the bug soon. I suspect it might be to do with how normalization information used to be stored but I might be wrong about that...

fcr added 2 commits July 16, 2018 15:47

Works with latest Holoviews and FeatureMapper

a64c55e

Updated externals except that featuremapper is not yet the one ported to

b2217c5

the latest holoviews.

jbednar approved these changes Jul 16, 2018

View reviewed changes

fcr added 5 commits July 17, 2018 10:03

Removed broken Compositor scriptUpdated external featuremapper

4e19854

Changes to holoviews can result in the string "_DELETE" being the

418c165

"other" argument. This change catches this case.

key_dimensions -> kdims

f472875

value_dimensions -> vdims Correct arguments passed to Histogram

Fixed additional calls to holoviews

b27c15a

Fixed integration of Tk with IPython >= 5.x Fixed matplotlib crash with Tk All unit tests pass if nosetests pointed to unit folder

Changes for TK compatible with Jupyter

502fe9d

jbednar requested changes Jul 27, 2018

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Port to latest holoviews #688

Port to latest holoviews #688

fcr commented Jul 16, 2018 •

edited by jlstevens

jbednar left a comment

jlstevens commented Jul 16, 2018

fcr commented Jul 27, 2018

jbednar left a comment

jbednar Jul 27, 2018

jbednar Jul 27, 2018

jbednar Jul 27, 2018

jbednar Jul 27, 2018

jbednar commented Jul 27, 2018

jlstevens commented Jul 27, 2018

fcr commented Jul 29, 2018 •

edited

fcr commented Jul 29, 2018

jbednar commented Jul 29, 2018

fcr commented Jul 30, 2018

jlstevens commented Jul 31, 2018

Port to latest holoviews #688

Are you sure you want to change the base?

Port to latest holoviews #688

Conversation

fcr commented Jul 16, 2018 • edited by jlstevens

jbednar left a comment

Choose a reason for hiding this comment

jlstevens commented Jul 16, 2018

fcr commented Jul 27, 2018

jbednar left a comment

Choose a reason for hiding this comment

jbednar Jul 27, 2018

Choose a reason for hiding this comment

jbednar Jul 27, 2018

Choose a reason for hiding this comment

jbednar Jul 27, 2018

Choose a reason for hiding this comment

jbednar Jul 27, 2018

Choose a reason for hiding this comment

jbednar commented Jul 27, 2018

jlstevens commented Jul 27, 2018

fcr commented Jul 29, 2018 • edited

@jlstevens. I simply run the GCAL_Tutorial.ipynb with the model = 'LISSOM'. The cell, for example: pref = data.OrientationPreference.V1 sel = data.OrientationSelectivity.V1 (pref*sel).select(Time=max_time) crashes with:

fcr commented Jul 29, 2018

jbednar commented Jul 29, 2018

fcr commented Jul 30, 2018

jlstevens commented Jul 31, 2018

fcr commented Jul 16, 2018 •

edited by jlstevens

fcr commented Jul 29, 2018 •

edited

@jlstevens.
I simply run the GCAL_Tutorial.ipynb with the model = 'LISSOM'.
The cell, for example:
`pref = data.OrientationPreference.V1 sel = data.OrientationSelectivity.V1 (pref*sel).select(Time=max_time)`
crashes with: