[BUG] Ipywidgets package update (8.0.1) broke latest Sparkmagic 0.20.0

[soobidou]

Lates version was fine couple days ago and broke because of Ipywidgets probably doing a major non-backward compatible change

I see in requirements.txt that SparkMagic constraints are ipywidgets>5.0.0

Here's what happen after the Livy session is loaded up:

TypeError                                 Traceback (most recent call last)
File /opt/conda/lib/python3.8/site-packages/IPython/core/formatters.py:916, in IPythonDisplayFormatter.__call__(self, obj)
    914     pass
    915 else:
--> 916     printer(obj)
    917     return True
    918 # Finally look for special method names

File /opt/conda/lib/python3.8/site-packages/autovizwidget/widget/utils.py:136, in display_dataframe(df)
    129 selected_y = select_y(df, selected_x)
    130 encoding = Encoding(
    131     chart_type=Encoding.chart_type_table,
    132     x=selected_x,
    133     y=selected_y,
    134     y_aggregation=Encoding.y_agg_max,
    135 )
--> 136 return AutoVizWidget(df, encoding)

File /opt/conda/lib/python3.8/site-packages/autovizwidget/widget/autovizwidget.py:49, in AutoVizWidget.__init__(self, df, encoding, renderer, ipywidget_factory, encoding_widget, ipython_display, nested_widget_mode, spark_events, testing, **kwargs)
     46 self.ipywidget_factory = ipywidget_factory
     48 if encoding_widget is None:
---> 49     encoding_widget = EncodingWidget(self.df, encoding, self.on_render_viz)
     50 self.encoding_widget = encoding_widget
     52 if ipython_display is None:

File /opt/conda/lib/python3.8/site-packages/autovizwidget/widget/encodingwidget.py:48, in EncodingWidget.__init__(self, df, encoding, change_hook, ipywidget_factory, testing, **kwargs)
     46 options_x_view = {text(i): text(i) for i in self.df.columns}
     47 options_x_view["-"] = None
---> 48 self.x_view = self.ipywidget_factory.get_dropdown(
     49     options=options_x_view, description="X", value=self.encoding.x
     50 )
     51 self.x_view.on_trait_change(self._x_changed_callback, "value")
     52 self.x_view.layout.width = "200px"

File /opt/conda/lib/python3.8/site-packages/hdijupyterutils/ipywidgetfactory.py:45, in IpyWidgetFactory.get_dropdown(**kwargs)
     43 @staticmethod
     44 def get_dropdown(**kwargs):
---> 45     return Dropdown(**kwargs)

File /opt/conda/lib/python3.8/site-packages/ipywidgets/widgets/widget_selection.py:173, in _Selection.__init__(self, *args, **kwargs)
    171 self._initializing_traits_ = True
    172 kwargs['options'] = _exhaust_iterable(kwargs.get('options', ()))
--> 173 self._options_full = _make_options(kwargs['options'])
    174 self._propagate_options(None)
    176 # Select the first item by default, if we can

File /opt/conda/lib/python3.8/site-packages/ipywidgets/widgets/widget_selection.py:115, in _make_options(x)
    106 """Standardize the options tuple format.
    107 
    108 The returned tuple should be in the format (('label', value), ('label', value), ...).
   (...)
    112 * an iterable of values, and labels will be generated
    113 """
    114 if isinstance(x, Mapping):
--> 115     raise TypeError("options must be a list of values or a list of (label, value) tuples")
    117 # only iterate once through the options.
    118 xlist = tuple(x)

TypeError: options must be a list of values or a list of (label, value) tuples

Still investigating.

[jasongrout]

For context:

ipywidgets 8.0 was released yesterday, which is a major version upgrade with breaking changes.

From the above error, it looks like this change in ipywidgets 8 is affecting you:

Drop support for mapping types as selection options (#2679, #1958)

From the looks of it, you may need to just update line 49 below:

File /opt/conda/lib/python3.8/site-packages/autovizwidget/widget/encodingwidget.py:48, in EncodingWidget.__init__(self, df, encoding, change_hook, ipywidget_factory, testing, **kwargs)
     46 options_x_view = {text(i): text(i) for i in self.df.columns}
     47 options_x_view["-"] = None
---> 48 self.x_view = self.ipywidget_factory.get_dropdown(
     49     options=options_x_view, description="X", value=self.encoding.x
     50 )

My guess is that options_x_view is a dictionary (using a dictionary there has been deprecated for years), and should be converted to a list of values or a list of (label, value) tuples. I think you can change it to options_x_view.items()

[jasongrout]

Or it looks like you can define options_x_view as:

options_x_view = [(text(i), text(i)) for i in self.df.columns]
options_x_view.append(("-", None))

[soobidou]

I worked around by installing this after sparkmagic==0.20.0:

jupyter labextension install @jupyter-widgets/jupyterlab-manager@3.1.1
pip install ipywidgets==7.7.0 jupyterlab-widgets==1.1.1 widgetsnbextension==3.6.1 

Not sure about the 7.7.x, I had mixed results during the day, they were obviously altered.
Might be good idea to pin such dependencies so that users relying on a pinned version get a consistent behavior.

[jasongrout]

FYI, with JupyterLab 3.x, if JupyterLab and the kernel are in the same environment, you don't need to do both the jupyter labextension install @jupyter-widgets/jupyterlab-manager@3.1.1 and install jupyterlab-widgets==1.1.1. We recommend just doing the pip install, without the jupyter labextension install these days.

Also, if you install ipywidgets 7.7.2, you'll get (a) some bugfixes in 7.7.1, as well as (b) the updated jupyterlab_widgets dependency on the correct version of jupyterlab_widgets. But yes, it's always a good idea to pin dependencies exactly (or at least to the minor version) for reproducibility.

tldr; if your jupyterlab and kernel are in the same environment, you should be able to do

pip install ipywidgets==7.7.2 jupyterlab-widgets==1.1.1 widgetsnbextension==3.6.1 

to pin dependencies exactly to ipywidgets 7.

[jasongrout]

Drop support for mapping types as selection options (#2679, #1958)

We just reverted this change in ipywidgets 8.0.2 (just released) to be backwards compatible with ipywidgets 7, so your code that creates a dropdown from a dictionary should continue working in ipywidgets 8.0.2.