interact based ipympl plots not working as expected

[stevejpurves]

Describe the bug

context
When thebe is used to render an ipympl plot that has been created using interact, this fails to render as expected. Instead of getting a canvas based render of the output, it seems ot fallback to html.

When manually rendering in an output widget, thebe performs as expected.

The code to produce the plot. is:

%matplotlib widget
import ipywidgets as widgets
import matplotlib.pyplot as plt
import numpy as np      
from ipywidgets import Output, FloatSlider, Layout
from IPython.display import clear_output

def sine_func(x, w, amp):
  return amp*np.sin(w*x)

fig_one, ax_one = plt.subplots(1,1)

x = np.linspace(0,10)

@widgets.interact(w=(0, 4, 0.25), amp=(0, 4, .1))
def update(w = 1, amp = 1):
    plt.figure(fig_one.number)
    plt.clf()
    plt.ylim(-4, 4)
    plt.plot(x, sine_func(x, w, amp))
    plt.show()

Which produces a text/html based plot on in the OutputArea

expectation
I expected to be able to get the same behaviour from interact, interactive_output and manual use of an Output widget. The same notebook displays as expected in JupyterLab

bug
But instead I get a static, non interactive plot.

problem
This is a problem for people doing trying to create interactive figures, as the standard ipympl controls are not available and the plot will not respond to other click and mouse events.

Reproduce the bug

Coming soon .. a link to the notebook

List your environment

No response

[stevejpurves]

It turns out that removing the customized Output widget that was included when switching to the KernelWidgetManager, fixes this particular issue https://github.com/executablebooks/thebe/blob/main/packages/core/src/output.ts

However it also (re)introduces errors when the widget manager is created use of the customized Output was based on similar changes in voila to remove those errors, seems like this is a side effect of those changes - so this could also be affecting voila? (cc @martinRenou)

Investigating to see if a different change to Output could be made to fix both problems

[stevejpurves]

After debugging it's clear that the manager class nor the customized Output widget class are not causing the issue.

After more debugging and another set of eyes from @agoose77 👀 -- removing the plt.show() at the end of the def update(): function gets this working in thebe and jupyterlab.

That's a positive step but also still points to something that thebe in missing - i.e. thebe makes use of juptyerlab's OutputArea and OutputAreaModel directly, immediately assigning the execute request result to the areas model's future -- however, with the additional plt.show() in place this still renders correctly in jupyterlab which it fails in thebe -- so thebe is missing some additional event or message handling that jupyterlab implements elsewhere.

[stevejpurves]

Upgrading to a recent release of @jupyter-widgets/juyterlab-manager means thebe can stop using the customized output area and just use the one from the extension directly. See jupyter-widgets/ipywidgets#3561

[stevejpurves]

update:

when removing plt.show() from the update function in the code in the OP, the figure then works as expected in thebe. However, the plt.show() continues to work in jupyterlab regardless.

In a further attempt to charactize the problem; updating the code to the following where we display an arbitrary widget in the update function:

fig_one, ax_one = plt.subplots(1,1)

x = np.linspace(0,10)

w_rand = FloatSlider(min=0.1, max=5.0, step=0.05, value=0.8, description="RANDOM")

@widgets.interact(w=(0, 4, 0.25), amp=(0, 4, .1))
def update(w = 1, amp = 1):
    plt.figure(fig_one.number)
    plt.clf()
    plt.ylim(-4, 4)
    plt.plot(x, sine_func(x, w, amp))
    display(w_rand)

results in the following figure in thebe

While is jupyterlab the slider renders as expected. This looks like what would happen if the widget was rendered using a rendermime factory that did not have the widget rendered installed i.e. the text/plain mimetype is being rendered.