nbagg backend: multiple execution of same cell with plots

[tritemio]

When using matplotlib in the Jupyter Notebook with the notebook (or nbagg) backend, multiple re-executions of of a cell (containing a plot) cause some problems.

1. When using the procedural API: after the second execution of the same cell no figure is displayed (until the next close() call).
2. When using the OO API: after each execution, the old fig and ax objects are overwritten, a new figure is created and the RAM is filled with invisible figures. Even plt.close('all') does not free any RAM.

Re-executing a cell to tweak or add elements to a plot is a very common use case during interactive sessions.

This issue was first mentioned in #4456.

[tacaswell]

Both of these behavior make sense and seem 'correct', even if they are they are very annoying.

In the first case, the figure is only shown in the output where it gets created. Subsequent calls re-use the same figure/axes, but because they are not created in those cells so it does not get shown, but it over writes the output cell where the figure used to be shown, hence no figure is shown, but it keeps getting updated.

In the second case, I suspect what is going on is that the output cells are keeping a reference to the objects, hence there is always a ref to them and they never get garbage collected.

A way to manage the first case is to create the figure and axes in one cell and then plot to from another cell. Not entirely sure how to manage the second. Weak-refs might work, but seems way too complicated for this use case.

cc @ellisonbg We probably should have some input from the IPython devs on how to manage this bit of state? Is there a way for a figure to tell if it is displayed on the page or not? Automatically redisplay a live figure which are about to be replaced by a new output?

[ellisonbg]

I think that transitioning the nbagg backend to using widgets would help with this. The widgets to carefully track their lifecycle in the kernel and frontend and I would expect that you will have to build this up with the comm layer you are using now.

However, we do sometimes run into garbage collection issues when notebook cells containing widgets are rerun. @jdfreder may have some ideas on this and will definitely know the details better than I.

[jdfreder]

Knowing if output exists, let alone specific output, before rerunning a cells isn't something our API provides. However, that doesn't mean you can't use Javascript to figure out yourself. Like @ellisonbg mentions, the widgets handle this type of thing very well. The reason being, a instance of a widget lives notebook level, instead of cell level. Widget views are what are rendered at the cell level and render the contents of the notebook level widget model.

[tacaswell]

This should have been closed by #4456 which implemented #4841 which came from a conversation with @minrk about how to do that.