PySAL 2.1.0 changing matplotlib backend (Jupyter)

[jreades]

I've encountered an issue with PySAL 2.1.0 (possibly conda specific?) in which importing pysal into a Jupyter notebook changes the matplotlib rendering back-end from whatever I specify (TkAgg) to agg, which doesn't work and yields the error:

ipykernel_launcher.py:12: UserWarning: Matplotlib is currently using agg, which is a non-GUI backend, so cannot show the figure.
   if sys.path[0] == '': 

The attached screenshot shows this behaviour in action in a notebook (I've also attached the notebook for testing purposes, though you'll need to change the extension back to .ipynb).

I've confirmed that this issue is specific to 2.1.0 by downgrading to 2.0.0 and the issue disappears (see second attached screenshot). Perhaps this is something specific to Jupyter/JupyterLab but the fact that downgrading PySAL makes the problem go away leaves me wondering...

jon

P.S. Can supply a full Conda YAML file if necessary, but this should be easily reproducible if it's actually something specific to PySAL 2.1.0.

PySAL Test.txt

[darribas]

Could this be done through splot @slumnitz ?

[ljwolf]

Hmm.... I don't get this in a fresh test environment when I isolate the packages in question:

That said, I do see exactly one match for .use(, and that's in pysal/model/spvcm/tests/test_trace.py... I don't think that ever runs, but it looks like I'm wrong if your site installation resets when importing regression props.

[slumnitz]

Could this be done through splot @slumnitz ?

I don't think so, I don't specify the backend within the package, just for travis testing, but separately. And also splot has not changed much in functionality from 2.0.0 to 2.1.0.

[jreades]

I’ll try reconstructing in two ways to see if I can narrow it down: 1. A minimal Jupyter + PySAL installation in anaconda 2. Messing about with the fuller install on which this problem appeared (will attach YAML file when back in front of my machine). Could it be something from testing percolating through to the anaconda install (I’m using conda-forge) and those somehow becoming the default configuration? Jon

…

-- m: 07976987392 e: jon@reades.com s: jreades

On 9 Aug 2019, 17:29 +0100, Stefanie Lumnitz ***@***.***>, wrote: > Could this be done through splot @slumnitz ? I don't think so, I don't specify the backend within the package, just for travis testing, but separately. And also splot has not changed much in functionality from 2.0.0 to 2.1.0. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[darribas]

I wonder how matplotlib changes by moving up to 2.1? I have not encountered this issue either building the Docker containers (which is pretty much the only way I install packages these days...).

On this, it might be useful to provide a full conda list dump from the install on 2.0 and 2.1 maybe?

[jreades]

As you know @darribas, I've been trying to switch over to Docker using your GDS image: I am seeing the same behaviour in that with my GSA (Geocomputation and Spatial Analysis) build on top of the gds_py stack. After importing PySAL the backend has suddenly changed from module://ipympl.backend_nbagg to agg. If I skip this cell and proceed to a geopandas cell then I get the interactive widgets in jupyterlab, but if I run this cell then I just get <matplotlib...>.

[slumnitz]

Maybe it is a new or changed dependency that is causing the problem when importing PySA:, not PySAL itself. @jreades if you are able to provide a list of all Conda dependencies that change via version number, or are new when installing PySAL 2.1.0 in your development environment, I'd be happy to help check these packages.

[jreades]

Will do! Possibly tomorrow as 10:30pm here :-)

Yes, a dependency issue seems reasonable, but wondering why my docker build on top of Dani's is triggering the error...

[sjsrey]

I just did a fresh install of gds_env and I can't get passed the first cell of this example:

So I'm wondering if you have some cross-environment stuff going on? Are you running this in a container?

[jreades]

Ah sorry, I'd extended the environment with a new container as we use a few modules at King's that Dani doesn't at Liverpool; up to that point I'd assumed that it was something Mac-specific but the container is Ubuntu which is why I was posting an update.

Taking @slumnitz's suggestion on board I went back to running conda directly on my Mac and created two fresh environments which differ only in having either PySAL 2.0.0 or 'latest' (i.e. 2.1.0) specified in the YAML file. The same issue recurs so the difference must be in either PySAL or one of the dependencies that changes between 2.0.0 and 2.1.0.

I've diffed the output of conda list and, perhaps helpfully, the list is pretty small (gsa2019alt2==2.0.0; gsa2019alt==2.1.0):

diff pysal2.0.txt pysal2.1.txt
1c1
< # packages in environment at /anaconda3/envs/gsa2019alt2:
---
> # packages in environment at /anaconda3/envs/gsa2019alt:
35a36
> deprecated                1.2.6                      py_0    conda-forge
168c169
< pysal                     2.0.0                      py_0    conda-forge
---
> pysal                     2.1.0                      py_0    conda-forge
205a207
> tqdm                      4.33.0                     py_0    conda-forge
214a217
> wrapt                     1.11.2           py36h01d97ff_0    conda-forge

Outputs from conda list attached as text files. Screengrabs just confirming the issue is still present also attached.
pysal2.0.txt
pysal2.1.txt

[darribas]

@sjsrey what is the %matplotlib widget? I've never seen it hence not tested it (as it's apparent...). I always do inline and that's worked so far. Happy to debug this one to see what's behind. Serge, maybe go ahead and open an issue on darribas/gds_env?

[darribas]

OK, I think the widget issue has to do with ipyml not being installed:

https://github.com/matplotlib/jupyter-matplotlib

A quick check confirms this:

liverbird:~ dani$ docker run -ti --rm darribas/gds:3.0 start.sh
Executing the command: bash
jovyan@b037b3136079:~$ conda list | grep ipyml
jovyan@b037b3136079:~$ 

Is this something people use commonly? If so I could install in on the next version...

[jreades]

@darribas that’s my code/fault, not @sjsrey's as it was stuff that had percolated through the work I was doing to modify your GDS container for use at King's — it gives you back interactive plots in jupyterlab if you install ipympl and the relevant jupyterlab extensions. I’ve reverted to %matplotlib inline in my subsequent tests (see my previous post above on this) to ensure that it wasn’t an issue [it wasn’t].

I’m inclined towards the dependency problem as being the most likely source of the issue, but the diff output suggests that it can only be coming from a couple of places and it’s not obvious to me how this would happen…

Of the libraries that I found to be different between installing PySAL 2.0.0 and 2.1.0, the only one that makes explicit reference to matplotlib (that I could find) was tqdm in the tqdm_gui. However, after poking around in tqdm I found _tqdm_notebook.py and a lot of references to IPython and ipywidgets and so on. I’m now wondering if perhaps jupyterlab isn’t supported so the issue is tqdm > pysal 2.1.0 > jupyterlab?

tqdm is referenced from explore/segregation/inference/inferences_wrappers.py, model/spvcm/abstracts.py, model/mgwr/gwr.py and model/mgwr/search.py.

Bizarrely, one of the developers of this package is @casperdcl who is currently working with @darribas and me on a cool Docker project! If you agree that my investigation has turned up the likely ‘culprit’ then I’ll get in touch with Casper.

[casperdcl]

Given that you just @ mentioned me, you're already in touch :D

[ljwolf]

I still can't replicate this with either development head, the pip install'd package, or the source release on pypi. Can you distribute the docker where this occurs?

For instance, using the source from pypi, this notebook (with #%matplotlib inline comment reverted) never gives agg as a backend in either jupyterlab or jupyter notebook (though, those probably wouldn't differ, I hope 🙃). I'm using jupyterlab & pysal from pypi. The conda list is here.

# In[1]:
import matplotlib
# %matplotlib inline
matplotlib.get_backend()

# In[2]:
import pysal
pysal.__file__

# In[3]:
matplotlib.get_backend()

# In[4]:
from pysal import explore

# In[5]:
matplotlib.get_backend()

[ljwolf]

Like, by distribute the docker image, I mean freeze & publish it in the broken state so we can check it out.

[casperdcl]

P.S. @jreades in the unlikely event that you guys conclude this is an issue with tqdm, feel free to add to my workload https://github.com/tqdm/tqdm/issues. Or even better, open a PR https://github.com/tqdm/tqdm/pulls. Also if anyone is feeling generous feel free to help me out on unrelated issues ;) (I can't offer payment because tqdm isn't funded)

[jreades]

I've got @darribas in town from the end of next week -- perhaps we can investigate further then...

[knaaptime]

I'm 99% sure this is happening because of pandana, which is a segregation dependency. In pysal 2.2, pandana isn't imported until you call the function that requires it so this should no longer happen on import pysal as long as you're >=2.2