New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

This call to matplotlib.use() has no effect because the backend has already #68

Closed
iweey opened this Issue Nov 12, 2017 · 13 comments

Comments

Projects
None yet
7 participants
@iweey

iweey commented Nov 12, 2017

/home/flash1/work/software/python/anaconda2/lib/python2.7/site-packages/pandas_profiling/base.py:20: UserWarning:
This call to matplotlib.use() has no effect because the backend has already
been chosen; matplotlib.use() must be called before pylab, matplotlib.pyplot,
or matplotlib.backends is imported for the first time.

The backend was originally set to 'module://ipykernel.pylab.backend_inline' by the following code:
File "/home/flash1/work/software/python/anaconda2/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/home/flash1/work/software/python/anaconda2/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/flash1/work/software/python/anaconda2/lib/python2.7/site-packages/ipykernel_launcher.py", line 16, in
app.launch_new_instance()
File "/home/flash1/work/software/python/anaconda2/lib/python2.7/site-packages/traitlets/config/application.py", line 658, in launch_instance
app.start()
File "/home/flash1/work/software/python/anaconda2/lib/python2.7/site-packages/ipykernel/kernelapp.py", line 477, in start
ioloop.IOLoop.instance().start()
File "/home/flash1/work/software/python/anaconda2/lib/python2.7/site-packages/zmq/eventloop/ioloop.py", line 177, in start
super(ZMQIOLoop, self).start()
File "/home/flash1/work/software/python/anaconda2/lib/python2.7/site-packages/tornado/ioloop.py", line 888, in start
handler_func(fd_obj, events)
File "/home/flash1/work/software/python/anaconda2/lib/python2.7/site-packages/tornado/stack_context.py", line 277, in null_wrapper
return fn(*args, **kwargs)
File "/home/flash1/work/software/python/anaconda2/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 440, in _handle_events
self._handle_recv()
File "/home/flash1/work/software/python/anaconda2/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 472, in _handle_recv
self._run_callback(callback, msg)
File "/home/flash1/work/software/python/anaconda2/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 414, in _run_callback
callback(*args, **kwargs)
File "/home/flash1/work/software/python/anaconda2/lib/python2.7/site-packages/tornado/stack_context.py", line 277, in null_wrapper
return fn(*args, **kwargs)
File "/home/flash1/work/software/python/anaconda2/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 283, in dispatcher
return self.dispatch_shell(stream, msg)
File "/home/flash1/work/software/python/anaconda2/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 235, in dispatch_shell
handler(stream, idents, msg)
File "/home/flash1/work/software/python/anaconda2/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 399, in execute_request
user_expressions, allow_stdin)
File "/home/flash1/work/software/python/anaconda2/lib/python2.7/site-packages/ipykernel/ipkernel.py", line 196, in do_execute
res = shell.run_cell(code, store_history=store_history, silent=silent)
File "/home/flash1/work/software/python/anaconda2/lib/python2.7/site-packages/ipykernel/zmqshell.py", line 533, in run_cell
return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
File "/home/flash1/work/software/python/anaconda2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2718, in run_cell
interactivity=interactivity, compiler=compiler, result=result)
File "/home/flash1/work/software/python/anaconda2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2822, in run_ast_nodes
if self.run_code(code, result):
File "/home/flash1/work/software/python/anaconda2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2882, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 8, in
import matplotlib.pyplot as plt
File "/home/flash1/work/software/python/anaconda2/lib/python2.7/site-packages/matplotlib/pyplot.py", line 69, in
from matplotlib.backends import pylab_setup
File "/home/flash1/work/software/python/anaconda2/lib/python2.7/site-packages/matplotlib/backends/init.py", line 14, in
line for line in traceback.format_stack()

matplotlib.use('Agg')

@romainx romainx added the invalid label Nov 13, 2017

@romainx

This comment has been minimized.

Collaborator

romainx commented Nov 13, 2017

Hello,

Could you please explain your use case and take the time to format the issue?

Many thanks

@ianozsvald

This comment has been minimized.

ianozsvald commented Nov 15, 2017

+1 I have this issue. I've narrowed it down to:

import seaborn
import pandas_profiling
# only these two lines will generate:
/home/ian/anaconda3/envs/data_science_delivered/lib/python3.6/site-packages/pandas_profiling/base.py:20: UserWarning: 
This call to matplotlib.use() has no effect because the backend has already
been chosen; matplotlib.use() must be called *before* pylab, matplotlib.pyplot,
or matplotlib.backends is imported for the first time.

The backend was *originally* set to 'module://ipykernel.pylab.backend_inline' by the following code:
  File "/home/ian/anaconda3/envs/data_science_delivered/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
...
    from matplotlib.backends import pylab_setup
  File "/home/ian/anaconda3/envs/data_science_delivered/lib/python3.6/site-packages/matplotlib/backends/__init__.py", line 14, in <module>
    line for line in traceback.format_stack()

  matplotlib.use('Agg')

My versions (using watermark):

2017-11-15 

CPython 3.6.2
IPython 6.1.0

numpy 1.13.3
matplotlib 2.1.0
seaborn 0.8.0

compiler   : GCC 4.8.2 20140120 (Red Hat 4.8.2-15)
system     : Linux
release    : 4.9.45-040945-generic
machine    : x86_64
processor  : x86_64
CPU cores  : 8
interpreter: 64bit

If I invert those lines (so seaborn is second) then the warning goes away, if I do either combo in my real import list then I still get the huge warning:

import datetime
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import pandas_profiling
import seaborn as sns
import tqdm

import eli5 # https://github.com/TeamHG-Memex/eli5 via conda
import lime # https://github.com/marcotcr/lime via PyPI (note this was built using MASTER at time-of-creation)
import yellowbrick # https://github.com/DistrictDataLabs/yellowbrick/ via conda
from eli5 import show_weights
from eli5 import show_prediction

import lime.lime_tabular
from matplotlib import pyplot as plt
%matplotlib inline

Why does base.py set matplotlib.use('Agg')? Doesn't that depend on the user's requirements?

@ianozsvald

This comment has been minimized.

ianozsvald commented Nov 23, 2017

I'll note that if comment-out matplotlib.use('Agg') in base.py - this problem goes away. Maybe that's the easy fix?

I think in my Notebook that the line is redundant as I'm using the Jupyter backend, my first few lines are:

import matplotlib
print(matplotlib.get_backend())
-> module://ipykernel.pylab.backend_inline

and when I subsequently import pandas_profiling the backend doesn't change (with the Agg line enabled in base.py).

@iweey are you using a Jupyter Notebook too? Your error looks to be the same as mine?

@iweey

This comment has been minimized.

iweey commented Nov 24, 2017

@ianozsvald yes,Jupyter Notebook meet this issue

@ianozsvald

This comment has been minimized.

ianozsvald commented Dec 8, 2017

I'll note that I've ignored this warning using:

import warnings
warnings.filterwarnings('ignore') 

over here: https://github.com/ianozsvald/simple_generative_insurance_model/blob/master/simple_generative_insurance_model.ipynb

This is a blunt-stick approach, commenting out matplotlib.use('Agg') in base.py works fine too and I do wonder if that needs to be set by pandas_profiling as it feels user-machine-specific?

@romainx

This comment has been minimized.

Collaborator

romainx commented Dec 10, 2017

Hello,

Thanks to all for your comments ! Here is a link to the reference documentation on backend.

If you use the use() function, this must be done before importing matplotlib.pyplot. Calling use() after pyplot has been imported will have no effect. Using use() will require changes in your code if users want to use a different backend. Therefore, you should avoid explicitly calling use() unless absolutely necessary.

So you may be right, the explicit call to use may be useless. However my guess is that it has been made to ensure that figures in the report are generated in PNG format.
I will dig in to find if it is necessary in all contexts, I have also to check the behavior when the tool is run outside Jupyter (call to profile_csv.py). And if so, maybe there is a better way to handle it when it's already set.

Best.

@romainx romainx added the question label Dec 10, 2017

@romainx romainx self-assigned this Dec 10, 2017

@conradoqg

This comment has been minimized.

Contributor

conradoqg commented Dec 31, 2017

Hey guys!

I commented the matplotlib.use('Agg') line and tested the profile_csv.py and jupyter notebook. Everything seems to work without it. I think we should remove it, I can do an PR if necessary.

Best.

@romainx romainx closed this in 3448d1c Jan 1, 2018

romainx added a commit that referenced this issue Jan 1, 2018

Merge pull request #80 from romainx/master
Fix #68, this call is not needed and brings side effects in some use cases.
@romainx

This comment has been minimized.

Collaborator

romainx commented Jan 1, 2018

Hello,

Many thanks for the test, I have commented the line and merged the change #80.
Really appreciate your contribution.

romainx added a commit to romainx/pandas-profiling that referenced this issue Jan 1, 2018

Several improvements:
* pandas-profiling#68: Backport the fix
* pandas-profiling#65: Describe module refactored
@ianozsvald

This comment has been minimized.

ianozsvald commented Jan 11, 2018

Hi @romainx, I don't think that this fix is right - having upgraded to 1.4.1 I still have the same issue as reported above. Specifically in a fresh Notebook, I see:

import matplotlib
matplotlib.get_backend()
-> 'module://ipykernel.pylab.backend_inline'

The proposed fix ( https://github.com/pandas-profiling/pandas-profiling/blob/master/pandas_profiling/plot.py#L12 ) checks if the backed == agg and if not, it forces to agg.

If I comment out the if statement (so the backend isn't changed) then the Warning goes away.

I don't know what the case is if you are just running the tests, but forcing Agg even though the user already has another renderer isn't the right way to go. Is the renderer something like a blank string when the tests run?

@aagostini-tada

This comment has been minimized.

aagostini-tada commented Feb 16, 2018

I've seen the same issue

@conradoqg

This comment has been minimized.

Contributor

conradoqg commented Feb 27, 2018

I'm gonna check this.

@sleitner

This comment has been minimized.

sleitner commented Sep 24, 2018

Was any progress made here? The unit tests hang for me, and I traced it back to this backend issue, when my default interactive matplotlib backend is used (TkAgg -- from my personal matplotlibrc) and the multiprocessing pool_size>1.

I agree that it's heavy handed to force the user to matplotlib.use('Agg') even when pool_size=1, but I would have appreciated a warning not to use an interactive backend, especially given that the default pool_size in describe.py is naturally going to be >1 for most everyone.

sleitner added a commit to sleitner/pandas-profiling that referenced this issue Sep 24, 2018

Warn about interactive matplotlib causing hangs
Another shot at issue pandas-profiling#68 where matplotlib and multiprocessing don't
play nice.
Also moves plotting functions into plot.py and simplifies mpl style
use (tests compatible with mpl 1.4 and 3).

sleitner added a commit to sleitner/pandas-profiling that referenced this issue Sep 24, 2018

Warn about interactive matplotlib causing hangs
Another shot at issue pandas-profiling#68 where matplotlib and multiprocessing don't
play nice.
Also moves plotting functions into plot.py and simplifies mpl style
use (tests compatible with mpl 1.4 and 3).
@anilsomisetty

This comment has been minimized.

anilsomisetty commented Oct 27, 2018

I am getting an error like this Can someone help me with this

utils/visualization_utils.py:27: UserWarning:
This call to matplotlib.use() has no effect because the backend has already
been chosen; matplotlib.use() must be called before pylab, matplotlib.pyplot,
or matplotlib.backends is imported for the first time.

The backend was originally set to 'module://ipykernel.pylab.backend_inline' by the following code:
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/anil/.virtualenvs/ml1/lib/python2.7/site-packages/ipykernel_launcher.py", line 16, in
app.launch_new_instance()
File "/home/anil/.virtualenvs/ml1/local/lib/python2.7/site-packages/traitlets/config/application.py", line 658, in launch_instance
app.start()
File "/home/anil/.virtualenvs/ml1/local/lib/python2.7/site-packages/ipykernel/kernelapp.py", line 499, in start
self.io_loop.start()
File "/home/anil/.virtualenvs/ml1/local/lib/python2.7/site-packages/tornado/ioloop.py", line 1017, in start
self._run_callback(self._callbacks.popleft())
File "/home/anil/.virtualenvs/ml1/local/lib/python2.7/site-packages/tornado/ioloop.py", line 758, in _run_callback
ret = callback()
File "/home/anil/.virtualenvs/ml1/local/lib/python2.7/site-packages/tornado/stack_context.py", line 300, in null_wrapper
return fn(*args, **kwargs)
File "/home/anil/.virtualenvs/ml1/local/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 536, in
self.io_loop.add_callback(lambda : self._handle_events(self.socket, 0))
File "/home/anil/.virtualenvs/ml1/local/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 450, in _handle_events
self._handle_recv()
File "/home/anil/.virtualenvs/ml1/local/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 480, in _handle_recv
self._run_callback(callback, msg)
File "/home/anil/.virtualenvs/ml1/local/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 432, in _run_callback
callback(*args, **kwargs)
File "/home/anil/.virtualenvs/ml1/local/lib/python2.7/site-packages/tornado/stack_context.py", line 300, in null_wrapper
return fn(*args, **kwargs)
File "/home/anil/.virtualenvs/ml1/local/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 283, in dispatcher
return self.dispatch_shell(stream, msg)
File "/home/anil/.virtualenvs/ml1/local/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 233, in dispatch_shell
handler(stream, idents, msg)
File "/home/anil/.virtualenvs/ml1/local/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 399, in execute_request
user_expressions, allow_stdin)
File "/home/anil/.virtualenvs/ml1/local/lib/python2.7/site-packages/ipykernel/ipkernel.py", line 208, in do_execute
res = shell.run_cell(code, store_history=store_history, silent=silent)
File "/home/anil/.virtualenvs/ml1/local/lib/python2.7/site-packages/ipykernel/zmqshell.py", line 537, in run_cell
return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
File "/home/anil/.virtualenvs/ml1/local/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2714, in run_cell
interactivity=interactivity, compiler=compiler, result=result)
File "/home/anil/.virtualenvs/ml1/local/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2824, in run_ast_nodes
if self.run_code(code, result):
File "/home/anil/.virtualenvs/ml1/local/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2878, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 2, in
get_ipython().magic(u'matplotlib inline')
File "/home/anil/.virtualenvs/ml1/local/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2160, in magic
return self.run_line_magic(magic_name, magic_arg_s)
File "/home/anil/.virtualenvs/ml1/local/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2081, in run_line_magic
result = fn(*args,**kwargs)
File "", line 2, in matplotlib
File "/home/anil/.virtualenvs/ml1/local/lib/python2.7/site-packages/IPython/core/magic.py", line 188, in
call = lambda f, *a, **k: f(*a, **k)
File "/home/anil/.virtualenvs/ml1/local/lib/python2.7/site-packages/IPython/core/magics/pylab.py", line 100, in matplotlib
gui, backend = self.shell.enable_matplotlib(args.gui)
File "/home/anil/.virtualenvs/ml1/local/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2966, in enable_matplotlib
pt.activate_matplotlib(backend)
File "/home/anil/.virtualenvs/ml1/local/lib/python2.7/site-packages/IPython/core/pylabtools.py", line 315, in activate_matplotlib
matplotlib.pyplot.switch_backend(backend)
File "/home/anil/.virtualenvs/ml1/local/lib/python2.7/site-packages/matplotlib/pyplot.py", line 231, in switch_backend
matplotlib.use(newbackend, warn=False, force=True)
File "/home/anil/.virtualenvs/ml1/local/lib/python2.7/site-packages/matplotlib/init.py", line 1422, in use
reload(sys.modules['matplotlib.backends'])
File "/home/anil/.virtualenvs/ml1/local/lib/python2.7/site-packages/matplotlib/backends/init.py", line 16, in
line for line in traceback.format_stack()

import matplotlib; matplotlib.use('Agg') # pylint: disable=multiple-statements

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment