Repeated warning about fc-list

[astrofrog]

Since Matplotlib 1.5.1, I see a warning (twice) each time I instantiate a FontManager():

In [2]: from matplotlib.font_manager import FontManager

In [3]: FontManager()
/Users/tom/miniconda3/envs/production/lib/python3.4/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')
/Users/tom/miniconda3/envs/production/lib/python3.4/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')
Out[3]: <matplotlib.font_manager.FontManager at 0x1057b6630>

In [4]: FontManager()
/Users/tom/miniconda3/envs/production/lib/python3.4/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')
/Users/tom/miniconda3/envs/production/lib/python3.4/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')
Out[4]: <matplotlib.font_manager.FontManager at 0x1064d4ba8>

This did not happen with 1.5.0. There's two issues here - the warning is repeated twice at each call, which indicates that the command is being run twice (inefficient), and the warning is being shown at all - why does fc-list need to be run every time FontManager is run? Even if there is a good reason for it to run, I'm not sure if a warning is necessary?

[jenshnielsen]

The only change in 1.5.1 is the addition of the warning in #5640 so there should be no performance difference. It's called twice to find both ttf and afm fonts. Not sure there is a good way to do that with just one call.

Internally matplotlib pickles the fontmanager and only ever runs this code if there is no font cache installed so this should not be seen on other than the first import of pyplot. The warning was added because that call may be really slow on a system with many fonts installed see #5640

In the long term there are plans to significantly rewrite the font handling see notes in #5414

[larrybradley]

This warning is triggering travis-ci failures for Sphinx builds for packages that use matplotlib in documentation (e.g. https://travis-ci.org/astropy/photutils/jobs/101686421#L521).

[mdboom]

This warning is triggering travis-ci failures for Sphinx builds for packages that use matplotlib in documentation (e.g. https://travis-ci.org/astropy/photutils/jobs/101686421#L521).

Hmmm... That's an interesting outcome of all this. We could suppress the warning if sphinx is in sys.modules. (Yuck). I am starting to wonder if this warning was worth the trouble. I'd love to get #5414 finished at some point and address this once and for all (by eliminating matplotlib's own font cache altogether).

[tacaswell]

I was always ambivalent to it and would be fine with removing it again. The strongest argument in favor of it is users having short attention spans which I do not really find persuasive.

warnings.warn seems like the wrong mechanism for providing user feed back here, there is nothing that can be done about this and there is no change to the user code that can suppress this warning. We probably should be using logging of some sort (but adding std logging is a huge task which is even more thankless than most).

[jenshnielsen]

What about just changing it to a print? The main point is to provide interactive feedback to users when importing pyplot that would be lost if we used logging.

[fmaussion]

On our travis tests we get the message at each build. On which kind of architecture will the cache take so long that it requires a message?

[WeatherGod]

The problem with "print" is that that can't be filtered at all. A warning
can at least be filtered if an application developer wants it to be
filtered.

As for what kinds of systems this can take so long, basically any system
with Adobe Illustrator installed. It can take several minutes to create the
initial cache.

On Fri, Jan 15, 2016 at 6:00 AM, Fabien Maussion notifications@github.com
wrote:

On our travis tests we get the message at each build. On which kind of
architecture will the cache take so long that it requires a message?

—
Reply to this email directly or view it on GitHub
#5836 (comment)
.

[astrofrog]

A somewhat crazy idea, but what about emitting the warning only if the process takes more than a certain time? For example:

import time
import warnings
import subprocess
import multiprocessing as mp

def delayed_warning():
    time.sleep(5)
    warnings.warn('Matplotlib is building the font cache using fc-list. '
                  'This may take a moment.')

proc = mp.Process(target=delayed_warning)
proc.start()
subprocess.call('fc-list')
proc.terminate()

[mdboom]

Not a crazy idea, if it's robust enough.

Note that all of this will be wasted effort following #5414, where we just call the font-config API directly and don't have to copy all of its contents into our own cache.

[dereknheiley]

not the prettiest work around, but i silenced the warnings for the time being (although the small startup delay is still present) with the following code, maybe a better filter could be used to allow other warnings

import warnings;
with warnings.catch_warnings():
    warnings.simplefilter("ignore"); 
    import matplotlib.pyplot as plt

[tacaswell]

@derekneil try removing everything the mpl cache, you should only have seen this once on the first import....

[dereknheiley]

@tacaswell i did try removing everything in the cache, but it happens every time on my system for some reason

[catubc]

I ran the python code w. sudo and it cured it...my guess was that there wasn't permission to write that table... good luck!

[tacaswell]

@catubc You should sort out what file you do not have permission to write as a user and fix the permissions on that file. Running python with sudo is almost always a bad idea.

[fmder]

I got this issue after upgrading matplotlib and clearing the cache solved my issue!

[mpenkov]

@derekneil @fmder +1 "rm -rf ~/.matplotlib" solved the problem for me too

[sebhahn]

For me it was rm -rf ~/.cache/matplotlib

[eakbas]

rm -rf ~/.cache/matplotlib solved the problem for me.

[matthewhoffman]

rm -rf ~/.matplotlib/fontList.cache worked for me.

[grantjenks]

rm -rf ~/.matplotlib worked for me.

[rmjarvis]

I do still believe that this should probably not be a warning but probably should be more at the logging.INFO level (since it's not something actionable on the part of the user). But if there's otherwise agreement this should be a warning, feel free to close this!

I strongly agree with @astrofrog about this. This is broken behavior, as far as I'm concerned. Warnings should be reserved for things that are either likely to be user error or where the code's behavior is probably not what the user is expecting. And in the latter case, there should be an easy way to suppress the warning if it really is what the user meant.

e.g. The warning emitted by matplotlib.use('Agg') is a good exemplar. It warns if the use statement happens too late to be effective, tells you how to make it effective, and also gives an option (warn=False) to suppress the warning.

The warning about fc-list, OTOH, is not actionable. And in many cases (e.g. imports by dependencies) it is not suppressible. And with the rising popularity of Travis CI (and others), we get a clean system each time, so people are seeing it a lot. It adds quite a lot of noise to our test output, especially with the warning being emitted multiple times, not just once as it should.

For my case, none of the above suggestions for Travis have worked. My solution has been to use matplotlib==1.5.0, which I now consider the latest working version of matplotlib. Please reopen this issue and fix this bug.

FWIW, I thought the suggestion by @astrofrog to only emit the warning if building the cache takes more than 5 seconds or so was a good one. This would solve most of the cases where this behavior is seen as a bug and preserve the cases where people find it useful. And figuring out why it is emitting the warning multiple times and fixing that would be nice as well.

[efiring]

I agree; we need to figure out a better way to handle this.

Also, as @astrofrog noted originally, the warning is emitted twice, which in itself indicates there is a real bug in the code.

[efiring]

Not a bug, but an inefficiency:

• The warning is in get_fontconfig_fonts(), right before it runs fc-list.
• get_fontconfig_fonts() is called once by getSystemFonts().
• But getSystemFonts() is called twice by FontManager.__init__(), first looking for ttf, then for afm.

[tacaswell]

Being emitted twice is correct behavior as we are looking for two types of fonts, maybe that should be included in the warning.

For my case, none of the above suggestions for Travis have worked. My solution has been to use matplotlib==1.5.0, which I now consider the latest working version of matplotlib. Please reopen this issue and fix this bug.

@rmjarvis I find it difficult to take this sort of over-the-top rhetoric seriously.

Almost all of the CI services let you keep a persistent cache between runs, push ~/.matplotlib in that cache and you will only see the warning occasionally. Failing that, run a step in your CI that just imports matplotlib with the warning suppressed before you run your main tests.

mpl sits at a weird edge between a library and a user interface. Providing some sort of feedback to users who hit surprisingly long import times (my understanding is that if you have all the adobe fonts installed this can take minutes) is important.

The problem with moving to a logging based solution is that it brings with it a tremendous amount of overhead (which mpl is currently using none of) and if it is not very noisy by default it will miss the target audience for this warning.

If there is a non-thread based way to get a 5s delay I am 👍 on that, but bringing threads in this seems like complexity overkill.

I think this should be closed again.

[efiring]

On 2016/11/29 4:53 AM, Thomas A Caswell wrote: Being emitted twice is correct behavior as we are looking for two types of fonts, maybe that should be included in the warning.

I don't agree. First, even though we are looking for two types of fonts, this does not require running fc-list twice, because as far as I can see the calls are identical, returning the same list. Second, because this is a very unusual warning in that it is just telling the user to be patient. There is no benefit at all in sending that message twice.

[astrofrog]

@tacaswell @efiring - see #7532

[NelleV]

This is annoying, but not release critical. Bumping this to 2.0.1, as we almost have a patch ready for that one (so it'll probably go in for 2.0).

[anntzer]

Closed by #7596.

[ghost]

For those arriving after experiencing this problem with docker specifically, I added the below to my Dockerfile:

RUN python -c "import matplotlib.pyplot"

That way it will create the cache when you first build the image, but then should not warn anymore on subsequent runs.

[empty16]

rm -rf ~/.cache/matplotlib
Remove the cache works for me, too. Thanks, you guys.

[jklymak]

It also seems this could be moved from warnings.warn to logging.info when #9313 gets merged...