DataFrame.hist() does not get along with matplotlib.pyplot.tight_layout()

[vfilimonov]

When tight_layout() is called after DataFrame.hist() it raises AttributeError: 'NoneType' object has no attribute 'is_bbox if any of subplots is empty (number of charts with histograms is smaller than number os subplots).

I.e. the following code:

pd.DataFrame({'a':np.random.randn(100),
              'b':np.random.randn(100),
              }).hist()
plt.tight_layout()

works fine, but this code:

pd.DataFrame({'a':np.random.randn(100),
              'b':np.random.randn(100),
              'c':np.random.randn(100),
              }).hist()
plt.tight_layout()

raises AttributeError.

My versions:
pandas: 0.15.2
numpy: 1.9.1
scipy: 0.14.1
IPython: 2.3.1
matplotlib: 1.4.2

[ericmjl]

Ditto here, I can't seem to figure out what's going on.

[TomAugspurger]

Would either of you want to dig into what's causing the problem?

It seems that

    129         tight_bbox_raw = union([ax.get_tightbbox(renderer) for ax in subplots])

in matplotlib.tight_layout.auto_adjust_subplotpars returns None when it's expected to return a BBox.

[goyodiaz]

Actually, just before the error is raised, subplots is a list of one AxesSubplot instance for which get_visible() returns False. ax.get_tightbbox() returns None for invisible axes and auto_adjust_subplotpars() pass this None to TransformedBbox(), which expects a real Bbox and raises the exception.

Looks like a matplotlib issue to me, or maybe pandas should not be creating that invisible Axes.

A quick workaround would be to add something like this:

if tight_bbox_raw is None:
    continue

But it might have side effects.

[miraculixx]

+1

[ResidentMario]

As of matplotlib 2.0 this now works as expected.

[jreback]

@ResidentMario can you add a test to confirm this behavior (that skips if mpl < 2.0)?

[naught101]

This might be a different cause, but tight_layout is still not playing well with pandas.plot(kind='hist', ...). A plot that I am generating works fine, but as soon as I add a fig.tight_layout() call, I get the following error:

$ scripts/exploratory_analysis/site_analysis.py QC_histogram
/home/naught101/miniconda3/envs/science/lib/python3.5/site-packages/matplotlib/__init__.py:898: MatplotlibDeprecationWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
  mplDeprecation)
Traceback (most recent call last):
  File "scripts/exploratory_analysis/site_analysis.py", line 122, in <module>
    main(args)
  File "scripts/exploratory_analysis/site_analysis.py", line 112, in main
    QC_plot(sites, variables, variant='histogram')
  File "scripts/exploratory_analysis/site_analysis.py", line 94, in QC_plot
    fig.tight_layout()
  File "/home/naught101/miniconda3/envs/science/lib/python3.5/site-packages/matplotlib/figure.py", line 1943, in tight_layout
    rect=rect)
  File "/home/naught101/miniconda3/envs/science/lib/python3.5/site-packages/matplotlib/tight_layout.py", line 345, in get_tight_layout_figure
    pad=pad, h_pad=h_pad, w_pad=w_pad)
  File "/home/naught101/miniconda3/envs/science/lib/python3.5/site-packages/matplotlib/tight_layout.py", line 126, in auto_adjust_subplotpars
    tight_bbox_raw = union([ax.get_tightbbox(renderer) for ax in subplots])
  File "/home/naught101/miniconda3/envs/science/lib/python3.5/site-packages/matplotlib/transforms.py", line 723, in union
    x0 = np.min([bbox.xmin for bbox in bboxes])
  File "/home/naught101/miniconda3/envs/science/lib/python3.5/site-packages/matplotlib/transforms.py", line 723, in <listcomp>
    x0 = np.min([bbox.xmin for bbox in bboxes])
AttributeError: 'NoneType' object has no attribute 'xmin'

I figured this might have been fixed in newer versions, so I updated most things to git versions (which is more painful that I'd have liked, with conda...). Current versions are:

matplotlib                2.0.0               np110py35_1  
numpy                     1.12.0                   py35_0  
pandas                    0.19.2              np112py35_1  
python                    3.5.3                         1  
scipy                     0.16.0             np110py35_p1  [mkl]

[ResidentMario]

😞 I'll try looking for a fix, then, now that I've mucked this issue up.

[ResidentMario]

OK, so the non-working code is:

df.hist()
plt.tight_layout()

I consciously avoid working with plot methods directly, preferring to use the plot accessor methods instead (so df.plot.hist instead of df.hist), so I wasn't aware of the rather important fact that although df.plot.hist and df.hist are the same method internally, they have different default parameter signatures. I think I was fiddling with something else involving histograms at the time, because I just plugged this into .plot.hist and said hey, it works, must have been a mpl 2.0.0 fix:

df.plot.hist()
plt.tight_layout()

TIL these actually result in radically different output! To replicate this bug using the accessor method you have to further specifiy:

df.plot.hist(subplots=True, layout=(2, 2))
plt.tight_layout()

The test I added in the closing commit did use hist, but for some reason in my test DataFrame a 3 became a 2:

df = DataFrame(randn(100, 2))

Which passes, because it creates a (1, 2) layout that works. Ouch. My apologies.

[ResidentMario]

So I haven't quite finished investigating this, but here's what I think so far: the core of pandas plotting methodology is very old, predating the current matplotlib orthodoxy. It looks like pandas uses a lower-level API—adding artists to the canvas—to do what would today be done using something like plt.subplots.

Since it does this in a loop which iterates over the data, this means that in this example case it only does so three times, leaving the last layout element undefined. That axis then has a BBox of None, which tight_layout trips over, which then causes everything else to fail.

This may be reportable as a bug in matplotlib. A band-aid can definitely be made in pandas though. In the long term, pandas defo needs to move on to doing something like plt.subplots(2, 2) instead.

[jreback]

ok reopening

[ResidentMario]

This boils down to a matplotlib bug. This is what pandas is doing internally, which fails:

import matplotlib.pyplot as plt
f, axarr = plt.subplots(2, 2)
axarr[1][1].set_visible(False)
plt.tight_layout()

Going about it a slightly different way, this works:

import matplotlib.pyplot as plt
f, axarr = plt.subplots(2, 2)
axarr[1][1].axis('off')
plt.tight_layout()

[ResidentMario]

This is now fixed downstream in matplotlib as #8255 by PR#8244. Once matplotlib 2.0.1 lands this issue will be fixed and can be closed (use mpl > 2.0.1). For the moment best to leave it open though.

[jreback]

thanks @ResidentMario ok can update the release note at some point. any additional tests needed?

[ResidentMario]

@jreback I would say no, the PR included a regression test for this in the matplotlib test suite.

[jreback]

@ResidentMario the regression test passed though, contrary to the bug fix that was needed, right? (IIRC you did another PR which added a slightly different test which failed though..?)

[jreback]

this was corrected in 03dca96

any reason not to close this issue?

[ResidentMario]

@jreback Wasn't sure what best practice is for telling users to use a version of a downstream library which may or may not be out yet (maybe pandas 0.20.0 will beat matplotlib 2.0.1 to release). But since the update note is already in place, sure this issue can be closed—but, you should remove the errant test first (see the PR above).

[jorisvandenbossche]

Yeah, I saw the typo of 0.2 vs 2.0, and thought to directly make this 2.0.1, because that is more correct.