[Bug]: inconsistent colorbar pad for `ImageGrid` with `cbar_mode="single"`

[mathause]

Bug summary

ImageGrid with cbar_mode="single" adds the axes_pad between the axes and the colorbar if the colorbar is to the left or at the bottom. That seems inconsistent and unnecessary as the space can be controlled with cbar_pad.

Code for reproduction

import matplotlib.pyplot as plt
from mpl_toolkits.axes_grid1 import ImageGrid


fig = plt.figure(figsize=(6, 6))

subfigs = fig.subfigures(2, 2, wspace=0.07, hspace=0.1, linewidth=1)
figs = subfigs.flatten()

locations = ["left", "bottom", "right", "top"]

for fig, location in zip(figs, locations):

    fig.set_edgecolor("r")

    axgr = ImageGrid(
        fig,
        111,
        nrows_ncols=(1, 1),
        axes_pad=1. / 2.54,
        cbar_mode="single",
        cbar_location=location,
        cbar_pad=0. / 2.54,
    )
    ax = axgr.axes_all[0]
    ax.text(0.5, 0.5, f"{location=}", transform=ax.transAxes, va="center", ha="center")
    ax.set(xticks=[], yticks=[])
    
    cbax = axgr.cbar_axes[0]
    cbax.set(xticks=[], yticks=[])

    plt.savefig("cbar_mode_single.png")

Actual outcome

Expected outcome

I expect all colorbars to hug the axes.

Additional information

• As mentioned this only happens for cbar_mode="single" and cbar_location "left" or "bottom"
• This behavior seems to have been around since forever.
• There is the cbar_pad param which can be used to control the distance between axes and colorbar
• Note that both cbar_pad and axes_pad are added between the axes and the colorbar
• I am happy to open to open a PR, but don't see a way to deprecate this gracefully

---

• the space is added in these lines respectively:
  

  matplotlib/lib/mpl_toolkits/axes_grid1/axes_grid.py

  Lines 442 to 443 in 479bd7a

  	if h:
  	h.append(self._horiz_pad_size)

  
  

  matplotlib/lib/mpl_toolkits/axes_grid1/axes_grid.py

  Lines 471 to 472 in 479bd7a

  	if v:
  	v.append(self._vert_pad_size)

• This is tested in
  

  matplotlib/lib/mpl_toolkits/axes_grid1/tests/test_axes_grid1.py

  Line 420 in 479bd7a

  @image_comparison(['image_grid_single_bottom_label_mode_1.png'], style='mpl20',

• and the image: https://github.com/matplotlib/matplotlib/blob/479bd7a0cad2b77ef522c1dd6d65080a4da62b4f/lib/mpl_toolkits/axes_grid1/tests/baseline_images/test_axes_grid1/image_grid_single_bottom_label_mode_1.png

Operating system

No response

Matplotlib Version

3.8.4

Matplotlib Backend

No response

Python version

No response

Jupyter version

No response

Installation

conda

[anntzer]

This seems like a clear bug with no obvious backcompat route; I would suggest changing to not including axes_pad when positioning the colorbar and having a clear note in the API changelog.