v1.1.0.rst

What's new in 1.1.0 (??)

These are the changes in pandas 1.1.0. See release for a full changelog including other versions of pandas.

{{ header }}

Enhancements

KeyErrors raised by loc specify missing labels

Previously, if labels were missing for a .loc call, a KeyError was raised stating that this was no longer supported.

Now the error message also includes a list of the missing labels (max 10 items, display width 80 characters). See 34272.

All dtypes can now be converted to StringDtype

Previously, declaring or converting to StringDtype was in general only possible if the data was already only str or nan-like (31204). StringDtype now works in all situations where astype(str) or dtype=str work:

For example, the below now works:

python

ser = pd.Series([1, "abc", np.nan], dtype="string") ser ser[0] pd.Series([1, 2, np.nan], dtype="Int64").astype("string")

Non-monotonic PeriodIndex Partial String Slicing

PeriodIndex now supports partial string slicing for non-monotonic indexes, mirroring DatetimeIndex behavior (31096)

For example:

python

dti = pd.date_range("2014-01-01", periods=30, freq="30D") pi = dti.to_period("D") ser_monotonic = pd.Series(np.arange(30), index=pi) shuffler = list(range(0, 30, 2)) + list(range(1, 31, 2)) ser = ser_monotonic[shuffler] ser

python

ser["2014"] ser.loc["May 2015"]

Comparing two DataFrame or two Series and summarizing the differences

We've added DataFrame.compare and Series.compare for comparing two DataFrame or two Series (30429)

python

df = pd.DataFrame(

{

"col1": ["a", "a", "b", "b", "a"], "col2": [1.0, 2.0, 3.0, np.nan, 5.0], "col3": [1.0, 2.0, 3.0, 4.0, 5.0]

}, columns=["col1", "col2", "col3"],

) df

python

df2 = df.copy() df2.loc[0, 'col1'] = 'c' df2.loc[2, 'col3'] = 4.0 df2

python

df.compare(df2)

See User Guide <merging.compare> for more details.

Allow NA in groupby key

With groupby <groupby.dropna> , we've added a dropna keyword to DataFrame.groupby and Series.groupby in order to allow NA values in group keys. Users can define dropna to False if they want to include NA values in groupby keys. The default is set to True for dropna to keep backwards compatibility (3729)

python

df_list = [[1, 2, 3], [1, None, 4], [2, 1, 3], [1, 2, 2]] df_dropna = pd.DataFrame(df_list, columns=["a", "b", "c"])

df_dropna

python

# Default dropna is set to True, which will exclude NaNs in keys df_dropna.groupby(by=["b"], dropna=True).sum()

# In order to allow NaN in keys, set dropna to False df_dropna.groupby(by=["b"], dropna=False).sum()

The default setting of dropna argument is True which means NA are not included in group keys.

Sorting with keys

We've added a key argument to the DataFrame and Series sorting methods, including DataFrame.sort_values, DataFrame.sort_index, Series.sort_values, and Series.sort_index. The key can be any callable function which is applied column-by-column to each column used for sorting, before sorting is performed (27237). See sort_values with keys <basics.sort_value_key> and sort_index with keys <basics.sort_index_key> for more information.

python

s = pd.Series(['C', 'a', 'B']) s

python

s.sort_values()

Note how this is sorted with capital letters first. If we apply the Series.str.lower method, we get

python

s.sort_values(key=lambda x: x.str.lower())

When applied to a DataFrame, they key is applied per-column to all columns or a subset if by is specified, e.g.

python

df = pd.DataFrame({'a': ['C', 'C', 'a', 'a', 'B', 'B'],

'b': [1, 2, 3, 4, 5, 6]})

df

python

df.sort_values(by=['a'], key=lambda col: col.str.lower())

For more details, see examples and documentation in DataFrame.sort_values, Series.sort_values, and ~DataFrame.sort_index.

Fold argument support in Timestamp constructor

Timestamp: now supports the keyword-only fold argument according to PEP 495 similar to parent datetime.datetime class. It supports both accepting fold as an initialization argument and inferring fold from other constructor arguments (25057, 31338). Support is limited to dateutil timezones as pytz doesn't support fold.

For example:

python

ts = pd.Timestamp("2019-10-27 01:30:00+00:00") ts.fold

python

ts = pd.Timestamp(year=2019, month=10, day=27, hour=1, minute=30,

tz="dateutil/Europe/London", fold=1)

ts

For more on working with fold, see Fold subsection <timeseries.fold> in the user guide.

Parsing timezone-aware format with different timezones in to_datetime

to_datetime now supports parsing formats containing timezone names (%Z) and UTC offsets (%z) from different timezones then converting them to UTC by setting utc=True. This would return a DatetimeIndex with timezone at UTC as opposed to an Index with object dtype if utc=True is not set (32792).

For example:

python

tz_strs = ["2010-01-01 12:00:00 +0100", "2010-01-01 12:00:00 -0100",

"2010-01-01 12:00:00 +0300", "2010-01-01 12:00:00 +0400"]

pd.to_datetime(tz_strs, format='%Y-%m-%d %H:%M:%S %z', utc=True) pd.to_datetime(tz_strs, format='%Y-%m-%d %H:%M:%S %z')

Grouper and resample now supports the arguments origin and offset

Grouper and DataFrame.resample now supports the arguments origin and offset. It let the user control the timestamp on which to adjust the grouping. (31809)

The bins of the grouping are adjusted based on the beginning of the day of the time series starting point. This works well with frequencies that are multiples of a day (like 30D) or that divides a day (like 90s or 1min). But it can create inconsistencies with some frequencies that do not meet this criteria. To change this behavior you can now specify a fixed timestamp with the argument origin.

Two arguments are now deprecated (more information in the documentation of DataFrame.resample):

• base should be replaced by offset.
• loffset should be replaced by directly adding an offset to the index DataFrame after being resampled.

Small example of the use of origin:

python

start, end = '2000-10-01 23:30:00', '2000-10-02 00:30:00' middle = '2000-10-02 00:00:00' rng = pd.date_range(start, end, freq='7min') ts = pd.Series(np.arange(len(rng)) * 3, index=rng) ts

Resample with the default behavior 'start_day' (origin is 2000-10-01 00:00:00):

python

ts.resample('17min').sum() ts.resample('17min', origin='start_day').sum()

Resample using a fixed origin:

python

ts.resample('17min', origin='epoch').sum() ts.resample('17min', origin='2000-01-01').sum()

If needed you can adjust the bins with the argument offset (a Timedelta) that would be added to the default origin.

For a full example, see: timeseries.adjust-the-start-of-the-bins.

fsspec now used for filesystem handling

For reading and writing to filesystems other than local and reading from HTTP(S), the optional dependency fsspec will be used to dispatch operations (33452). This will give unchanged functionality for S3 and GCS storage, which were already supported, but also add support for several other storage implementations such as Azure Datalake and Blob, SSH, FTP, dropbox and github. For docs and capabilities, see the fsspec docs.

The existing capability to interface with S3 and GCS will be unaffected by this change, as fsspec will still bring in the same packages as before.

Other enhancements

• Compatibility with matplotlib 3.3.0 (34850)
• IntegerArray.astype now supports datetime64 dtype (32538)
• IntegerArray now implements the sum operation (33172)
• Added pandas.errors.InvalidIndexError (34570).
• Added DataFrame.value_counts (5377)
• Added a pandas.api.indexers.FixedForwardWindowIndexer class to support forward-looking windows during rolling operations.
• Added a pandas.api.indexers.VariableOffsetWindowIndexer class to support rolling operations with non-fixed offsets (34994)
• ~DataFrame.describe now includes a datetime_is_numeric keyword to control how datetime columns are summarized (30164, 34798)
• ~pandas.io.formats.style.Styler may now render CSS more efficiently where multiple cells have the same styling (30876)
• ~pandas.io.formats.style.Styler.highlight_null now accepts subset argument (31345)
• When writing directly to a sqlite connection DataFrame.to_sql now supports the multi method (29921)
• pandas.errors.OptionError is now exposed in pandas.errors (27553)
• Added api.extensions.ExtensionArray.argmax and api.extensions.ExtensionArray.argmin (24382)
• timedelta_range will now infer a frequency when passed start, stop, and periods (32377)
• Positional slicing on a IntervalIndex now supports slices with step > 1 (31658)
• Series.str now has a fullmatch method that matches a regular expression against the entire string in each row of the Series, similar to re.fullmatch (32806).
• DataFrame.sample will now also allow array-like and BitGenerator objects to be passed to random_state as seeds (32503)
• Index.union will now raise RuntimeWarning for MultiIndex objects if the object inside are unsortable. Pass sort=False to suppress this warning (33015)
• Added Series.dt.isocalendar and DatetimeIndex.isocalendar that returns a DataFrame with year, week, and day calculated according to the ISO 8601 calendar (33206, 34392).
• The DataFrame.to_feather method now supports additional keyword arguments (e.g. to set the compression) that are added in pyarrow 0.17 (33422).
• The cut will now accept parameter ordered with default ordered=True. If ordered=False and no labels are provided, an error will be raised (33141)
• DataFrame.to_csv, DataFrame.to_pickle, and DataFrame.to_json now support passing a dict of compression arguments when using the gzip and bz2 protocols. This can be used to set a custom compression level, e.g., df.to_csv(path, compression={'method': 'gzip', 'compresslevel': 1} (33196)
• melt has gained an ignore_index (default True) argument that, if set to False, prevents the method from dropping the index (17440).
• Series.update now accepts objects that can be coerced to a Series, such as dict and list, mirroring the behavior of DataFrame.update (33215)
• ~pandas.core.groupby.DataFrameGroupBy.transform and ~pandas.core.groupby.DataFrameGroupBy.aggregate have gained engine and engine_kwargs arguments that support executing functions with Numba (32854, 33388)
• ~pandas.core.resample.Resampler.interpolate now supports SciPy interpolation method scipy.interpolate.CubicSpline as method cubicspline (33670)
• ~pandas.core.groupby.DataFrameGroupBy and ~pandas.core.groupby.SeriesGroupBy now implement the sample method for doing random sampling within groups (31775)
• DataFrame.to_numpy now supports the na_value keyword to control the NA sentinel in the output array (33820)
• Added api.extension.ExtensionArray.equals to the extension array interface, similar to Series.equals (27081)
• The minimum supported dta version has increased to 105 in read_stata and ~pandas.io.stata.StataReader (26667).
• ~DataFrame.to_stata supports compression using the compression keyword argument. Compression can either be inferred or explicitly set using a string or a dictionary containing both the method and any additional arguments that are passed to the compression library. Compression was also added to the low-level Stata-file writers ~pandas.io.stata.StataWriter, ~pandas.io.stata.StataWriter117, and ~pandas.io.stata.StataWriterUTF8 (26599).
• HDFStore.put now accepts a track_times parameter. This parameter is passed to the create_table method of PyTables (32682).
• Series.plot and DataFrame.plot now accepts xlabel and ylabel parameters to present labels on x and y axis (9093).
• Made pandas.core.window.rolling.Rolling and pandas.core.window.expanding.Expanding iterable（11704)
• Made option_context a contextlib.ContextDecorator, which allows it to be used as a decorator over an entire function (34253).
• DataFrame.to_csv and Series.to_csv now accept an errors argument (22610)
• ~pandas.core.groupby.DataFrameGroupBy.groupby.transform now allows func to be pad, backfill and cumcount (31269).
• read_json now accepts an nrows parameter. (33916).
• DataFrame.hist, Series.hist, core.groupby.DataFrameGroupBy.hist, and core.groupby.SeriesGroupBy.hist have gained the legend argument. Set to True to show a legend in the histogram. (6279)
• concat and ~DataFrame.append now preserve extension dtypes, for example combining a nullable integer column with a numpy integer column will no longer result in object dtype but preserve the integer dtype (33607, 34339, 34095).
• read_gbq now allows to disable progress bar (33360).
• read_gbq now supports the max_results kwarg from pandas-gbq (34639).
• DataFrame.cov and Series.cov now support a new parameter ddof to support delta degrees of freedom as in the corresponding numpy methods (34611).
• DataFrame.to_html and DataFrame.to_string's col_space parameter now accepts a list or dict to change only some specific columns' width (28917).
• DataFrame.to_excel can now also write OpenOffice spreadsheet (.ods) files (27222)
• ~Series.explode now accepts ignore_index to reset the index, similar to pd.concat or DataFrame.sort_values (34932).
• DataFrame.to_markdown and Series.to_markdown now accept index argument as an alias for tabulate's showindex (32667)
• read_csv now accepts string values like "0", "0.0", "1", "1.0" as convertible to the nullable Boolean dtype (34859)
• pandas.core.window.ExponentialMovingWindow now supports a times argument that allows mean to be calculated with observations spaced by the timestamps in times (34839)
• DataFrame.agg and Series.agg now accept named aggregation for renaming the output columns/indexes. (26513)
• compute.use_numba now exists as a configuration option that utilizes the numba engine when available (33966, 35374)
• Series.plot now supports asymmetric error bars. Previously, if Series.plot received a "2xN" array with error values for yerr and/or xerr, the left/lower values (first row) were mirrored, while the right/upper values (second row) were ignored. Now, the first row represents the left/lower error values and the second row the right/upper error values. (9536)

Notable bug fixes

These are bug fixes that might have notable behavior changes.

MultiIndex.get_indexer interprets method argument correctly

This restores the behavior of MultiIndex.get_indexer with method='backfill' or method='pad' to the behavior before pandas 0.23.0. In particular, MultiIndexes are treated as a list of tuples and padding or backfilling is done with respect to the ordering of these lists of tuples (29896).

As an example of this, given:

python

df = pd.DataFrame({

'a': [0, 0, 0, 0], 'b': [0, 2, 3, 4], 'c': ['A', 'B', 'C', 'D'],

}).set_index(['a', 'b']) mi_2 = pd.MultiIndex.from_product([[0], [-1, 0, 1, 3, 4, 5]])

The differences in reindexing df with mi_2 and using method='backfill' can be seen here:

pandas >= 0.23, < 1.1.0:

In [1]: df.reindex(mi_2, method='backfill')
Out[1]:
      c
0 -1  A
   0  A
   1  D
   3  A
   4  A
   5  C

pandas <0.23, >= 1.1.0

python

df.reindex(mi_2, method='backfill')

And the differences in reindexing df with mi_2 and using method='pad' can be seen here:

pandas >= 0.23, < 1.1.0

In [1]: df.reindex(mi_2, method='pad')
Out[1]:
        c
0 -1  NaN
   0  NaN
   1    D
   3  NaN
   4    A
   5    C

pandas < 0.23, >= 1.1.0

python

df.reindex(mi_2, method='pad')

Failed Label-Based Lookups Always Raise KeyError

Label lookups series[key], series.loc[key] and frame.loc[key] used to raise either KeyError or TypeError depending on the type of key and type of Index. These now consistently raise KeyError (31867)

python

ser1 = pd.Series(range(3), index=[0, 1, 2]) ser2 = pd.Series(range(3), index=pd.date_range("2020-02-01", periods=3))

Previous behavior:

In [3]: ser1[1.5]
...
TypeError: cannot do label indexing on Int64Index with these indexers [1.5] of type float

In [4] ser1["foo"]
...
KeyError: 'foo'

In [5]: ser1.loc[1.5]
...
TypeError: cannot do label indexing on Int64Index with these indexers [1.5] of type float

In [6]: ser1.loc["foo"]
...
KeyError: 'foo'

In [7]: ser2.loc[1]
...
TypeError: cannot do label indexing on DatetimeIndex with these indexers [1] of type int

In [8]: ser2.loc[pd.Timestamp(0)]
...
KeyError: Timestamp('1970-01-01 00:00:00')

New behavior:

In [3]: ser1[1.5]
...
KeyError: 1.5

In [4] ser1["foo"]
...
KeyError: 'foo'

In [5]: ser1.loc[1.5]
...
KeyError: 1.5

In [6]: ser1.loc["foo"]
...
KeyError: 'foo'

In [7]: ser2.loc[1]
...
KeyError: 1

In [8]: ser2.loc[pd.Timestamp(0)]
...
KeyError: Timestamp('1970-01-01 00:00:00')

Similarly, DataFrame.at and Series.at will raise a TypeError instead of a ValueError if an incompatible key is passed, and KeyError if a missing key is passed, matching the behavior of .loc[] (31722)

Failed Integer Lookups on MultiIndex Raise KeyError

Indexing with integers with a MultiIndex that has an integer-dtype first level incorrectly failed to raise KeyError when one or more of those integer keys is not present in the first level of the index (33539)

python

idx = pd.Index(range(4)) dti = pd.date_range("2000-01-03", periods=3) mi = pd.MultiIndex.from_product([idx, dti]) ser = pd.Series(range(len(mi)), index=mi)

Previous behavior:

In [5]: ser[[5]]
Out[5]: Series([], dtype: int64)

New behavior:

In [5]: ser[[5]]
...
KeyError: '[5] not in index'

DataFrame.merge preserves right frame's row order

DataFrame.merge now preserves the right frame's row order when executing a right merge (27453)

python

left_df = pd.DataFrame({'animal': ['dog', 'pig'],

'max_speed': [40, 11]})

right_df = pd.DataFrame({'animal': ['quetzal', 'pig'],

'max_speed': [80, 11]})

left_df right_df

Previous behavior:

>>> left_df.merge(right_df, on=['animal', 'max_speed'], how="right")
    animal  max_speed
0      pig         11
1  quetzal         80

New behavior:

python

left_df.merge(right_df, on=['animal', 'max_speed'], how="right")

Assignment to multiple columns of a DataFrame when some columns do not exist

Assignment to multiple columns of a DataFrame when some of the columns do not exist would previously assign the values to the last column. Now, new columns will be constructed with the right values. (13658)

python

df = pd.DataFrame({'a': [0, 1, 2], 'b': [3, 4, 5]}) df

Previous behavior:

In [3]: df[['a', 'c']] = 1
In [4]: df
Out[4]:
   a  b
0  1  1
1  1  1
2  1  1

New behavior:

python

df[['a', 'c']] = 1 df

Consistency across groupby reductions

Using DataFrame.groupby with as_index=True and the aggregation nunique would include the grouping column(s) in the columns of the result. Now the grouping column(s) only appear in the index, consistent with other reductions. (32579)

python

df = pd.DataFrame({"a": ["x", "x", "y", "y"], "b": [1, 1, 2, 3]}) df

Previous behavior:

In [3]: df.groupby("a", as_index=True).nunique()
Out[4]:
   a  b
a
x  1  1
y  1  2

New behavior:

python

df.groupby("a", as_index=True).nunique()

Using DataFrame.groupby with as_index=False and the function idxmax, idxmin, mad, nunique, sem, skew, or std would modify the grouping column. Now the grouping column remains unchanged, consistent with other reductions. (21090, 10355)

Previous behavior:

In [3]: df.groupby("a", as_index=False).nunique()
Out[4]:
   a  b
0  1  1
1  1  2

New behavior:

python

df.groupby("a", as_index=False).nunique()

The method ~pandas.core.groupby.DataFrameGroupBy.size would previously ignore as_index=False. Now the grouping columns are returned as columns, making the result a DataFrame instead of a Series. (32599)

Previous behavior:

In [3]: df.groupby("a", as_index=False).size()
Out[4]:
a
x    2
y    2
dtype: int64

New behavior:

python

df.groupby("a", as_index=False).size()

~pandas.core.groupby.DataFrameGroupby.agg lost results with as_index=False when relabeling columns

Previously ~pandas.core.groupby.DataFrameGroupby.agg lost the result columns, when the as_index option was set to False and the result columns were relabeled. In this case the result values were replaced with the previous index (32240).

python

df = pd.DataFrame({"key": ["x", "y", "z", "x", "y", "z"],

"val": [1.0, 0.8, 2.0, 3.0, 3.6, 0.75]})

df

Previous behavior:

In [2]: grouped = df.groupby("key", as_index=False)
In [3]: result = grouped.agg(min_val=pd.NamedAgg(column="val", aggfunc="min"))
In [4]: result
Out[4]:
     min_val
 0   x
 1   y
 2   z

New behavior:

python

grouped = df.groupby("key", as_index=False) result = grouped.agg(min_val=pd.NamedAgg(column="val", aggfunc="min")) result

apply and applymap on DataFrame evaluates first row/column only once

python

df = pd.DataFrame({'a': [1, 2], 'b': [3, 6]})

def func(row):

print(row) return row

Previous behavior:

In [4]: df.apply(func, axis=1)
a    1
b    3
Name: 0, dtype: int64
a    1
b    3
Name: 0, dtype: int64
a    2
b    6
Name: 1, dtype: int64
Out[4]:
   a  b
0  1  3
1  2  6

New behavior:

python

df.apply(func, axis=1)

Increased minimum versions for dependencies

Some minimum supported versions of dependencies were updated (33718, 29766, 29723, pytables >= 3.4.3). If installed, we now require:

Package	Minimum Version	Required	Changed
numpy	1.15.4	X	X
pytz	2015.4	X
python-dateutil	2.7.3	X	X
bottleneck	1.2.1
numexpr	2.6.2
pytest (dev)	4.0.2

For optional libraries the general recommendation is to use the latest version. The following table lists the lowest version per library that is currently being tested throughout the development of pandas. Optional libraries below the lowest tested version may still work, but are not considered supported.

Package	Minimum Version	Changed
beautifulsoup4	4.6.0
fastparquet	0.3.2
fsspec	0.7.4
gcsfs	0.6.0	X
lxml	3.8.0
matplotlib	2.2.2
numba	0.46.0
openpyxl	2.5.7
pyarrow	0.13.0
pymysql	0.7.1
pytables	3.4.3	X
s3fs	0.4.0	X
scipy	1.2.0	X
sqlalchemy	1.1.4
xarray	0.8.2
xlrd	1.1.0
xlsxwriter	0.9.8
xlwt	1.2.0
pandas-gbq	1.2.0	X

See install.dependencies and install.optional_dependencies for more.

Development Changes

• The minimum version of Cython is now the most recent bug-fix version (0.29.16) (33334).

Deprecations

• Lookups on a Series with a single-item list containing a slice (e.g. ser[[slice(0, 4)]]) are deprecated and will raise in a future version. Either convert the list to a tuple, or pass the slice directly instead (31333)
• DataFrame.mean and DataFrame.median with numeric_only=None will include datetime64 and datetime64tz columns in a future version (29941)
• Setting values with .loc using a positional slice is deprecated and will raise in a future version. Use .loc with labels or .iloc with positions instead (31840)
• DataFrame.to_dict has deprecated accepting short names for orient and will raise in a future version (32515)
• Categorical.to_dense is deprecated and will be removed in a future version, use np.asarray(cat) instead (32639)
• The fastpath keyword in the SingleBlockManager constructor is deprecated and will be removed in a future version (33092)
• Providing suffixes as a set in pandas.merge is deprecated. Provide a tuple instead (33740, 34741).
• Indexing a Series with a multi-dimensional indexer like [:, None] to return an ndarray now raises a FutureWarning. Convert to a NumPy array before indexing instead (27837)
• Index.is_mixed is deprecated and will be removed in a future version, check index.inferred_type directly instead (32922)
• Passing any arguments but the first one to read_html as positional arguments is deprecated. All other arguments should be given as keyword arguments (27573).
• Passing any arguments but path_or_buf (the first one) to read_json as positional arguments is deprecated. All other arguments should be given as keyword arguments (27573).
• Passing any arguments but the first two to read_excel as positional arguments is deprecated. All other arguments should be given as keyword arguments (27573).
• pandas.api.types.is_categorical is deprecated and will be removed in a future version; use pandas.api.types.is_categorical_dtype instead (33385)
• Index.get_value is deprecated and will be removed in a future version (19728)
• Series.dt.week and Series.dt.weekofyear are deprecated and will be removed in a future version, use Series.dt.isocalendar().week instead (33595)
• DatetimeIndex.week and DatetimeIndex.weekofyear are deprecated and will be removed in a future version, use DatetimeIndex.isocalendar().week instead (33595)
• DatetimeArray.week and DatetimeArray.weekofyear are deprecated and will be removed in a future version, use DatetimeArray.isocalendar().week instead (33595)
• DateOffset.__call__ is deprecated and will be removed in a future version, use offset + other instead (34171)
• ~pandas.tseries.offsets.BusinessDay.apply_index is deprecated and will be removed in a future version. Use offset + other instead (34580)
• DataFrame.tshift and Series.tshift are deprecated and will be removed in a future version, use DataFrame.shift and Series.shift instead (11631)
• Indexing an Index object with a float key is deprecated, and will raise an IndexError in the future. You can manually convert to an integer key instead (34191).
• The squeeze keyword in ~DataFrame.groupby is deprecated and will be removed in a future version (32380)
• The tz keyword in Period.to_timestamp is deprecated and will be removed in a future version; use per.to_timestamp(...).tz_localize(tz) instead (34522)
• DatetimeIndex.to_perioddelta is deprecated and will be removed in a future version. Use index - index.to_period(freq).to_timestamp() instead (34853)
• DataFrame.melt accepting a value_name that already exists is deprecated, and will be removed in a future version (34731)
• The center keyword in the DataFrame.expanding function is deprecated and will be removed in a future version (20647)

Performance improvements

• Performance improvement in Timedelta constructor (30543)
• Performance improvement in Timestamp constructor (30543)
• Performance improvement in flex arithmetic ops between DataFrame and Series with axis=0 (31296)
• Performance improvement in arithmetic ops between DataFrame and Series with axis=1 (33600)
• The internal index method ~Index._shallow_copy now copies cached attributes over to the new index, avoiding creating these again on the new index. This can speed up many operations that depend on creating copies of existing indexes (28584, 32640, 32669)
• Significant performance improvement when creating a DataFrame with sparse values from scipy.sparse matrices using the DataFrame.sparse.from_spmatrix constructor (32821, 32825, 32826, 32856, 32858).
• Performance improvement for groupby methods ~pandas.core.groupby.groupby.Groupby.first and ~pandas.core.groupby.groupby.Groupby.last (34178)
• Performance improvement in factorize for nullable (integer and Boolean) dtypes (33064).
• Performance improvement when constructing Categorical objects (33921)
• Fixed performance regression in pandas.qcut and pandas.cut (33921)
• Performance improvement in reductions (sum, prod, min, max) for nullable (integer and Boolean) dtypes (30982, 33261, 33442).
• Performance improvement in arithmetic operations between two DataFrame objects (32779)
• Performance improvement in pandas.core.groupby.RollingGroupby (34052)
• Performance improvement in arithmetic operations (sub, add, mul, div) for MultiIndex (34297)
• Performance improvement in DataFrame[bool_indexer] when bool_indexer is a list (33924)
• Significant performance improvement of io.formats.style.Styler.render with styles added with various ways such as io.formats.style.Styler.apply, io.formats.style.Styler.applymap or io.formats.style.Styler.bar (19917)

Bug fixes

Categorical

• Passing an invalid fill_value to Categorical.take raises a ValueError instead of TypeError (33660)
• Combining a Categorical with integer categories and which contains missing values with a float dtype column in operations such as concat or ~DataFrame.append will now result in a float column instead of an object dtype column (33607)
• Bug where merge was unable to join on non-unique categorical indices (28189)
• Bug when passing categorical data to Index constructor along with dtype=object incorrectly returning a CategoricalIndex instead of object-dtype Index (32167)
• Bug where Categorical comparison operator __ne__ would incorrectly evaluate to False when either element was missing (32276)
• Categorical.fillna now accepts Categorical other argument (32420)
• Repr of Categorical was not distinguishing between int and str (33676)

Datetimelike

• Passing an integer dtype other than int64 to np.array(period_index, dtype=...) will now raise TypeError instead of incorrectly using int64 (32255)
• Series.to_timestamp now raises a TypeError if the axis is not a PeriodIndex. Previously an AttributeError was raised (33327)
• Series.to_period now raises a TypeError if the axis is not a DatetimeIndex. Previously an AttributeError was raised (33327)
• Period no longer accepts tuples for the freq argument (34658)
• Bug in Timestamp where constructing a Timestamp from ambiguous epoch time and calling constructor again changed the Timestamp.value property (24329)
• DatetimeArray.searchsorted, TimedeltaArray.searchsorted, PeriodArray.searchsorted not recognizing non-pandas scalars and incorrectly raising ValueError instead of TypeError (30950)
• Bug in Timestamp where constructing Timestamp with dateutil timezone less than 128 nanoseconds before daylight saving time switch from winter to summer would result in nonexistent time (31043)
• Bug in Period.to_timestamp, Period.start_time with microsecond frequency returning a timestamp one nanosecond earlier than the correct time (31475)
• Timestamp raised a confusing error message when year, month or day is missing (31200)
• Bug in DatetimeIndex constructor incorrectly accepting bool-dtype inputs (32668)
• Bug in DatetimeIndex.searchsorted not accepting a list or Series as its argument (32762)
• Bug where PeriodIndex raised when passed a Series of strings (26109)
• Bug in Timestamp arithmetic when adding or subtracting an np.ndarray with timedelta64 dtype (33296)
• Bug in DatetimeIndex.to_period not inferring the frequency when called with no arguments (33358)
• Bug in DatetimeIndex.tz_localize incorrectly retaining freq in some cases where the original freq is no longer valid (30511)
• Bug in DatetimeIndex.intersection losing freq and timezone in some cases (33604)
• Bug in DatetimeIndex.get_indexer where incorrect output would be returned for mixed datetime-like targets (33741)
• Bug in DatetimeIndex addition and subtraction with some types of DateOffset objects incorrectly retaining an invalid freq attribute (33779)
• Bug in DatetimeIndex where setting the freq attribute on an index could silently change the freq attribute on another index viewing the same data (33552)
• DataFrame.min and DataFrame.max were not returning consistent results with Series.min and Series.max when called on objects initialized with empty pd.to_datetime
• Bug in DatetimeIndex.intersection and TimedeltaIndex.intersection with results not having the correct name attribute (33904)
• Bug in DatetimeArray.__setitem__, TimedeltaArray.__setitem__, PeriodArray.__setitem__ incorrectly allowing values with int64 dtype to be silently cast (33717)
• Bug in subtracting TimedeltaIndex from Period incorrectly raising TypeError in some cases where it should succeed and IncompatibleFrequency in some cases where it should raise TypeError (33883)
• Bug in constructing a Series or Index from a read-only NumPy array with non-ns resolution which converted to object dtype instead of coercing to datetime64[ns] dtype when within the timestamp bounds (34843).
• The freq keyword in Period, date_range, period_range, pd.tseries.frequencies.to_offset no longer allows tuples, pass as string instead (34703)
• Bug in DataFrame.append when appending a Series containing a scalar tz-aware Timestamp to an empty DataFrame resulted in an object column instead of datetime64[ns, tz] dtype (35038)
• OutOfBoundsDatetime issues an improved error message when timestamp is out of implementation bounds. (32967)
• Bug in AbstractHolidayCalendar.holidays when no rules were defined (31415)
• Bug in Tick comparisons raising TypeError when comparing against timedelta-like objects (34088)
• Bug in Tick multiplication raising TypeError when multiplying by a float (34486)

Timedelta

• Bug in constructing a Timedelta with a high precision integer that would round the Timedelta components (31354)
• Bug in dividing np.nan or None by Timedelta incorrectly returning NaT (31869)
• Timedelta now understands µs as an identifier for microsecond (32899)
• Timedelta string representation now includes nanoseconds, when nanoseconds are non-zero (9309)
• Bug in comparing a Timedelta object against an np.ndarray with timedelta64 dtype incorrectly viewing all entries as unequal (33441)
• Bug in timedelta_range that produced an extra point on a edge case (30353, 33498)
• Bug in DataFrame.resample that produced an extra point on a edge case (30353, 13022, 33498)
• Bug in DataFrame.resample that ignored the loffset argument when dealing with timedelta (7687, 33498)
• Bug in Timedelta and pandas.to_timedelta that ignored the unit argument for string input (12136)

Timezones

• Bug in to_datetime with infer_datetime_format=True where timezone names (e.g. UTC) would not be parsed correctly (33133)

Numeric

• Bug in DataFrame.floordiv with axis=0 not treating division-by-zero like Series.floordiv (31271)
• Bug in to_numeric with string argument "uint64" and errors="coerce" silently fails (32394)
• Bug in to_numeric with downcast="unsigned" fails for empty data (32493)
• Bug in DataFrame.mean with numeric_only=False and either datetime64 dtype or PeriodDtype column incorrectly raising TypeError (32426)
• Bug in DataFrame.count with level="foo" and index level "foo" containing NaNs causes segmentation fault (21824)
• Bug in DataFrame.diff with axis=1 returning incorrect results with mixed dtypes (32995)
• Bug in DataFrame.corr and DataFrame.cov raising when handling nullable integer columns with pandas.NA (33803)
• Bug in arithmetic operations between DataFrame objects with non-overlapping columns with duplicate labels causing an infinite loop (35194)
• Bug in DataFrame and Series addition and subtraction between object-dtype objects and datetime64 dtype objects (33824)
• Bug in Index.difference giving incorrect results when comparing a Float64Index and object Index (35217)
• Bug in DataFrame reductions (e.g. df.min(), df.max()) with ExtensionArray dtypes (34520, 32651)
• Series.interpolate and DataFrame.interpolate now raise a ValueError if limit_direction is 'forward' or 'both' and method is 'backfill' or 'bfill' or limit_direction is 'backward' or 'both' and method is 'pad' or 'ffill' (34746)

Conversion

• Bug in Series construction from NumPy array with big-endian datetime64 dtype (29684)
• Bug in Timedelta construction with large nanoseconds keyword value (32402)
• Bug in DataFrame construction where sets would be duplicated rather than raising (32582)
• The DataFrame constructor no longer accepts a list of DataFrame objects. Because of changes to NumPy, DataFrame objects are now consistently treated as 2D objects, so a list of DataFrame objects is considered 3D, and no longer acceptable for the DataFrame constructor (32289).
• Bug in DataFrame when initiating a frame with lists and assign columns with nested list for MultiIndex (32173)
• Improved error message for invalid construction of list when creating a new index (35190)

Strings

• Bug in the ~Series.astype method when converting "string" dtype data to nullable integer dtype (32450).
• Fixed issue where taking min or max of a StringArray or Series with StringDtype type would raise. (31746)
• Bug in Series.str.cat returning NaN output when other had Index type (33425)
• pandas.api.dtypes.is_string_dtype no longer incorrectly identifies categorical series as string.

Interval

• Bug in IntervalArray incorrectly allowing the underlying data to be changed when setting values (32782)

Indexing

• DataFrame.xs now raises a TypeError if a level keyword is supplied and the axis is not a MultiIndex. Previously an AttributeError was raised (33610)
• Bug in slicing on a DatetimeIndex with a partial-timestamp dropping high-resolution indices near the end of a year, quarter, or month (31064)
• Bug in PeriodIndex.get_loc treating higher-resolution strings differently from PeriodIndex.get_value (31172)
• Bug in Series.at and DataFrame.at not matching .loc behavior when looking up an integer in a Float64Index (31329)
• Bug in PeriodIndex.is_monotonic incorrectly returning True when containing leading NaT entries (31437)
• Bug in DatetimeIndex.get_loc raising KeyError with converted-integer key instead of the user-passed key (31425)
• Bug in Series.xs incorrectly returning Timestamp instead of datetime64 in some object-dtype cases (31630)
• Bug in DataFrame.iat incorrectly returning Timestamp instead of datetime in some object-dtype cases (32809)
• Bug in DataFrame.at when either columns or index is non-unique (33041)
• Bug in Series.loc and DataFrame.loc when indexing with an integer key on a object-dtype Index that is not all-integers (31905)
• Bug in DataFrame.iloc.__setitem__ on a DataFrame with duplicate columns incorrectly setting values for all matching columns (15686, 22036)
• Bug in DataFrame.loc and Series.loc with a DatetimeIndex, TimedeltaIndex, or PeriodIndex incorrectly allowing lookups of non-matching datetime-like dtypes (32650)
• Bug in Series.__getitem__ indexing with non-standard scalars, e.g. np.dtype (32684)
• Bug in Index constructor where an unhelpful error message was raised for NumPy scalars (33017)
• Bug in DataFrame.lookup incorrectly raising an AttributeError when frame.index or frame.columns is not unique; this will now raise a ValueError with a helpful error message (33041)
• Bug in Interval where a Timedelta could not be added or subtracted from a Timestamp interval (32023)
• Bug in DataFrame.copy not invalidating _item_cache after copy caused post-copy value updates to not be reflected (31784)
• Fixed regression in DataFrame.loc and Series.loc throwing an error when a datetime64[ns, tz] value is provided (32395)
• Bug in Series.__getitem__ with an integer key and a MultiIndex with leading integer level failing to raise KeyError if the key is not present in the first level (33355)
• Bug in DataFrame.iloc when slicing a single column DataFrame with ExtensionDtype (e.g. df.iloc[:, :1]) returning an invalid result (32957)
• Bug in DatetimeIndex.insert and TimedeltaIndex.insert causing index freq to be lost when setting an element into an empty Series (33573)
• Bug in Series.__setitem__ with an IntervalIndex and a list-like key of integers (33473)
• Bug in Series.__getitem__ allowing missing labels with np.ndarray, Index, Series indexers but not list, these now all raise KeyError (33646)
• Bug in DataFrame.truncate and Series.truncate where index was assumed to be monotone increasing (33756)
• Indexing with a list of strings representing datetimes failed on DatetimeIndex or PeriodIndex (11278)
• Bug in Series.at when used with a MultiIndex would raise an exception on valid inputs (26989)
• Bug in DataFrame.loc with dictionary of values changes columns with dtype of int to float (34573)
• Bug in Series.loc when used with a MultiIndex would raise an IndexingError when accessing a None value (34318)
• Bug in DataFrame.reset_index and Series.reset_index would not preserve data types on an empty DataFrame or Series with a MultiIndex (19602)
• Bug in Series and DataFrame indexing with a time key on a DatetimeIndex with NaT entries (35114)

Missing

• Calling fillna on an empty Series now correctly returns a shallow copied object. The behaviour is now consistent with Index, DataFrame and a non-empty Series (32543).
• Bug in Series.replace when argument to_replace is of type dict/list and is used on a Series containing <NA> was raising a TypeError. The method now handles this by ignoring <NA> values when doing the comparison for the replacement (32621)
• Bug in ~Series.any and ~Series.all incorrectly returning <NA> for all False or all True values using the nulllable Boolean dtype and with skipna=False (33253)
• Clarified documentation on interpolate with method=akima. The der parameter must be scalar or None (33426)
• DataFrame.interpolate uses the correct axis convention now. Previously interpolating along columns lead to interpolation along indices and vice versa. Furthermore interpolating with methods pad, ffill, bfill and backfill are identical to using these methods with DataFrame.fillna (12918, 29146)
• Bug in DataFrame.interpolate when called on a DataFrame with column names of string type was throwing a ValueError. The method is now independent of the type of the column names (33956)
• Passing NA into a format string using format specs will now work. For example "{:.1f}".format(pd.NA) would previously raise a ValueError, but will now return the string "<NA>" (34740)
• Bug in Series.map not raising on invalid na_action (32815)

MultiIndex

• DataFrame.swaplevels now raises a TypeError if the axis is not a MultiIndex. Previously an AttributeError was raised (31126)
• Bug in Dataframe.loc when used with a MultiIndex. The returned values were not in the same order as the given inputs (22797)

python

df = pd.DataFrame(np.arange(4),

index=[["a", "a", "b", "b"], [1, 2, 1, 2]])

# Rows are now ordered as the requested keys df.loc[(['b', 'a'], [2, 1]), :]

• Bug in MultiIndex.intersection was not guaranteed to preserve order when sort=False. (31325)
• Bug in DataFrame.truncate was dropping MultiIndex names. (34564)

python

left = pd.MultiIndex.from_arrays([["b", "a"], [2, 1]]) right = pd.MultiIndex.from_arrays([["a", "b", "c"], [1, 2, 3]]) # Common elements are now guaranteed to be ordered by the left side left.intersection(right, sort=False)

• Bug when joining two MultiIndex without specifying level with different columns. Return-indexers parameter was ignored. (34074)

I/O

• Passing a set as names argument to pandas.read_csv, pandas.read_table, or pandas.read_fwf will raise ValueError: Names should be an ordered collection. (34946)
• Bug in print-out when display.precision is zero. (20359)
• Bug in read_json where integer overflow was occurring when json contains big number strings. (30320)
• read_csv will now raise a ValueError when the arguments header and prefix both are not None. (27394)
• Bug in DataFrame.to_json was raising NotFoundError when path_or_buf was an S3 URI (28375)
• Bug in DataFrame.to_parquet overwriting pyarrow's default for coerce_timestamps; following pyarrow's default allows writing nanosecond timestamps with version="2.0" (31652).
• Bug in read_csv was raising TypeError when sep=None was used in combination with comment keyword (31396)
• Bug in HDFStore that caused it to set to int64 the dtype of a datetime64 column when reading a DataFrame in Python 3 from fixed format written in Python 2 (31750)
• read_sas() now handles dates and datetimes larger than Timestamp.max returning them as datetime.datetime objects (20927)
• Bug in DataFrame.to_json where Timedelta objects would not be serialized correctly with date_format="iso" (28256)
• read_csv will raise a ValueError when the column names passed in parse_dates are missing in the Dataframe (31251)
• Bug in read_excel where a UTF-8 string with a high surrogate would cause a segmentation violation (23809)
• Bug in read_csv was causing a file descriptor leak on an empty file (31488)
• Bug in read_csv was causing a segfault when there were blank lines between the header and data rows (28071)
• Bug in read_csv was raising a misleading exception on a permissions issue (23784)
• Bug in read_csv was raising an IndexError when header=None and two extra data columns
• Bug in read_sas was raising an AttributeError when reading files from Google Cloud Storage (33069)
• Bug in DataFrame.to_sql where an AttributeError was raised when saving an out of bounds date (26761)
• Bug in read_excel did not correctly handle multiple embedded spaces in OpenDocument text cells. (32207)
• Bug in read_json was raising TypeError when reading a list of Booleans into a Series. (31464)
• Bug in pandas.io.json.json_normalize where location specified by record_path doesn't point to an array. (26284)
• pandas.read_hdf has a more explicit error message when loading an unsupported HDF file (9539)
• Bug in ~DataFrame.read_feather was raising an ArrowIOError when reading an s3 or http file path (29055)
• Bug in ~DataFrame.to_excel could not handle the column name render and was raising an KeyError (34331)
• Bug in ~SQLDatabase.execute was raising a ProgrammingError for some DB-API drivers when the SQL statement contained the % character and no parameters were present (34211)
• Bug in ~pandas.io.stata.StataReader which resulted in categorical variables with different dtypes when reading data using an iterator. (31544)
• HDFStore.keys has now an optional include parameter that allows the retrieval of all native HDF5 table names (29916)
• TypeError exceptions raised by read_csv and read_table were showing as parser_f when an unexpected keyword argument was passed (25648)
• Bug in read_excel for ODS files removes 0.0 values (27222)
• Bug in ujson.encode was raising an OverflowError with numbers larger than sys.maxsize (34395)
• Bug in HDFStore.append_to_multiple was raising a ValueError when the min_itemsize parameter is set (11238)
• Bug in ~HDFStore.create_table now raises an error when column argument was not specified in data_columns on input (28156)
• read_json now could read line-delimited json file from a file url while lines and chunksize are set.
• Bug in DataFrame.to_sql when reading DataFrames with -np.inf entries with MySQL now has a more explicit ValueError (34431)
• Bug where capitalised files extensions were not decompressed by read* functions (35164)
• Bug in read_excel that was raising a TypeError when header=None and index_col is given as a list (31783)
• Bug in read_excel where datetime values are used in the header in a MultiIndex (34748)
• read_excel no longer takes **kwds arguments. This means that passing in the keyword argument chunksize now raises a TypeError (previously raised a NotImplementedError), while passing in the keyword argument encoding now raises a TypeError (34464)
• Bug in DataFrame.to_records was incorrectly losing timezone information in timezone-aware datetime64 columns (32535)

Plotting

• DataFrame.plot for line/bar now accepts color by dictionary (8193).
• Bug in DataFrame.plot.hist where weights are not working for multiple columns (33173)
• Bug in DataFrame.boxplot and DataFrame.plot.boxplot lost color attributes of medianprops, whiskerprops, capprops and boxprops (30346)
• Bug in DataFrame.hist where the order of column argument was ignored (29235)
• Bug in DataFrame.plot.scatter that when adding multiple plots with different cmap, colorbars always use the first cmap (33389)
• Bug in DataFrame.plot.scatter was adding a colorbar to the plot even if the argument c was assigned to a column containing color names (34316)
• Bug in pandas.plotting.bootstrap_plot was causing cluttered axes and overlapping labels (34905)
• Bug in DataFrame.plot.scatter caused an error when plotting variable marker sizes (32904)

Groupby/resample/rolling

• Using a pandas.api.indexers.BaseIndexer with count, min, max, median, skew, cov, corr will now return correct results for any monotonic pandas.api.indexers.BaseIndexer descendant (32865)
• DataFrameGroupby.mean and SeriesGroupby.mean (and similarly for ~DataFrameGroupby.median, ~DataFrameGroupby.std and ~DataFrameGroupby.var) now raise a TypeError if a non-accepted keyword argument is passed into it. Previously an UnsupportedFunctionCall was raised (AssertionError if min_count passed into ~DataFrameGroupby.median) (31485)
• Bug in GroupBy.apply raises ValueError when the by axis is not sorted, has duplicates, and the applied func does not mutate passed in objects (30667)
• Bug in DataFrameGroupBy.transform produces an incorrect result with transformation functions (30918)
• Bug in Groupby.transform was returning the wrong result when grouping by multiple keys of which some were categorical and others not (32494)
• Bug in GroupBy.count causes segmentation fault when grouped-by columns contain NaNs (32841)
• Bug in DataFrame.groupby and Series.groupby produces inconsistent type when aggregating Boolean Series (32894)
• Bug in DataFrameGroupBy.sum and SeriesGroupBy.sum where a large negative number would be returned when the number of non-null values was below min_count for nullable integer dtypes (32861)
• Bug in SeriesGroupBy.quantile was raising on nullable integers (33136)
• Bug in DataFrame.resample where an AmbiguousTimeError would be raised when the resulting timezone aware DatetimeIndex had a DST transition at midnight (25758)
• Bug in DataFrame.groupby where a ValueError would be raised when grouping by a categorical column with read-only categories and sort=False (33410)
• Bug in GroupBy.agg, GroupBy.transform, and GroupBy.resample where subclasses are not preserved (28330)
• Bug in SeriesGroupBy.agg where any column name was accepted in the named aggregation of SeriesGroupBy previously. The behaviour now allows only str and callables else would raise TypeError. (34422)
• Bug in DataFrame.groupby lost the name of the Index when one of the agg keys referenced an empty list (32580)
• Bug in Rolling.apply where center=True was ignored when engine='numba' was specified (34784)
• Bug in DataFrame.ewm.cov was throwing AssertionError for MultiIndex inputs (34440)
• Bug in core.groupby.DataFrameGroupBy.quantile raised TypeError for non-numeric types rather than dropping the columns (27892)
• Bug in core.groupby.DataFrameGroupBy.transform when func='nunique' and columns are of type datetime64, the result would also be of type datetime64 instead of int64 (35109)
• Bug in DataFrame.groupby raising an AttributeError when selecting a column and aggregating with as_index=False (35246).
• Bug in DataFrameGroupBy.first and DataFrameGroupBy.last that would raise an unnecessary ValueError when grouping on multiple Categoricals (34951)

Reshaping

• Bug effecting all numeric and Boolean reduction methods not returning subclassed data type. (25596)
• Bug in DataFrame.pivot_table when only MultiIndexed columns is set (17038)
• Bug in DataFrame.unstack and Series.unstack can take tuple names in MultiIndexed data (19966)
• Bug in DataFrame.pivot_table when margin is True and only column is defined (31016)
• Fixed incorrect error message in DataFrame.pivot when columns is set to None. (30924)
• Bug in crosstab when inputs are two Series and have tuple names, the output will keep a dummy MultiIndex as columns. (18321)
• DataFrame.pivot can now take lists for index and columns arguments (21425)
• Bug in concat where the resulting indices are not copied when copy=True (29879)
• Bug in SeriesGroupBy.aggregate was resulting in aggregations being overwritten when they shared the same name (30880)
• Bug where Index.astype would lose the name attribute when converting from Float64Index to Int64Index, or when casting to an ExtensionArray dtype (32013)
• Series.append will now raise a TypeError when passed a DataFrame or a sequence containing DataFrame (31413)
• DataFrame.replace and Series.replace will raise a TypeError if to_replace is not an expected type. Previously the replace would fail silently (18634)
• Bug on inplace operation of a Series that was adding a column to the DataFrame from where it was originally dropped from (using inplace=True) (30484)
• Bug in DataFrame.apply where callback was called with Series parameter even though raw=True requested. (32423)
• Bug in DataFrame.pivot_table losing timezone information when creating a MultiIndex level from a column with timezone-aware dtype (32558)
• Bug in concat where when passing a non-dict mapping as objs would raise a TypeError (32863)
• DataFrame.agg now provides more descriptive SpecificationError message when attempting to aggregate a non-existent column (32755)
• Bug in DataFrame.unstack when MultiIndex columns and MultiIndex rows were used (32624, 24729 and 28306)
• Appending a dictionary to a DataFrame without passing ignore_index=True will raise TypeError: Can only append a dict if ignore_index=True instead of TypeError: Can only append a :class:`Series` if ignore_index=True or if the :class:`Series` has a name (30871)
• Bug in DataFrame.corrwith(), DataFrame.memory_usage(), DataFrame.dot(), DataFrame.idxmin(), DataFrame.idxmax(), DataFrame.duplicated(), DataFrame.isin(), DataFrame.count(), Series.explode(), Series.asof() and DataFrame.asof() not returning subclassed types. (31331)
• Bug in concat was not allowing for concatenation of DataFrame and Series with duplicate keys (33654)
• Bug in cut raised an error when the argument labels contains duplicates (33141)
• Ensure only named functions can be used in eval() (32460)
• Bug in Dataframe.aggregate and Series.aggregate was causing a recursive loop in some cases (34224)
• Fixed bug in melt where melting MultiIndex columns with col_level > 0 would raise a KeyError on id_vars (34129)
• Bug in Series.where with an empty Series and empty cond having non-bool dtype (34592)
• Fixed regression where DataFrame.apply would raise ValueError for elements with S dtype (34529)

Sparse

• Creating a SparseArray from timezone-aware dtype will issue a warning before dropping timezone information, instead of doing so silently (32501)
• Bug in arrays.SparseArray.from_spmatrix wrongly read scipy sparse matrix (31991)
• Bug in Series.sum with SparseArray raised a TypeError (25777)
• Bug where DataFrame containing an all-sparse SparseArray filled with NaN when indexed by a list-like (27781, 29563)
• The repr of SparseDtype now includes the repr of its fill_value attribute. Previously it used fill_value's string representation (34352)
• Bug where empty DataFrame could not be cast to SparseDtype (33113)
• Bug in arrays.SparseArray was returning the incorrect type when indexing a sparse dataframe with an iterable (34526, 34540)

ExtensionArray

• Fixed bug where Series.value_counts would raise on empty input of Int64 dtype (33317)
• Fixed bug in concat when concatenating DataFrame objects with non-overlapping columns resulting in object-dtype columns rather than preserving the extension dtype (27692, 33027)
• Fixed bug where StringArray.isna would return False for NA values when pandas.options.mode.use_inf_as_na was set to True (33655)
• Fixed bug in Series construction with EA dtype and index but no data or scalar data fails (26469)
• Fixed bug that caused Series.__repr__() to crash for extension types whose elements are multidimensional arrays (33770).
• Fixed bug where Series.update would raise a ValueError for ExtensionArray dtypes with missing values (33980)
• Fixed bug where StringArray.memory_usage was not implemented (33963)
• Fixed bug where DataFrameGroupBy would ignore the min_count argument for aggregations on nullable Boolean dtypes (34051)
• Fixed bug where the constructor of DataFrame with dtype='string' would fail (27953, 33623)
• Bug where DataFrame column set to scalar extension type was considered an object type rather than the extension type (34832)
• Fixed bug in IntegerArray.astype to correctly copy the mask as well (34931).

Other

• Set operations on an object-dtype Index now always return object-dtype results (31401)
• Fixed pandas.testing.assert_series_equal to correctly raise if the left argument is a different subclass with check_series_type=True (32670).
• Getting a missing attribute in a DataFrame.query or DataFrame.eval string raises the correct AttributeError (32408)
• Fixed bug in pandas.testing.assert_series_equal where dtypes were checked for Interval and ExtensionArray operands when check_dtype was False (32747)
• Bug in DataFrame.__dir__ caused a segfault when using unicode surrogates in a column name (25509)
• Bug in DataFrame.equals and Series.equals in allowing subclasses to be equal (34402).

Contributors

v1.0.5..v1.1.0|HEAD