Description
Pandas version checks
I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandas.
I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import gc
import pandas as pd
class Test(object):
def __init__(self, ab):
self.__a = ab
a = pd.Series([Test(i) for i in range(100000)])
a = a[-1:]
gc.collect()
Issue Description
The memory allocating in the line a = pd.Series([Test(i) for i in range(100000)])
does not free when slicing var a and assigning to itself.
Expected Behavior
Free the memory after slicing like build-in list
behavior.
Installed Versions
INSTALLED VERSIONS
commit : 0691c5c
python : 3.12.8
python-bits : 64
OS : Linux
OS-release : 5.4.0-204-generic
Version : #224-Ubuntu SMP Thu Dec 5 13:38:28 UTC 2024
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 2.2.3
numpy : 2.2.1
pytz : 2024.2
dateutil : 2.9.0.post0
pip : 24.3.1
Cython : None
sphinx : None
IPython : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : None
lxml.etree : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
psycopg2 : None
pymysql : None
pyarrow : None
pyreadstat : None
pytest : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlsxwriter : None
zstandard : None
tzdata : 2024.2
qtpy : None
pyqt5 : None
Activity
daxh733 commentedon Jan 2, 2025
Thank you for reporting this issue. I appreciate the detailed description and steps you've provided.
a = pd.Series([Test(i) for i in range(100000)])
a = a[-1:]
del a
gc.collect()
hefvcjm commentedon Jan 3, 2025
Thanks for your replying. But then, after
del a
, I could not usea
. What I expected isa
is a series that just holds last item after slicing while the other items memory should free.daxh733 commentedon Jan 4, 2025
ok!
import pandas as pd
import gc
a = pd.Series([f"Test({i})" for i in range(100000)])
a = a[-1:].copy() ///Using .copy() creates an independent object.
gc.collect()
print(a)
i hope this method using copy will fix the issue
rhshadrach commentedon Jan 4, 2025
Thanks for the report! Confirmed on main and with
copy_on_write=True
, further investigations and PRs to fix are welcome.This is not sufficient. Even without copy, users lose all access to the values and they should be garbage collected. In other words, users should not need to invoke copy here.
niranjanorkat commentedon May 10, 2025
@rhshadrach , I think we have a problem here.
If we internally modify _slice to solve this issue (by breaking the reference to the underlying data block) and free up memory, it seems like we lose functionality that was explicitly designed as a feature.
In particular, tests in
copy_view/test_indexing
andcopy_view/test_methods
start failing, which suggests this shared-memory behavior is intentional and tied to pandas' copy-on-write system.I'm not sure how best to proceed here — would really appreciate your thoughts here.
rhshadrach commentedon May 21, 2025
I think there may be nothing to do here, this is also how NumPy behaves as well.
cc @jorisvandenbossche