Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Index._engine creates cyclic reference #27585

Closed
crepererum opened this issue Jul 25, 2019 · 1 comment · Fixed by #27607

Comments

@crepererum
Copy link
Contributor

commented Jul 25, 2019

Code Sample, a copy-pastable example if possible

import gc
import pandas as pd

gc.disable()

idx = pd.Int64Index(data=[0])

# trigger a call to idx._engine using public API, but a unit test might also be
# able to use the private API here
idx.is_monotonic_increasing
del idx

remaining_indices = [o for o in gc.get_objects() if isinstance(o, pd.Int64Index)]
assert remaining_indices == []

Problem description

The (indirect) call to idx._engine creates a cyclic reference, therefore indices are not removed without calling the GC:

import objgraph
objgraph.show_backrefs(
    remaining_indices,
    filename='obj.png',
    shortnames=False,
    max_depth=10,
)

obj

On certain payloads (e.g. on dask.distributed clusters), this might increase memory consumptions significantly.

Expected Output

Index is cleared on del idx call.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.6.6.final.0
python-bits : 64
OS : Linux
OS-release : 4.9.125-linuxkit
machine : x86_64
processor :
byteorder : little
LC_ALL : en_US.UTF-8
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 0.25.0
numpy : 1.16.4
pytz : 2019.1
dateutil : 2.8.0
pip : 10.0.1
setuptools : 39.1.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None

@jreback

This comment has been minimized.

Copy link
Contributor

commented Jul 25, 2019

cyclic references are very natural in python; have a destructor thru __del__ is not. I suppose you could replace these as weakrefs. If you want to try go for it (though not sure if this would work / pass)

crepererum added a commit to crepererum/pandas that referenced this issue Jul 25, 2019
crepererum added a commit to crepererum/pandas that referenced this issue Jul 25, 2019
@crepererum crepererum referenced this issue Jul 26, 2019
5 of 5 tasks complete

@jbrockmendel jbrockmendel added the Index label Jul 26, 2019

crepererum added a commit to crepererum/pandas that referenced this issue Jul 29, 2019
crepererum added a commit to crepererum/pandas that referenced this issue Jul 29, 2019

@jbrockmendel jbrockmendel changed the title Index._engine creates cyclig reference Index._engine creates cyclic reference Jul 29, 2019

crepererum added a commit to crepererum/pandas that referenced this issue Jul 30, 2019
crepererum added a commit to crepererum/pandas that referenced this issue Jul 31, 2019

@jreback jreback added this to the 0.25.1 milestone Aug 1, 2019

crepererum added a commit to crepererum/pandas that referenced this issue Aug 2, 2019
crepererum added a commit to crepererum/pandas that referenced this issue Aug 5, 2019
crepererum added a commit to crepererum/pandas that referenced this issue Aug 5, 2019
TomAugspurger added a commit that referenced this issue Aug 8, 2019
quintusdias added a commit to quintusdias/pandas_dev that referenced this issue Aug 16, 2019
gabriellm1 added a commit to gabriellm1/pandas that referenced this issue Aug 27, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.