Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory Leak with rolling quantile #21965

Closed
fl4p opened this Issue Jul 18, 2018 · 0 comments

Comments

Projects
None yet
3 participants
@fl4p
Copy link
Contributor

fl4p commented Jul 18, 2018

Code Sample

import os
import psutil
import pandas as pd
import numpy as np

process = psutil.Process(os.getpid())
i = 0

def sink():
    global i
    i += 1
    if i % 100 == 0:
        mem = process.memory_info().rss / 1e6
        print("mem %fMB" % mem)

while True:
     s = pd.Series(np.random.randn(5e6))
     m = s.rolling(1e3).quantile(0.1) #  leaks (unexpected)
     # m = s.rolling(1e3).quantile(0) # no leak (expected)
     sink()

Problem description

Rolling quantile leaks memory with quantiles within (0,1)

Related Code

return _window.roll_quantile(arg, window, minp, indexi,

def roll_quantile(ndarray[float64_t, cast=True] input, int64_t win,

The leak should be in roll_quantile cython code. I could not trace it with objgraph.

I believe a free for skiplist is missing.

Program Output

Expected
mem 66.822144MB
mem 66.801664MB
mem 67.072000MB
mem 67.072000MB
mem 67.072000MB
mem 67.072000MB
mem 67.072000MB
mem 67.072000MB
mem 67.072000MB
Actual
mem 80.261120MB
mem 91.615232MB
mem 103.239680MB
mem 114.864128MB
mem 126.218240MB
mem 137.842688MB
mem 149.467136MB
mem 161.091584MB
mem 172.445696MB
mem 184.070144MB
mem 195.694592MB
mem 207.048704MB
mem 218.673152MB
mem 230.297600MB
mem 241.651712MB
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.