Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Funcanimation memory leak? #8528

Closed
bretcj7 opened this issue Apr 23, 2017 · 14 comments

Comments

Projects
None yet
5 participants
@bretcj7
Copy link

commented Apr 23, 2017

I have seen similar issues on here closed or fixed but I don't understand if my issue is how the code is implemented or actual issue. Animation seems to keep frames in memory and not release them.

Bug report

I'm using pyplot 2.0 with the FuncAnimation for streaming serial data to a figure where it is continously updated.

For the FuncAnimation to work you have a Figure, Func (function to call at each frame), and Frames (Source of data to pass func and each frame of the animation) fairly simple I thought and it does animate the way I would expect.

However the figure continues to chew up memory until it crashes the Raspberry Pi even though I have the blit flag set to optimize the drawing and only render the new data points.

Full analysis and description here:
http://stackoverflow.com/questions/43548826/pyplot-animation-memory-leak-with-funcanimation?noredirect=1#comment74171567_43548826

Code for reproduction

import matplotlib.pyplot as plt
import matplotlib.animation as animation

#Setup plotting, one figure and two subplots
fig, (ax1, ax2) = plt.subplots(2,1)#,sharex=True)
g_line = ax1.plot([],[],'g')
r_line = ax1.plot([],[],'r')

lines = []
#Add each line to o ur tuples of lines to be plotted
lines.extend(g_line)
lines.extend(r_line)

#Data plotting function, this is call from the animation
#Each data element is in the data list being returned below
def plot_data(data):
  x1 = data[0]; y1 = data[1]; 
  x2 = data[2]; y2 = data[3]; 

  lines[0].set_data(x1,y1)              # Audio - green
  lines[1].set_data(x2,y2)              # Audio thresh - red
  return tuple(lines)

def data_gen():
  my_list = []
  my_list2 = []

  #Main infinite loop for processing of serial data
  while True:
    # Grab input RX serial data
    (my_list, my_list2) = readUART(serReader)

    yield my_list, my_list2

ani = animation.FuncAnimation(fig, plot_data, data_gen, interval=1, blit=True)
plt.show()

Actual outcome

Note - I ran a memory profiler on the entire program minus the plotting and just did it on the while TRUE loop and the memory didn't move past 55mb the entire time I let it run.

# If applicable, paste the console output here
#
#

Expected outcome

Matplotlib version

  • Operating System:
  • Matplotlib Version:
  • Python Version:
  • Jupyter Version (if applicable):
  • Other Libraries:
@bretcj7

This comment has been minimized.

Copy link
Author

commented Apr 23, 2017

Python 3, matplotlib 2.0.0

@tacaswell

This comment has been minimized.

Copy link
Member

commented Apr 24, 2017

https://github.com/matplotlib/matplotlib/blob/master/lib/matplotlib/animation.py#L1608 is probably the line to blame, we keep a cache of recent frame data. We cull that list, but maybe they are never getting gc'd

Try adding a few

import gc
gc.collect()

into the infinite loop (maybe not every loop, but ever 100 or so).

@tacaswell tacaswell added the Animation label Apr 24, 2017

@tacaswell

This comment has been minimized.

Copy link
Member

commented Apr 24, 2017

Related to #8278

@rberlich

This comment has been minimized.

Copy link

commented May 2, 2017

Hi, some feedback: On MacOS 10.12, Python 3.6 matplotlib 2.0, for an "animation" which is permanently "fed" with measured data, through a generator, the following call

ani = animation.FuncAnimation(self.__m_figure, self.animator, self.retriever
, init_func=self.init, blit=True, repeat=False
, save_count=None
)

leads to a maximum memory usage of about 6 GB, whereas "save_count=0" will quickly run out of memory (no difference to just leaving save_count out ...). Adding the gc.collect() makes no difference.

self.retriever retrieves data sets from a queue and feeds them to the animator-function one by one, which in turn updates an array and feeds it to the plot via set_data.

@tacaswell tacaswell added this to the 2.1 (next point release) milestone May 2, 2017

@rberlich

This comment has been minimized.

Copy link

commented May 2, 2017

Another bit of information O.k., I have tried to comment out the line

self._save_seq.append(framedata)

in animation.py (I have copied the file to the local directory and changed the import statement accordingly). There does not seem to be any effect on the problem, though. The memory rapidly vanishes ...

@rberlich

This comment has been minimized.

Copy link

commented May 3, 2017

Here is a simple test case that will double its memory usage (on MacOS 10.12, Python 3.6, Matplotlib 2.0) in about 10-15 minutes: https://gist.github.com/rberlich/67b915ae9a9067373f93c57b3ec78301

@rberlich

This comment has been minimized.

Copy link

commented May 3, 2017

Apologies for the many messages -- I just think that maybe this may help: I have removed everything with respect to TK from the above example and memory consumption seems to be pretty much stable. So the problem seems to be related to the interaction between tkinter and the animation function (which is quite basic in the example shown in the gist in the last message, I believe).

@WeatherGod

This comment has been minimized.

Copy link
Member

commented May 3, 2017

@rberlich

This comment has been minimized.

Copy link

commented May 3, 2017

Hi, not sure I understand where the timer comes in ? In my production code (i.e. not the toy example from the gist) I use a generator, so the animation function is controlled by whether data can be obtained from a queue or not. My "real-world" plots are far more complex than those from the toy example, and produce Gigabytes instead of Megabytes of memory loss. Hence I believe that caching of images by the animation function might indeed be a problem, even though I would not understand why the interaction with tkinter should play a role there.

One more thing: A friend of mine has tested the toy example from the gist on Linux, and it doesn't loose memory. Hence this may be a specific MacOS problem.

@rberlich

This comment has been minimized.

Copy link

commented May 13, 2017

Below is another example that quickly looses memory on a Mac (newest Anaconda, Matplotlib 2.02, MacOS 10.12.4). Memory consumption increases from 102 to about 150 MB in roughly 30 minutes on that machine. It is stable and does not increase on a Ubuntu 17.04 system, with the newest Anaconda. I am already explicitly triggering gc and have added the "save_count=None" suggested in other posts.

import numpy as np
import matplotlib.pyplot as plt
import matplotlib.animation as animation
import random
import time
import gc


class PlotRandom:
    def __init__(self):
        self.__m_n_samples = 100
        self.__m_random = random.Random()
        self.__m_fig, self.__m_ax = plt.subplots()
        self.__m_line_data = np.abs(np.random.normal(size=self.__m_n_samples))
        self.__m_line, = self.__m_ax.plot(range(self.__m_n_samples), self.__m_line_data)
        self.__m_ax.set_ylim(0, 1.1*np.max(self.__m_line_data))

    def animator(self, new_number):
        self.__m_line.set_xdata(range(self.__m_n_samples))
        self.__m_line_data = np.roll(self.__m_line_data, -1)
        self.__m_line_data[-1] = new_number
        self.__m_line.set_ydata(self.__m_line_data)
        self.__m_ax.set_ylim(0, 1.1*np.max(self.__m_line_data))
        gc.collect()
        return self.__m_line,

    def generator(self):
        while True:
            yield abs(self.__m_random.gauss(mu=0., sigma=1.))

    def run(self):
        ani = animation.FuncAnimation(self.__m_fig, self.animator, self.generator, interval=50, save_count=None)
        plt.show()


if __name__ == "__main__":
    pr = PlotRandom()
    pr.run()
@tacaswell

This comment has been minimized.

Copy link
Member

commented May 14, 2017

Just to check, this is all with the tk backend?

Do you leak memory with the Qt backend on a mac?

Thanks for being persistent on this!

@rberlich

This comment has been minimized.

Copy link

commented May 14, 2017

Hi there,

no, there is no tk backend involved here. It happens with the code as quoted in my text, which "only" uses Matplotlib. As I am not using any backend here, I believe testing with Qt will not yield any new information. Let me know, though, if I should give it a try. I'd have to find out how, though, as so far I haven't used PyQt yet.

One more observation: commenting out the self.__m_line_data = np.roll(self.__m_line_data, -1) also results in a memory loss, so this has nothing to do with the permanent creation of new ndarrays inside of np.roll.

One more funny thing: When adding "blit=True" to the FuncAnimation call, no curve is displayed on the Mac, while it will appear o.k. on the Linux machine. Setting "blit=False" will also show the curve on the Mac.

Kind Regards,
Beet (and sorry for being persistent :-)

@tacaswell

This comment has been minimized.

Copy link
Member

commented May 15, 2017

If you are using pyplot, you are using are using some backend (see https://matplotlib.org/faq/usage_faq.html#what-is-a-backend ) . What does matplotlib.get_backend() return? Given that there is a difference between linux and mac, I suspect that there something in the GUI toolkit (which has the most variation OS to OS).

bliting should work in 2.0 on a mac...

@tacaswell

This comment has been minimized.

Copy link
Member

commented May 15, 2017

Also, we ❤️ persistent users.

@tacaswell tacaswell modified the milestones: 2.1 (next point release), 2.2 (next next feature release) Sep 24, 2017

pclove1 added a commit to pclove1/matplotlib that referenced this issue Sep 3, 2018

pclove1 added a commit to pclove1/matplotlib that referenced this issue Oct 9, 2018

pclove1 added a commit to pclove1/matplotlib that referenced this issue Oct 9, 2018

pclove1 added a commit to pclove1/matplotlib that referenced this issue Dec 30, 2018

pclove1 added a commit to pclove1/matplotlib that referenced this issue Jan 9, 2019

add `cache_frame_data` kwarg into `FuncAnimation`. fixes matplotlib#8528
.

- add ```2018-12-30-add-cache_frame_data-kwarg-to-FuncAnimation.rst``` into ```doc/users/next_whats_new```

timhoffm added a commit that referenced this issue Jan 10, 2019

add `cache_frame_data` kwarg into `FuncAnimation`. fixes #8528. (#11894)
- add ```2018-12-30-add-cache_frame_data-kwarg-to-FuncAnimation.rst``` into ```doc/users/next_whats_new```

@QuLogic QuLogic removed this from the needs sorting milestone Jan 10, 2019

@QuLogic QuLogic added this to the v3.1 milestone Jan 10, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.