Increasing memory (possible leak, inner cache mechanism)

[jakubxy08]

Checklist

• I have searched the existing issues for similar issues.
• I added a very descriptive title to this issue.
• I have provided sufficient information below to help reproduce this issue.

Summary

Good evening to all streamlit users and developers.

I wanted to find out how I can clear the cache or remove objects from streamlit memory so that when running a long running application that updates charts every e.g. 15 seconds, I can enjoy a fairly constant memory consumption rather than a constant increase.

In the code below if we run it uncommented line 34 or 35 or even only 33, we have increasing memory consumption. On the other hand, commenting all lines 33-35, we have constant memory consumption.
If I understand correctly, this is not a problem of plotly just streamlit and its internal caching mechanism. The program behaved similarly even when setting the decorators @st.cache, @st.cache_data, @st.cache_resource, and their internal parameters (e.g. ttl).
To monitor memory, I used memory_profiler and tremalloc as described here: (https://blog.streamlit.io/3-steps-to-fix-app-memory-leaks/).
The code was tested on both Windows 10, Windows 11, Ubuntu 20.04 and all versions of streamlit from 1.17.0 - 1.21.0.
I am very much asking for help

Reproducible Code Example

import streamlit as st
import plotly.graph_objs as go
import pandas as pd
import time
import logging
import numpy as np
import gc
import random
from memory_profiler import profile


def create_plot(df, num_points):
    x, y = df['x'], df['y']
    xi = pd.DatetimeIndex(x)
    x2 = pd.date_range(start=xi.min(), end=xi.max(), periods=num_points)
    y2 = np.interp(x2, xi, y)
    fig = go.Figure(data=go.Scatter(x=x2, y=y2, mode='lines'))
    fig.update_layout(title='Line Plot', xaxis_title='X Axis', yaxis_title='Y Axis')
    return fig


def load_data(size):
    c = 1681581250
    return pd.DataFrame({'x': [c + 1000 * i for i in range(size)], 'y': [random.randint(1, 10) for _ in range(size)]})

gc.enable()
chart_container = st.empty()
while True:
    logging.info("load data")
    data = load_data(1000)
    fig = create_plot(data, 500)

    chart_container.empty() # -- line 33
    # chart_container.write(fig)  # -- line 34
    chart_container.plotly_chart(fig, config={'displayModeBar': False})  # -- line 35

    del fig
    del data
    gc.collect()
    time.sleep(2)

Steps To Reproduce

streamlit run filename.py

Expected Behavior

Constant usage of memory

Current Behavior

Increasing memory over time.

Is this a regression?

• Yes, this used to work in a previous version.

Debug info

• Streamlit version: 1.17.0 - 1.21.0
• Python version: 3.8.9 - 3.10.10
• Operating System: Windows 10, Windows 11, Ubuntu 20.04
• Browser: Chrome, Firefox
• Virtual environment:

Additional Information

Are you willing to submit a PR?

• Yes, I am willing to submit a PR!

[HHest]

Thank you for doing this, @jakubxy08. I've noticed an increase in our long-running app as well, but have never been able to isolate it. Am quite surprised to hear that only having line 33 would leak as well. That line doesn't do anything but call empty() on the chart_container element.

[jakubxy08]

Just the fact that also "container.empty()" called in a loop also causes an increase in memory puzzled me a lot, so I decided to check what "container.empty()" and "container.plotly_chart" have in common. It is the "_enqueue()" method in streamlit.delta_generator.py, which passes the "element_proto: Message" protobuf message, if I'm not mistaken.

So is it possible that not all resources are released until the main message finishes: await called in streamlit.runtime.runtime.py lines 577-616 (I was inspired by the protocolbuffers/protobuf#5737 thread). So would changing something here can help, or using a different version of protobuf (latest 4.22, steamlit uses protobuf>=3.19, <4)?

I don't have experience with this, so I don't know how to test it, or if what I wrote makes any sense. I would appreciate any comments

[jakubxy08]

A small update, finally after many attempts I was able to test with a newer version of protobuf, but it did not do anything.

In addition, I checked whether deleting proto messages, and intermediate variables will in any way affect memory consumption (I tested on versions streamlit 1.17.0 - 1.21.0 x python 3.8 - 3.10 x windows and Ubuntu) - unfortunately, as before, memory consumption increases steadily.

I have no idea what else could be causing this. Related to this and this (#6354 (comment)), I would like to propose some new enchacement () related to reducing memory consumption for long living apps.

Maybe as more people are interested in it, it will get some higher priority ;)

[vdonato]

Thanks for the investigation on this + the new enhancement issue @jakubxy08!

I'm still trying to track down some memory leaks related to this issue and #6354, but even when those are addressed, I do think that one of the fundamental reasons these are slipping by (both for this issue as well as #6354) is that Streamlit currently isn't designed to support the long-lived script use-case very well.

I do have a workaround for you for this particular issue, though. It turns out the high memory growth from calling the plotly_chart function is coming from our ForwardMsgCache caching large messages being sent to the frontend. In most scripts, this mechanism works quite well for allowing us to avoid repeatedly sending a ton of unchanging data over the wire. The issue with long-running scripts, though, is that cache cleanup happens at the end of a script run after a cached message is deemed stale, which we determine by the number of script runs since the message was first generated. This means that this just never happens if a script is meant to run indefinitely.

We have a hidden config option global.minCachedMessageSize (see config.py) that determines the minimum message size (in bytes) before we try to cache a protobuf sent from server -> frontend. This defaults to 10KB, but setting it to a sufficiently high value that our message caching never kicks in should slow down memory growth for your repro case considerably -- probably to the much slower growth rate observed when only st.write() occurs in the loop body (which will require its own investigation).

[vdonato]

Update: I have a PR (#6617) out fixing the memory leak with our ForwardMsgCache, which I think will be sufficient to close this issue as it gets rid of the memory leak that currently occurs where fwd message cache entries aren't deleted on session shutdown. (Note that even with the fix, the entries won't immediately be deleted when a browser tab is closed as we defer session cleanup when the websocket disconnects in case it was caused by a transient network issue).

There's still the issue of Streamlit not playing very nicely with super long-lived sessions, but I think it'll be more appropriate to discuss that in #6602. In the meantime, I think the config option workaround that I described above should prevent memory growth caused by the ForwardMsgCache in a long-running script.

[vdonato]

Closing with the merge of #6617 + having #6602 be the new place for further discussion of long-lived session support

[lambdaTW]

@vdonato I got the same memory leak issue and traced the PR code, current release, and latest pre-release.

The line of code in your pull request is not present in version 1.29.0 and 1.29.1dev, including the develop branch.

Is it called somewhere I never traced? Or just the release issue?

[ks00x]

I have the same issue I think (version 1.3). The code below is the simplest I could come up with to demonstrate it. With the image shown the memory usage increases rapidly into the GB regime. However, even without the video shown, the memory increases because of the np.random call. Restarting the script causes the memory to become free!

import streamlit as st
import numpy as np
import time

st.button('restart')
show = st.checkbox('show video',value=True)
vid = st.empty()

while True :
    im = np.random.random((500,500))
    if show :
        vid.image(im)        
    time.sleep(0.02)

[vendra]

Having the same issue. When displaying multiple images, page update starts to become slower and slower.