Is memory leaking?

[rodja]

Description

Even an empty auto-index page is leaking memory. This can be observed by regularly getting the process memory via psutil:

from nicegui import ui

def observe_memory_growth() -> None:
    import logging
    import gc
    import os
    import psutil
    from psutil._common import bytes2human

    def get_process_memory() -> int:
        process = psutil.Process(os.getpid())
        mem_info = process.memory_info()
        return mem_info.rss

    async def stats() -> None:
        nonlocal prev_memory
        gc.collect()
        growth = get_process_memory() - prev_memory
        logging.warning(
            f'memory growth: {bytes2human(growth)}, '
            f"now it's {bytes2human(get_process_memory())}"
        )
        prev_memory = get_process_memory()

    prev_memory: int = 0
    ui.timer(5.0, stats)

observe_memory_growth()

ui.run()

The growth is 0 if you do nothing, but if you request the empty index page, the code reports a memory growth.

[falkoschindler]

It's pretty confusing... I created a MemoryMonitor to track the memory and instance counts:

import gc
import os
from collections import Counter
import psutil
from psutil._common import bytes2human

class MemoryMonitor:

    def __init__(self) -> None:
        self.last_mem: int = 0
        self.obj_count: dict[str, int] = {}

    def print_stats(self) -> None:
        gc.collect()
        memory = psutil.Process(os.getpid()).memory_info().rss
        growth = memory - self.last_mem
        print(bytes2human(memory), end=' ')
        print(bytes2human(growth, r"%(value)+.1f%(symbol)s") if growth else '', end=' ')
        counter: Counter = Counter()
        for obj in gc.get_objects():
            counter[type(obj).__name__] += 1
        for cls, count in counter.items():
            prev_count = self.obj_count.get(cls, 0)
            if count != prev_count:
                print(f'{cls}={count} ({count - prev_count:+})', end=' ')
                self.obj_count[cls] = count
        self.obj_count = {cls: count for cls, count in self.obj_count.items() if count > 0}
        print()
        self.last_mem = memory

It can be used like this:

from nicegui.debug import MemoryMonitor

monitor = MemoryMonitor()
ui.timer(5.0, monitor.print_stats)

My observations for a page with 10,000 labels: Each new client connection causes a few extra MB of memory. The number of object instances also increases significantly. When closing a connection, the instance counts drop, but the memory doesn't decrease. This might be due to internal workings of Python's Memory Allocator. So to me it doens't necessarily looks like a general leak in NiceGUI, at least for a simple "Hello World" (x10,000) app.

When analyzing our website, the main page seems to cost around 10MB per client. When opening a bunch of connections, each consumes 10MB. When closing them, lots of instances are removed, but the memory doesn't decrease. But when connecting new clients again, they consume much less memory. This supports the theory that the Memory Allocator somehow keeps the memory allocated for later.

If the memory usage of nicegui.io keeps growing, this could either be due to lots of open connections, or the memory gets too fragmented. Or there is still a subtle leak somewhere...

[rodja]

The fact that Python does not hand back once claimed memory makes it a bit harder to investigate potential issues. But your MemoryMonitor with instance counting is a nice approach. Could we put the data into a ui.echart? And maybe host it under nicegui.io/memory so we can investigate in the real world...

[rodja]

An E-Chart "ThemeRiver" would be a great way to visualize: https://echarts.apache.org/examples/en/editor.html?c=themeRiver-basic

[rodja]

I just saw the log:

2023-10-16T14:27:21Z app[7811077c575158] lhr [info]Traceback (most recent call last):
2023-10-16T14:27:21Z app[7811077c575158] lhr [info]  File "/app/nicegui/events.py", line 401, in wait_for_result
2023-10-16T14:27:21Z app[7811077c575158] lhr [info]    await result
2023-10-16T14:27:21Z app[7811077c575158] lhr [info]  File "/app/website/demo.py", line 59, in handle_intersection
2023-10-16T14:27:21Z app[7811077c575158] lhr [info]    window.remove(spinner)
2023-10-16T14:27:21Z app[7811077c575158] lhr [info]  File "/app/nicegui/element.py", line 414, in remove
2023-10-16T14:27:21Z app[7811077c575158] lhr [info]    self.client.remove_elements(elements)
2023-10-16T14:27:21Z app[7811077c575158] lhr [info]  File "/app/nicegui/client.py", line 172, in remove_elements
2023-10-16T14:27:21Z app[7811077c575158] lhr [info]    del self.elements[element_id]
2023-10-16T14:27:21Z app[7811077c575158] lhr [info]        ~~~~~~~~~~~~~^^^^^^^^^^^^
2023-10-16T14:27:21Z app[7811077c575158] lhr [info]KeyError: 260

If we exit the loop due to key error we may miss deleting some objects. Or not?

[falkoschindler]

@rodja Yes, that sounds plausible. I just wonder why the element is missing in the dictionary. I'll try to reproduce it locally.

[falkoschindler]

So far no luck reproducing it. But looking into the traceback more closely, it seems like remove_elements is called with only the spinner, so there are no more elements waiting for deletion afterwards. If this isn't happening constantly, I don't think this is our memory leak.

[rodja]

The analysis made on #2502 did not discover a memory leak. But on our servers for https://nicegui.io we occasionally still see

ERROR:nicegui:'2ff7cf2e-1146-4b0b-aafa-86a9b0b59352'
Traceback (most recent call last):
  File "/app/nicegui/background_tasks.py", line 52, in _handle_task_result
    task.result()
  File "/app/nicegui/client.py", line 257, in handle_disconnect
    self.delete()
  File "/app/nicegui/client.py", line 316, in delete
    del Client.instances[self.id]
        ~~~~~~~~~~~~~~~~^^^^^^^^^
KeyError: '2ff7cf2e-1146-4b0b-aafa-86a9b0b59352'

[Ezbaze]

I'm not sure if this is the cause of the issue but BaseHTTPMiddleware might be at fault? ( Side note: encode/starlette#1843 (comment) )

It's used in a few places: https://github.com/search?q=repo%3Azauberzeug%2Fnicegui%20BaseHTTPMiddleware&type=code

[falkoschindler]

Interesting, @Ezbaze! Can you elaborate? It is marked as "potential issue"... What makes you think it is causing a leak in the NiceGUI library? Can we somehow reproduce a growth in memory consumption?