Memory leak in pypowerwall proxy

[jasonacox]

From @youzer-name jasonacox/Powerwall-Dashboard#25 (comment)

I use Grafana and InfluxDB to monitor my Pi4's vital stats. Recently I noticed that my memory used was steadily climbing. When I checked the Task Manager in the Pi4 GUI I found that Python3 was using a lot of memory in the "VM-Size" column. After rebooting the Pi the total memory use dropped down a lot.

I've been watching the memory usage since my last reboot about 6 days ago. While pypowerwall isn't the only thing using more memory, restarting my pypowerwall container immediately dropped the total memory usage by about 11%. The memory use by Python3 dropped from over 400MB to < 20MB. Based on the numbers I saw before the reboot, I expect that if I had let it go longer, the Python3 memory utilization would have just continued to climb.

Is this normal or could there be a memory leak in pypowerwall or Python3?

This image shows the last 7 days. Towards the left you can see the before/after reboot. At the very right edge you can see the restart of the pypowerwall docker container.

Originally posted by @youzer-name in jasonacox/Powerwall-Dashboard#25

[jasonacox]

It seems very compelling that there is a leak. It should grow as it buffers API calls but that should level off and not continue to grow as your graph shows. The pypowerwall proxy uses ThreadingHTTPServer. I did some quick research and believe there are conditions with python 3.7+ with ThreadingHTTPServer that can cause this memory leak. I have put in a recommended fix and I'm running some tests before releasing.

[youzer-name]

Pypowerwall version:

{"version": "22.9.1 bce98cad", "vint": 220901}

python --version = 2.7.16

python3 --version = 3.7.3

Per your reply in the Powerwall-Dashboard area, I'm trying out the t10 version. I'll check back on it later today, but it might take a few days to notice if there is anything going on with the memory use.

[jasonacox]

For others following, as @youzer-name mentions, If you want to test the patch - Edit powerwall.yml and change this line:

# from
    pypowerwall:
        image: jasonacox/pypowerwall:latest

# to
    pypowerwall:
        image: jasonacox/pypowerwall:0.4.0t10

Delete and pull down the t10 proxy:

docker stop pypowerwall

# delete pypowerwall
docker rm pypowerwall
docker images | grep pypowerwall | awk '{print $3}' | xargs docker rmi -f

# rebuild
docker-compose -f powerwall.yml up -d

[jasonacox]

Not 100% conclusive yet, but memory footprint is staying fairly stead with the patch (more than it was before).

Using a cron with this to push into InfluxDB/Grafana:

#!/bin/bash

# Push memory usage of pypowerwall to influxdb
PID=`ps aux | grep server.py | grep -v grep | awk '{print $2}'`
MEM=`sudo pmap ${PID} | tail -1 | awk '{print $2}' | cut -f1 -dK`

# Send
curl -i -XPOST 'http://localhost:8086/write?db=powerwall' --data-binary "pypowerwall mem=${MEM}"

[youzer-name]

Likewise. I've been running the new version for about 8 hours and it looks good.

I adjusted the Y axis on the chart below to make the trends more visible. The dip in the middle is a container restart using the old pypowerwall version. The smaller dip to the right on 5/5 at about 0800 is the installation of the new version. It stayed really flat until noon, jumped a little and then was really flat again after noon. One of my cron backup jobs runs at noon, so that may explain that increase. I also see something going on each night at midnight which is when my other cron backups run. It looks like there's a dip and then a jump in memory use around midnight each night.

[jasonacox]

Nice! Thanks @youzer-name. So far, stability seems to be good with the patch (no side effects) at least from my observations. Let me know if you see anything odd. If it hold good for another ~4 hours, I'll merge this change and push out t10 as the latest for new installs/upgrades for everyone.

[youzer-name]

I haven't seen any issues. I have alerts set up that notify me if the Powerwall data drops out for more than a minute and they haven't gone off, so the whole system seems to be stable. (I was originally connected via WiFi and had lots of dropouts so needed to keep a close eye on the data stream... I'm now using Ethernet but the alerts are still set up).

Here is the last 7 days memory use. The vertical line is when I installed the new version of pyPowerwall.

[jasonacox]

Nice! Thanks @youzer-name ! It's clear that the update made a difference. Thanks for reporting this and helping get it fixed!

My 24-hr graph showing the natural but small spikes and memory collections as different APIs are processed over time:

Like you, I had to switch from WiFi to hardwire ethernet. Even with strong WiFi signal reported by the Powerwall, it would see 10-20m dropouts all the time. Now it is solid.

Thanks again!

[jasonacox]

To others using https://github.com/jasonacox/Powerwall-Dashboard and finding this who need to apply the fix, you can use the upgrade.sh script or follow these manual steps:

Ensure your powerwall.yml has this (pulling latest):

    pypowerwall:
        image: jasonacox/pypowerwall:latest

Delete the old and pull the upgraded pypowerwall proxy container:

docker stop pypowerwall

# delete pypowerwall
docker rm pypowerwall
docker images | grep pypowerwall | awk '{print $3}' | xargs docker rmi -f

# rebuild
docker-compose -f powerwall.yml up -d

[jasonacox]

pyPowerwall Proxy now also provide a memory stat in the /stats call:

"mem": 37120

I'm seeing fairly consistent memory usage an no leaks. Closing this out for now, but will re-open if there are any reports of memory leak issues.