[Bug]: 6.2 large memory usage exit code 137

[Mattheinrichs]

Controller Version

6.2.0.12

Describe the Bug

6.2 is using up a lot more memory than previous versions and is being OOM killed. Note that I'm using the J9 builds.

Yes - I realize this may well be a TPLink problem not a problem with this (excellent) Docker container. Listing it here for visibility and for whatever can be done on the Docker config side.

When I first started 6.2 my docker config included:
mem_limit: 768m
memswap_limit: 768m
This worked well with 6.1. It dropped me into a restart loop when I upgraded to 6.2. (exit code 137)

I removed the limits completely and the container started successfully. However - about 8-10 hours later, the container was again killed (OOM). (exit code 137)

(fwiw - I attempted a rollback to 6.1 by removing the LAST_VER file, but was unsuccessful. And broke my install.)

Expected Behavior

I would expect to not require more memory for the new version, or at minimum, not having a rapidly growing memory footprint.

Steps to Reproduce

Start a beta 6.2 J9 image on your docker server.

How You're Launching the Container

services:
  omada-sdn:
    container_name: omada
    image: mbentley/omada-controller:beta-6.2-openj9
    ulimits:
      nofile:
        soft: 4096
        hard: 8192
    stop_grace_period: 60s
    environment:
      - PUID=508
      - PGID=508
      - TZ=America/Chicago
      - MANAGE_HTTP_PORT=8088
      - MANAGE_HTTPS_PORT=8043
      - PORTAL_HTTP_PORT=8088
      - PORTAL_HTTPS_PORT=8043
      - PORT_APP_DISCOVERY=27001
      - PORT_DISCOVERY=29810
      - PORT_MANAGER_V1=29811
      - PORT_ADOPT_V1=29812
      - PORT_UPGRADE_V1=29813
      - PORT_MANAGER_V2=29814
      - PORT_TRANSFER_V2=29815
      - PORT_RTTY=29816
      - PORT_DEVICE_MONITOR=29817
      - SHOW_SERVER_LOGS=true
      - SHOW_MONGODB_LOGS=false
      - SSL_CERT_NAME="tls.crt"
      - SSL_KEY_NAME="tls.key"
    network_mode: host
    # mem_limit: 768m
    # memswap_limit: 768m
    volumes:
      - omada-data:/opt/tplink/EAPController/data
      - omada-logs:/opt/tplink/EAPController/logs
    restart: unless-stopped
volumes:
  omada-data: null
  omada-logs: null
networks: {}

Container Logs

exit with code 137

MongoDB Logs

Additional Context

No response

[mbentley]

I haven't yet had a chance to look at this but I do want to compare usage between 6.1 and 6.2 along with the OpenJDK and OpenJ9 versions just to narrow down if there is a specific combination or what.

[mbentley]

I've started a test by starting 4 containers with combinations of OpenJDK vs OpenJ9 and 6.1 vs beta-6.2. At the moment, I have only just started the containers, not even done any configuration, literally just starting the containers to get a baseline first.

Not really surprising to initially see that the OpenJDK versions are using more memory than the OpenJ9 versions but this baseline of just the last 30 minutes so far is indicating that the 6.2 versions are using less memory out of the box so it might be either a configuration or a feature that may be causing the memory usage increase.

Any thoughts on what sort of configuration within the controller you may have that could be unique? What sort of devices (router, switches, APs, etc) are connected in your setup?

Another thing that might be helpful as well would be posting on the beta thread of the TP-Link forums to mention this as it may require TP-Link support and engineering to help figure out what might be the trigger.

[Kunalb963]

[I suggest/guess(till now what actually i have noticed) that UPLOAD not showing at all and DOWNLOAD rate it mismatches for all devices except for hard wired devices , previously (before 6.1.0.19) not showing CORRECTLY in client previously image down below. Still having some issues on Tx and Rx rate it mismatches for all devices except for hard wired devices (but now some time hard wired devices get the same issue). also they have enabled health info of devices of Tplink's Omada some time it worked very weirdly. So as the time passed the memory consumption increases rapidly/gradually if you haven't set the memory limit but if set container crashes.
Suggestion: optimising the Health info and Rx and Tx in topology and in client section rates may be decrease the memory usage.

](#742 (comment))

[Mattheinrichs]

Thanks for testing! My setup is pretty simple - router (er605), switch(SG2008), two APs(EAP660HDs) and ~100 wireless clients.
I need to rebuild my setup right now (I broke it with my previous upgrade/downgrade AND discovered all of my backups were zero-length...). Once I do - I'll try to get some more test data.

[mbentley]

Yeah, so it seems pretty clear that it needs a configured environment as the memory usage has been stable here on the clean installs for the past two days:

You can see my actual controller that runs my home environment there as well for comparison, currently it's on 6.1. My current test environment is pretty limited as I only have some old EAPs to test with and that might not trigger anything but I'll give it a shot and see where that gets me with 6.1 and 6.2. Will probably have to let each version run for a couple days in a configured state to see if there is anything noteworthy happening.

[mbentley]

Well, after doing some tests on 6.1 and the 6.2 beta, I can't replicate it in my little test environment which I imagine is due to either a lack of hardware or clients to trigger it. I've posted on the TP-Link forums about what could be provided to give information to help them troubleshoot on their end. Excessive memory usage would also be a big deal for low powered devices as well, especially where they are not able to just increase the available memory.

[mbentley]

One thing that I think would be interesting to try @Mattheinrichs would be to change the JVM args for -Xmx considering that the memory limits that you had set previously were lower than the default to see if it's just a mismatch seeing as the default -Xmx value is 1024m. I've added a section to the README about this just now and there is also a little blurb in the known issues.

The situation that @Kunalb963 was showing is a bit different as they had a much larger container memory limit which is what leaves me a bit puzzled since the default -Xmx arg should be fine I would think and the OOM kill might be related but somewhat different on how/when it's getting killed.