Memory leaks in YDK C++ core

[ygorelik]

The issue is reported by Ryan Rozanski rrozansk@cisco.com (CrossWorks team at Cisco)

Expected Behavior

Memory consumption for YDK based application should be stable (going up and down) over big period of time.

Current Behavior

Here is some snipped output from 'top', which shows the memory keeps rising over time:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
  930 root      20   0  230248  65008  16756 S   5.7  0.1   2:36.31 python
  930 root      20   0  236780  71540  16756 R   5.7  0.1   3:02.55 python
  930 root      20   0  239784  74544  16756 S   5.0  0.1   3:14.81 python
  930 root      20   0  242016  76776  16756 S   4.3  0.1   3:23.65 python

We see a larger leak when running python binary, but this script only tests providers.

Python Script

"""
Script showcasing the YDK memory leak for Providers...
"""
from time import sleep
from ydk.path import Repository
from ydk.providers import NetconfServiceProvider
from mem_top import mem_top

REPOSITORY = Repository('/opt/robot/data/cache')

PROVIDER_PARAMETERS = {
    'repo': REPOSITORY,
    'address': '192.168.123.111',
    'username': 'nso',
    'password': 'cisco123',
    'port': '22',
}

MEMORY_PARAMETERS = {
    'limit': 1000,                         # limit of top lines per section
    'width': 1000,                         # width of each line in chars
    'sep': '\n',                           # char to separate lines with
    'refs_format': '{num}\t{type} {obj}',  # format of line in "references" section
    'bytes_format': '{num}\t {obj}',       # format of line in "bytes" section
    'types_format': '{num}\t {obj}',       # format of line in "types" section
    'verbose_types': None,                 # list of types to get their values sorted by repr length
    'verbose_file_name': '/tmp/mem_top',   # name of file to store "verbose_types" in
}

# Print out memory every 'batch' created.
PROVIDER_BATCH = 100

PROVIDER = 0
while True:
    if (PROVIDER % PROVIDER_BATCH) == 0:
        print('MEMORY STATISTICS:\n\n{0}\n\n'.format(mem_top(**MEMORY_PARAMETERS)))

    PROVIDER += 1
    NetconfServiceProvider(**PROVIDER_PARAMETERS)

    sleep(1)  # Slow things down to investigate live.

System Information

YDK-0.7.2 with NSO Service Provider

[ygorelik]

Ran Valgrind Memcheck for YDK core and bundle tests, here is the summary.
YDK core tests

==17153== LEAK SUMMARY:
==17153==    definitely lost: 932 bytes in 18 blocks
==17153==    indirectly lost: 124,146 bytes in 1,068 blocks
==17153==      possibly lost: 0 bytes in 0 blocks
==17153==    still reachable: 73,830 bytes in 22 blocks

YDK bundle tests

==17111== LEAK SUMMARY:
==17111==    definitely lost: 2,503,654 bytes in 1,134 blocks
==17111==    indirectly lost: 519,190 bytes in 3,153 blocks
==17111==      possibly lost: 0 bytes in 0 blocks
==17111==    still reachable: 75,482 bytes in 43 blocks

Full reports:
memcheck_core_tests.txt
memcheck_bundle_tests.txt

The most significant memory leaks reported in ietf_parser.cpp and xml_subtree_codec.cpp.

[ygorelik]

Adding script and memory leak monitoring BEFORE the fix.

Script

#!/usr/bin/python
"""
Script showcasing the YDK memory leak for Netconf Service Provider
GitHub issue #899 (https://github.com/CiscoDevNet/ydk-gen/issues/899)
"""
import logging
import os

from time import sleep
from ydk.path import Repository
from ydk.providers import NetconfServiceProvider

def enable_logging(level):
    log = logging.getLogger('ydk')
    log.setLevel(level)
    handler = logging.StreamHandler()
    formatter = logging.Formatter(("%(asctime)s - %(name)s - %(levelname)s - %(message)s"))
    handler.setFormatter(formatter)
    log.addHandler(handler)

def invoke_netconf_provider():
    repo = Repository('/opt/robot/data/cache')
    NetconfServiceProvider(repo=repo,
                           address='192.168.123.111',
                           username='nso',
                           password='cisco123',
                           port=2022)

if __name__ == "__main__":

    enable_logging(logging.ERROR)

    # Print top table header
    os.system('top -n 1 | grep USER')
    for i in range(20):
        invoke_netconf_provider()
        os.system('top -n 1 | grep test_899.py')
        sleep(1)  # Slow things down to investigate live.

Running results showing steady memory increase (RES)

root@robot-nca-5b5db75f86-k6nxk:/var/log/robot# ./test_899.py
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
13527 root      20   0  188900  23476  16816 S   0.0  0.0   0:00.04 test_899.py
13527 root      20   0  188900  23648  16940 S   0.0  0.0   0:00.07 test_899.py
13527 root      20   0  188992  23740  16940 S   0.0  0.0   0:00.10 test_899.py
13527 root      20   0  188992  23748  16940 S   0.0  0.0   0:00.11 test_899.py
13527 root      20   0  189016  23772  16940 S   0.0  0.0   0:00.13 test_899.py
13527 root      20   0  189060  23780  16940 S   0.0  0.0   0:00.15 test_899.py
13527 root      20   0  189060  23804  16940 S   0.0  0.0   0:00.17 test_899.py
13527 root      20   0  189132  23852  16940 S   0.0  0.0   0:00.20 test_899.py
13527 root      20   0  189132  23852  16940 S   0.0  0.0   0:00.22 test_899.py
13527 root      20   0  189132  23852  16940 S   0.0  0.0   0:00.25 test_899.py
13527 root      20   0  189132  23876  16940 S   0.0  0.0   0:00.27 test_899.py
13527 root      20   0  189132  23892  16940 S   0.0  0.0   0:00.29 test_899.py
13527 root      20   0  189204  23920  16940 S   0.0  0.0   0:00.31 test_899.py
13527 root      20   0  189204  23936  16940 S   0.0  0.0   0:00.33 test_899.py
13527 root      20   0  189204  23936  16940 S   0.0  0.0   0:00.35 test_899.py
13527 root      20   0  189232  23972  16940 S   0.0  0.0   0:00.38 test_899.py
13527 root      20   0  189232  23972  16940 S   0.0  0.0   0:00.40 test_899.py
13527 root      20   0  189232  23988  16940 S   0.0  0.0   0:00.42 test_899.py
13527 root      20   0  189288  24008  16940 S   0.0  0.0   0:00.44 test_899.py
13527 root      20   0  189332  24052  16940 S   0.0  0.0   0:00.47 test_899.py
13527 root      20   0  189332  24064  16940 S   0.0  0.0   0:00.49 test_899.py
13527 root      20   0  189332  24076  16940 S   0.0  0.0   0:00.52 test_899.py
13527 root      20   0  189360  24100  16940 S   0.0  0.0   0:00.54 test_899.py

[ygorelik]

Resolved in release 0.8.3.