Memory leak while running

[phucnguyenit]

Memory leak while running. On bootstrapping, nodes memory usage is 40MB, but memory usage is increased all the time.

Example, memory usage after 2 days

CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
2d0e7b1ee7c4        node1               0.46%               1.866GiB / 3.699GiB   50.43%              9.34GB / 8.39GB     6.57MB / 9.33GB     12

For profiling of go process, we can start node with cmd pump in tool/pump/main.go, with parameters in *.yaml file

Debug:
  Port: ":6000"

Example config file: (change *.txt to *.yaml)

pump.yaml.txt

Then run cmd to get pdf file:

go tool pprof -pdf http://localhost:6000/debug/pprof/heap

Example:
profile001.pdf

[phucnguyenit]

@thientn @vnn2509 : need to investigate where memory leak in nodes

[phucnguyenit]

Thanks for node metric from @dqnguyen210, we can see in RPC

curl -X POST http://13.212.9.216:8545 -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","method":"node_metrics","params":[],"id":3}'

{
    "jsonrpc": "2.0",
    "id": 3,
    "result": {
        "system/cpu/goroutines": {
            "value": 1854
        },
        "system/cpu/procload": {
            "value": 10
        },
        "system/cpu/sysload": {
            "value": 7
        },
        "system/cpu/syswait": {
            "value": 0
        },
        "system/cpu/threads": {
            "value": 7
        },
        "system/disk/readbytes": {
            "count": 0
        },
        "system/disk/readcount": {
            "15m.rate": 0,
            "1m.rate": 0,
            "5m.rate": 0,
            "count": 0,
            "mean.rate": 0
        },
        "system/disk/readdata": {
            "15m.rate": 0,
            "1m.rate": 0,
            "5m.rate": 0,
            "count": 0,
            "mean.rate": 0
        },
        "system/disk/writebytes": {
            "count": 0
        },
        "system/disk/writecount": {
            "15m.rate": 0,
            "1m.rate": 0,
            "5m.rate": 0,
            "count": 0,
            "mean.rate": 0
        },
        "system/disk/writedata": {
            "15m.rate": 0,
            "1m.rate": 0,
            "5m.rate": 0,
            "count": 0,
            "mean.rate": 0
        },
        "system/memory/allocs": {
            "15m.rate": 18433.218333802593,
            "1m.rate": 20859.565880058628,
            "5m.rate": 20083.51531076742,
            "count": 13736932,
            "mean.rate": 19311.580438461286
        },
        "system/memory/frees": {
            "15m.rate": 17169.907722092532,
            "1m.rate": 22110.06709174081,
            "5m.rate": 19992.57501566034,
            "count": 13620463,
            "mean.rate": 19147.846741229634
        },
        "system/memory/held": {
            "value": 41025536
        },
        "system/memory/pauses": {
            "15m.rate": 14810.642853638557,
            "1m.rate": 3619.334190942926,
            "5m.rate": 6080.553373012645,
            "count": 2862928,
            "mean.rate": 4024.746094522273
        },
        "system/memory/used": {
            "value": 28359952
        }
    }
}

[lewtran]

close due to #20