node-agent consumes huge amount of resources

[dahorak]

Several recent versions of tendrl-node-agent constantly consumes huge amount of resources (CPU).

Check the the %CPU column in the following (truncated) output from top (this is one time hit, but it's very similar for long period of time):

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                          
11192 root      20   0  357212  26528   5004 R  91.6  1.4   9:35.22 tendrl-node-age   

You can check it via following commands, which print the CPU and Memory usage of tendrl-node-agent each 5 second:

while true;
  do ps -p $(echo $(ps aux | grep [t]endrl-node-agent | awk '{print $2}') | sed 's/ /,/') -o %cpu,%mem,cmd -h;
  sleep 5;
done

For example (header added manually):

%CPU %MEM CMD
65.5  1.4 /usr/bin/python /usr/bin/tendrl-node-agent
65.6  1.4 /usr/bin/python /usr/bin/tendrl-node-agent
65.5  1.4 /usr/bin/python /usr/bin/tendrl-node-agent
65.5  1.4 /usr/bin/python /usr/bin/tendrl-node-agent
65.6  1.4 /usr/bin/python /usr/bin/tendrl-node-agent
65.5  1.4 /usr/bin/python /usr/bin/tendrl-node-agent
65.5  1.4 /usr/bin/python /usr/bin/tendrl-node-agent

When you stop the tendrl-node-agent service, it unfailingly lower the system load:

# uptime 
 15:35:21 up 33 min,  4 users,  load average: 1.20, 1.10, 0.98

# systemctl stop tendrl-node-agent
...
# uptime 
 15:38:33 up 36 min,  4 users,  load average: 0.05, 0.57, 0.79

This apply on the following versions of tendrl packages:

tendrl-commons-1.2-02_01_2017_18_29_12.noarch
tendrl-gluster-integration-1.2-02_01_2017_14_39_11.noarch
tendrl-node-agent-1.2-02_01_2017_14_21_11.noarch

[dahorak]

It seems to be some regression in last two or three builds, because this is the graph of load of one of our hypervisors, where are running bunch of VMs with Tendrl.

[sankarshanmukhopadhyay]

Is it possible to add detail around specific configuration of the VMs or, instances being used when we observe the resource consumption spike?

Also, do we have a process tree output available?

[dahorak]

The VM have 2GB Ram, 2 CPU cores:

# free -h
              total        used        free      shared  buff/cache   available
Mem:           1.8G        182M        1.0G         24M        615M        1.4G
Swap:            0B          0B          0B

# cat /proc/cpuinfo | grep processor
processor	: 0
processor	: 1

Not sure, if you meant this by process tree?

# pstree -alp 10688
tendrl-node-age,10688 /usr/bin/tendrl-node-agent
  └─{tendrl-node-age},10703

The process tree switches between that above and this below periodically in few second:

# pstree -alp 10688
tendrl-node-age,10688 /usr/bin/tendrl-node-agent
  ├─ansible_exec225,27570 /var/lib/tendrl/.tendrl/node-agent/ansible_exec225a293e-445e-44bc-9e16-ffce627607d0
  │   └─python,27571 /tmp/ansible_KnYYX7/ansible_module_command.py
  └─{tendrl-node-age},10703

It seems like tendrl-node-agent launches each few (5-10) seconds some ansible processes similar to this (not sure, if that's the core of this issue):

root      2475  0.0  0.4 191572  8096 ?        S    19:07   0:00 /usr/bin/python /var/lib/tendrl/.tendrl/node-agent/ansible_exec0938d05f-ba65-4ef5-884f-9ad3a535b0ec
root      2483  0.0  0.1  18908  3224 ?        R    19:07   0:00 /usr/bin/python /tmp/ansible_Kv4wJm/ansible_module_command.py

[nnDarshan]

@dahorak As you have mentioned, Currently the node inventory detail is synced every 3 seconds which is very aggressive, considering they change very rarely. In the previous builds we did not see this as the number of inventory collected was less, in the recent builds we have added inventory collection for new types like service and disk because of which we see this huge surge in cpu utilization. Checking all these inventory details every 3 sec is impractical.
Have sent a PR to address this: #199 . Waiting to see if others are okay with this approach

[r0h4n]

@dahorak we have identified and resolved this one. Lets turn this into a functional test to catch irregular cpu/mem/netio usages?

@nnDarshan please check if we can gauge cpu/mem/netio in travis somehow, file a new bug for this

[r0h4n]

Build fixing this issue: https://copr-be.cloud.fedoraproject.org/results/tendrl/tendrl/epel-7-x86_64/00506669-tendrl-node-agent/ (or latest)

Please reopen if issue persists