glusterfs process is consuming high memory

[sryeddla]

Description of problem:
We have installed glusterfs in two of our servers to replicate data. The two servers are part of active-passive cluster. On the active cluster node, glusterfs process memory consumption is steadily increasing and is not releasing memory. We are restarting glusterd service to bring the memory usage down but again the memory usage is steadily increasing.

The exact command to reproduce the issue:
output of top command:
KiB Mem : 32946332 total, 5793052 free, 20681404 used, 6471876 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 11290772 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15166 root 20 0 17.992g 0.017t 7144 S 13.0 56.4 85:38.15 glusterfs
1884 root 20 0 1709008 114768 5452 S 9.0 0.3 227:48.49 glusterfsd

- The output of the gluster volume info command:
Volume Name: ma******me
Type: Replicate
Volume ID: c3c3fec2-d178-4a83-a3ee-979a93dc7bb0
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: plsl-A:/glusterfs1/vmail
Brick2: pll-B:/glusterfs2/vmail
Options Reconfigured:
performance.client-io-threads: off
nfs.disable: on
storage.fips-mode-rchecksum: on
transport.address-family: inet

- The operating system / glusterfs version:
OS: ubuntu 16.04
gluster version: glusterfs 7.7

Please check and let us know of possible solution.

[mohit84]

Are you sure you have captured statedump at the time of consuming high memory , The statedump is showing memory consumption is nominal(38301376 38M)
sum=0;for val in grep -iw size gluster.dump.txt| grep -v "size=0" | awk -F "=" '{print $2}'; do sum=$((sum+val));done; echo $sum
38301376

[sryeddla]

Okay. That was the dump of the replication volume. Have generated a dump for the glusterfs process, but the file size is huge (2.8 GB) and can't upload here. Any other way that I send over to you?

[mohit84]

Can you try to disable open-behind and then observe memory consumption for a client process.

gluster volume set <volname> open-behind off

[sryeddla]

After turning off open-behind option, the memory usage of glusterfs process didn't increase. Thank you. As it is now turned off, what would be the consequence of that. I see that it is used for sending success notification to application read request, but want to know if there is anything more that we need to be aware of.

[xhernandez]

@sryeddla what operations you do on the volume ? a simple test doesn't seem to cause this problem, so I need to know what is your workload to try to reproduce the issue.

[sryeddla]

We are using gluster to replicate the contents of the mailboxes between both of our active-passive mail servers. On an average we would receive 1000 emails each day. We observed this issue, when there is continuously new data that needs to be replicated.

[xhernandez]

@sryeddla I've found the issue. There's a patch that should fix it. Are you able to verify it in a testing environment ?

[sryeddla]

@xhernandez good to know that patch is released to fix the issue. We do have a lower environment but due to some factors we wouldn't be able to install/test this patch now. However, in future if it gets bundled in next version, we would be able to install/test it. Thank you.

[sryeddla]

@xhernandez : Are there any known compatibility issues between glusterfs 7.7 and McAfee ePO agent 5.6.4.151. When we installed ePO agent, the overall memory usage on the active server is steadily increasing but neither the gluster process nor the ePO or any other processes on the system doesn't show a proportionate increase of memory usage. It appears like there is a memory leak but cant tell which one is triggering it.

[sryeddla]

correction : epo version is 5.10.0

[xhernandez]

@xhernandez : Are there any known compatibility issues between glusterfs 7.7 and McAfee ePO agent 5.6.4.151. When we installed ePO agent, the overall memory usage on the active server is steadily increasing but neither the gluster process nor the ePO or any other processes on the system doesn't show a proportionate increase of memory usage. It appears like there is a memory leak but cant tell which one is triggering it.

I'm not aware of any issue.

How are you measuring the used memory if you say that there isn't any process consuming a proportional amount of memory ?

[sryeddla]

the top command, which show the overall memory usage along with %memory used by individual processes. We see an increase in the total memory used on the system, but the %MEM of the individual process on the machine doesnt show a proportionate increase. The total used memory is also verified by free command which shows same as top, and the memory usage of individual process also verified by ps command.

[xhernandez]

@sryeddla then the memory is probably used by the kernel cache. If that's true, it's not a real memory leak. Whenever the kernel will need more memory, it will take it from the cache.

You can easily check that from the output of 'free' command. The memory that appears under 'buff/cache' column is in use but it can mostly be reused by the system when necessary. It shouldn't cause any issue.

[sryeddla]

Actually buffer/cache memory of the system remained almost same at 9 GB when the used memory increased from 3GB to 16GB. Even if we consider that Kernel is taking the memory, problem starts arising when it continously needs additional memory without releasing the allocated memory. Sooner or later, system will be exhaust of available memory and start experiencing performance degradation. Moreover, observed this behaviour when there is both gluster and ePO on the box. There are other systems which have ePO but not gluster and they dont expereince the kind of memory increase that we are seeing on these boxes. We are simultaneously checking with McAfee if they have an answer for this.

[stale]

Thank you for your contributions.
Noticed that this issue is not having any activity in last ~6 months! We are marking this issue as stale because it has not had recent activity.
It will be closed in 2 weeks if no one responds with a comment here.

[stale]

Closing this issue as there was no update since my last update on issue. If this is an issue which is still valid, feel free to open it.