Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Do not release memory, there may be a memory leak #623

Closed
zyTvT opened this issue Nov 19, 2020 · 11 comments
Closed

Do not release memory, there may be a memory leak #623

zyTvT opened this issue Nov 19, 2020 · 11 comments

Comments

@zyTvT
Copy link

zyTvT commented Nov 19, 2020

Hi，
When I run lsyncd, I found that RSS is growing with the sending of files, and it will not decrease. Is this normal?
@fabrizio2210
Copy link

Hi,
I have a similar behavior. In the last months, after some weeks I have to restart lsyncd because it has a big memory consumption.
The affected package is lsyncd 2.2.3-1 armhf.

image
In this image, I had to restart on 27th Jan.

In my opinion the problem is in the heap memory, where the references to the transmitted files are stored.
For example I have webserver with some logs about the accesses. The most frequent mentioned files are the logs.

Way to reproduce and show:

# get PID of the process, in my case is 261

# size of the whole process is 169MB
 PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                              
 261 root      20   0  169192 147512    992 R  98.3  15.5   2648:26 lsyncd 

# get heap memory size (166MB) and address
root@d24201d1dd09:/# fgrep -A 1 '[heap]' /proc/261/smaps
00f7e000-0b286000 rw-p 00000000 00:00 0          [heap]
Size:             166944 kB

# Dump the heap memory
root@d24201d1dd09:/# gdb --pid [pid]
(gdb) dump memory /tmp/output 0x00f7e000 0x0b286000

# read the content and sort by occurence
root@d24201d1dd09:/# strings /tmp/output | sort | uniq -c | sort > /tmp/sorted 

# In my top 15 string with '/' I have the logs
root@d24201d1dd09:/# grep / /tmp/sorted | tail -15
   3591         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/monthly.json
   3598         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/visitors.json
   3953         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/totals.json
   4079         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/daily.json
   4616 s/popula
   5693 opt/data/log
   5862 mao8mo:/opt/
   5872 /logs/popula
  11364         /venE
  12083 s/po
  72145 /cachM'
  73486 /dat
 141996 on/UD
 142191 opt/D
 264074 /venE

# filtering for that folder I find
root@d24201d1dd09:/# grep /opt/data/logs/ /tmp/sorted | tail -15
    258         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/.daily.json.2zHQSD
    259         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/.visitors.json.1BI81P
    261         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/.monthly.json.w7WjpB
    264         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/.totals.json.wpXpIZ
    264         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/.visitors.json.gZnrAk
    265         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/.totals.json.MhqprR
    265         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/.visitors.json.oNqR39
    269         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/.visitors.json.krc2av
    274         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/.totals.json.UBLSfJ
    357         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/monthly.jso
    361         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/visitors.js
   3591         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/monthly.json
   3598         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/visitors.json
   3953         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/totals.json
   4079         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/daily.json

# Content of the folder
root@d24201d1dd09:/# ls -la /opt/data/logs/popularity/
total 24
drwsrwsr-x 2 _apt input 4096 Jan 31 10:27 .
drwsrwsr-x 3 _apt input 4096 Jan 27 10:06 ..
-rw-rw-r-- 1 _apt input  537 Jan 31 10:27 daily.json
-rw-rw-r-- 1 _apt input   63 Jan 31 10:27 monthly.json
-rw-rw-r-- 1 _apt input   83 Jan 31 10:27 totals.json
-rw-rw-r-- 1 _apt input 1081 Jan 31 10:27 visitors.json

So, it seems lsyncd memorizes all the files that it handled without removing them from memory.

Is this an intended behavior?

@zyTvT
Copy link
Author

zyTvT commented Feb 1, 2021

Hi,
I have a similar behavior. In the last months, after some weeks I have to restart lsyncd because it has a big memory consumption.
The affected package is lsyncd 2.2.3-1 armhf.

image
In this image, I had to restart on 27th Jan.

In my opinion the problem is in the heap memory, where the references to the transmitted files are stored.
For example I have webserver with some logs about the accesses. The most frequent mentioned files are the logs.

Way to reproduce and show:

# get PID of the process, in my case is 261

# size of the whole process is 169MB
 PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                              
 261 root      20   0  169192 147512    992 R  98.3  15.5   2648:26 lsyncd 

# get heap memory size (166MB) and address
root@d24201d1dd09:/# fgrep -A 1 '[heap]' /proc/261/smaps
00f7e000-0b286000 rw-p 00000000 00:00 0          [heap]
Size:             166944 kB

# Dump the heap memory
root@d24201d1dd09:/# gdb --pid [pid]
(gdb) dump memory /tmp/output 0x00f7e000 0x0b286000

# read the content and sort by occurence
root@d24201d1dd09:/# strings /tmp/output | sort | uniq -c | sort > /tmp/sorted 

# In my top 15 string with '/' I have the logs
root@d24201d1dd09:/# grep / /tmp/sorted | tail -15
   3591         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/monthly.json
   3598         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/visitors.json
   3953         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/totals.json
   4079         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/daily.json
   4616 s/popula
   5693 opt/data/log
   5862 mao8mo:/opt/
   5872 /logs/popula
  11364         /venE
  12083 s/po
  72145 /cachM'
  73486 /dat
 141996 on/UD
 142191 opt/D
 264074 /venE

# filtering for that folder I find
root@d24201d1dd09:/# grep /opt/data/logs/ /tmp/sorted | tail -15
    258         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/.daily.json.2zHQSD
    259         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/.visitors.json.1BI81P
    261         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/.monthly.json.w7WjpB
    264         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/.totals.json.wpXpIZ
    264         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/.visitors.json.gZnrAk
    265         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/.totals.json.MhqprR
    265         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/.visitors.json.oNqR39
    269         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/.visitors.json.krc2av
    274         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/.totals.json.UBLSfJ
    357         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/monthly.jso
    361         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/visitors.js
   3591         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/monthly.json
   3598         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/visitors.json
   3953         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/totals.json
   4079         gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/daily.json

# Content of the folder
root@d24201d1dd09:/# ls -la /opt/data/logs/popularity/
total 24
drwsrwsr-x 2 _apt input 4096 Jan 31 10:27 .
drwsrwsr-x 3 _apt input 4096 Jan 27 10:06 ..
-rw-rw-r-- 1 _apt input  537 Jan 31 10:27 daily.json
-rw-rw-r-- 1 _apt input   63 Jan 31 10:27 monthly.json
-rw-rw-r-- 1 _apt input   83 Jan 31 10:27 totals.json
-rw-rw-r-- 1 _apt input 1081 Jan 31 10:27 visitors.json

So, it seems lsyncd memorizes all the files that it handled without removing them from memory.

Is this an intended behavior?

Well,It;s an intended behavior,The memory lsyncd used will continue to grow with the number of files sent,To solve this problem,you can use collectgarbage() After rsync send files.

Thanks.

@fabrizio2210
Copy link

fabrizio2210 commented Feb 1, 2021 via email

@zyTvT
Copy link
Author

zyTvT commented Feb 2, 2021

Is something that I have to put in the configuration file? Something like "inlets" at layer 1? I have found no examples about that. Il lun 1 feb 2021, 02:28 zyTvT notifications@github.com ha scritto:
…
Hi, I have a similar behavior. In the last months, after some weeks I have to restart lsyncd because it has a big memory consumption. The affected package is lsyncd 2.2.3-1 armhf. [image: image] https://user-images.githubusercontent.com/34220075/106380866-5a4a9600-63b5-11eb-9694-5f19a2643b19.png In this image, I had to restart on 27th Jan. In my opinion the problem is in the heap memory, where the references to the transmitted files are stored. For example I have webserver with some logs about the accesses. The most frequent mentioned files are the logs. Way to reproduce and show: # get PID of the process, in my case is 261 # size of the whole process is 169MB PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 261 root 20 0 169192 147512 992 R 98.3 15.5 2648:26 lsyncd # get heap memory size (166MB) and address @.:/# fgrep -A 1 '[heap]' /proc/261/smaps 00f7e000-0b286000 rw-p 00000000 00:00 0 [heap] Size: 166944 kB # Dump the heap memory @.:/# gdb --pid [pid] (gdb) dump memory /tmp/output 0x00f7e000 0x0b286000 # read the content and sort by occurence @.:/# strings /tmp/output | sort | uniq -c | sort > /tmp/sorted # In my top 15 string with '/' I have the logs @.:/# grep / /tmp/sorted | tail -15 3591 gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/monthly.json 3598 gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/visitors.json 3953 gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/totals.json 4079 gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/daily.json 4616 s/popula 5693 opt/data/log 5862 mao8mo:/opt/ 5872 /logs/popula 11364 /venE 12083 s/po 72145 /cachM' 73486 /dat 141996 on/UD 142191 opt/D 264074 /venE # filtering for that folder I find @.:/# grep /opt/data/logs/ /tmp/sorted | tail -15 258 gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/.daily.json.2zHQSD 259 gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/.visitors.json.1BI81P 261 gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/.monthly.json.w7WjpB 264 gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/.totals.json.wpXpIZ 264 gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/.visitors.json.gZnrAk 265 gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/.totals.json.MhqprR 265 gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/.visitors.json.oNqR39 269 gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/.visitors.json.krc2av 274 gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/.totals.json.UBLSfJ 357 gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/monthly.jso 361 gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/visitors.js 3591 gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/monthly.json 3598 gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/visitors.json 3953 gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/totals.json 4079 gravervisaasyncnjq98bs81i8r6ooixd8hl8ic0k3i2k4zxkq9geama9vimao8mo:/opt/data/logs/popularity/daily.json # Content of the folder @.:/# ls -la /opt/data/logs/popularity/ total 24 drwsrwsr-x 2 _apt input 4096 Jan 31 10:27 . drwsrwsr-x 3 _apt input 4096 Jan 27 10:06 .. -rw-rw-r-- 1 _apt input 537 Jan 31 10:27 daily.json -rw-rw-r-- 1 _apt input 63 Jan 31 10:27 monthly.json -rw-rw-r-- 1 _apt input 83 Jan 31 10:27 totals.json -rw-rw-r-- 1 _apt input 1081 Jan 31 10:27 visitors.json So, it seems lsyncd memorizes all the files that it handled without removing them from memory. Is this an intended behavior? Well,It;s an intended behavior,The memory lsyncd used will continue to grow with the number of files sent,To solve this problem,you can use collectgarbage() After rsync send files. Thanks. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#623 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIFCQK7OHZYL63EP4TEN2RLS4X7URANCNFSM4T3LSFDA .

Try to add collectgarbage in lsyncd.lua ：）

@zyTvT zyTvT closed this as completed Feb 19, 2021
@fabrizio2210
Copy link

Just a question about new releases.
If I were able to create a patch to reduce the memory consumption, it would be merged in master branch?
It seems that the deploy of new releases stopped in 2018, although there are new PR.

@axkibe
Copy link
Collaborator

axkibe commented Feb 20, 2021

No, I stopped actively working on Lsyncd.. I welcome forks tough...

Had you something in mind changing the datastructures or just call the Lua garbage collector manually ever so often?

I don't think there is a memory leak, but I didn't bother with the GC and let Lua handle it, so dunno how far it allows it to grow..

@fabrizio2210
Copy link

I do not have a clear idea at the moment.
I was thinking about to call GC every now and then (maybe using a counter or setting a threshold on memory).
But if the problem is related to a growing table, the GC is not the solution. I should change the data-structure.

Anyway, knowing that a new release will not land to a distributable package discourages me from continuing.
I would avoid dealing with self-compiled version, I am not good at maintaining them.

@axkibe
Copy link
Collaborator

axkibe commented Feb 20, 2021 •
edited

I am not good at maintaining them.

It's a lot of work ;)

@jackysteps
Copy link

jackysteps commented Apr 20, 2021 •
edited

Try to add collectgarbage in lsyncd.lua ：）

collectgarbage seems not working in my case and the RSS is continuously growing all the time.

Here is what I'm doing, any ideas about this?

collectgarbage('setpause', 100) collectgarbage("collect") collectgarbage("collect")

@fabrizio2210
Copy link

My impression is that it is not matter of Garbage Collector, I think it is how the data is structured.

I haven't verified, but I think there is a table with every files that has changed. The table is monotonically increasing and never emptied.

@jackysteps
Copy link

My impression is that it is not matter of Garbage Collector, I think it is how the data is structured.

I haven't verified, but I think there is a table with every files that has changed. The table is monotonically increasing and never emptied.

New update:

It is like that in my case, inotfify for file-changing are too much and lsyncd can't handle them timely, so these events will be queued, and leads to memory consumption grows always.

Because in my case, the applications are changing some dedicated files too often, it means that there could be tons of events corresponding to the same file that exists in the queue at the same time, and we only care about the latest one, so I'm wondering how can we do some optimizing and do some kinds of clean work for events belong to the same file?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@axkibe @jackysteps @fabrizio2210 @zyTvT