Cannot allocate memory when using a large number of hosts

[laszlojau]

First of all, thank you for creating this application. It is really awesome to be able to use blocklists without any extra hardware.

I have recently been having issues with an USG. The application runs out of memory before completion.
I was using v1.2.4.2, and I tried upgrading to v1.2.4.5, but it didn't help. I tried rebooting and re-running the update task, but it ran out of memory again.

I don't know much about Go development, but I tried to do some debugging.

I was using the following command in each test case: time /config/scripts/update-dnsmasq -v
And while the script was running, I was watching memory usage in another session: watch free -h

RAM before running the script (in each test case)

             total       used       free     shared    buffers     cached
Mem:          483M       305M       177M         0B        24M       115M
-/+ buffers/cache:       165M       317M
Swap:           0B         0B         0B

Test case for v1.2.4.5

NOTI[02f]22:05:01.208: Total entries found: 418129
NOTI[030]22:05:01.209: Total entries extracted 362205
NOTI[031]22:05:01.210: Total entries dropped 55924
ERRO[032]22:05:01.213: ReloadDNS(): error: fork/exec /bin/bash: cannot allocate memory

The least amount of free memory after Total entries dropped, right before the error:

             total       used       free     shared    buffers     cached
Mem:          483M       437M        46M         0B        22M       108M
-/+ buffers/cache:       305M       178M
Swap:           0B         0B         0B

Time:

real	1m45.933s
user	1m55.940s
sys	0m19.680s

Test case: GC 20%

I've added 2 lines in main.go in the latest master branch:

   6   │     "runtime/debug"
...
 113   │     debug.SetGCPercent(20)

I've built with make mips and SCP'd the executable to the USG.

It took a longer time to finish, but it never ran out of memory this way.

Sample run with GC 20%:

NOTI[02f]21:53:22.825: Total entries found: 418129
NOTI[030]21:53:22.852: Total entries extracted 362209
NOTI[031]21:53:22.854: Total entries dropped 55920

The least amount of free memory after Total entries dropped:

             total       used       free     shared    buffers     cached
Mem:          483M       429M        54M         0B        24M       114M
-/+ buffers/cache:       290M       192M
Swap:           0B         0B         0B

Memory freed by GC:

             total       used       free     shared    buffers     cached
Mem:          483M       331M       152M         0B        24M       114M
-/+ buffers/cache:       193M       290M
Swap:           0B         0B         0B

Memory before completion:

             total       used       free     shared    buffers     cached
Mem:          483M       377M       106M         0B        24M       114M
-/+ buffers/cache:       238M       245M
Swap:           0B         0B         0B

Time:

real	3m20.654s
user	3m39.210s
sys	0m31.110s

[britannic]

@laszlojau, thank you for taking the time to figure out a workaround for processing arger lists. My preference is not to use the debug.runtime as a fix. You should also be able to achieve the same result without changing the code like this:

GOGC=20 /config/scripts/update-dnsmasq -v

[laszlojau]

Thank you for your help. Sorry, forgot to reply with my findings.

I have changed the scheduled task to include GOGC=20 as per your comment and it has been working perfectly these past few months.

set system task-scheduler task update_blacklists executable path "GOGC=20 /config/scripts/update-dnsmasq-cronjob.sh"

The only caveat is that it throws the following error when you issue the command:

sh: line 0: [: GOGC=20: binary operator expected

Warning: 'GOGC=20' lies outside of /config/scripts/update-dnsmasq-cronjob.sh directory. It will not get preserved during image upgrade.

But it seems like it can be safely ignored, as the generated cron spec is valid:

0 0 */1 * * root GOGC=21 /config/scripts/update-dnsmasq-cronjob.sh 10800

It also works for a USG, the below line has to be changed in config.gateway.json:

"path": "GOGC=20 /config/scripts/update-dnsmasq-cronjob.sh"