Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Filter directory filling up with old filter files #3567

Closed
gabrielwhite opened this issue Sep 8, 2021 · 10 comments
Closed

Filter directory filling up with old filter files #3567

gabrielwhite opened this issue Sep 8, 2021 · 10 comments
Assignees
@EugeneOne1
Labels
bug P3: Medium
Milestone
v0.107.0

Comments

@gabrielwhite
Copy link

gabrielwhite commented Sep 8, 2021 •
edited
Loading

Issue Details

  • Version of AdGuard Home server:
    • v0.107.0-a.155+b77ceaaf
  • How did you install AdGuard Home:
    • docker
  • CPU architecture:
    • ARM64
  • Operating system and version:
    • Armbian 21.08.1 Buster with Linux 5.10.60-sunxi64

Expected Behavior

That old filter files are deleted automatically

Actual Behavior

I got a "disk full" warning from my OrangePi One Plus and found that the filters directory had 22GB of files in it across more than 100 files in dockers/adguardhome/data/work/data/filters

I have 2 filters enabled, which total around 60MB.

I deleted all the files in the directory, reinstalled the filters. This issue is likely accumulated over time earlier than the current version as there were many files dating back at least weeks.

Subsequently when I check for updates one of the filters generates an error shown in a red toast:

Error: control/filtering/refresh | <html> ,<head><title>502 Bad Gateway</title></head>, ,<body> ,<center><h1>502 Bad Gateway</h1></center>, ,<hr />,<center>openresty</center>, </body>, </html> | 502

This error doesn't show on the other adguard server I use with the same configuration, but running v0.106.3.

When I look in the filter directory I see a new file created without a "txt" extension.

-rw-r--r-- 1 root users  63M Sep  8 09:55 1631105697.txt
-rw-r--r-- 1 root users 746K Sep  8 09:56 1631105792.txt
-rw-r--r-- 1 root users  63M Sep  8 10:01 2463411087
-rw-r--r-- 1 root users  63M Sep  8 10:39 2963478244
@ainar-g
Copy link
Contributor

ainar-g commented Sep 9, 2021

Hello and thank you for your report. We can't seem to reproduce this issue on our machines. There are filter backup files, but they all must have a .old extension. If you stop AGH, delete the files, and restart it, do they reappear?

@ainar-g ainar-g added the waiting for data Waiting for users to provide more data. label Sep 9, 2021
@gabrielwhite
Copy link
Author

Here's the state of the directory one day later:

total 753M
drwxr-S--- 2 root users  12K Sep  9 14:31 .
drwxr-S--- 3 root users 4.0K Sep  9 14:28 ..
-rw-r--r-- 1 root users  63M Sep  8 10:42 1558400762
-rw-r--r-- 1 root users  63M Sep  9 14:31 1571306289
-rw-r--r-- 1 root users  63M Sep  9 13:01 1631105697.txt
-rw-r--r-- 1 root users 747K Sep  9 13:23 1631105792.txt
-rw-r--r-- 1 root users  63M Sep  9 10:45 1642326027
-rw-r--r-- 1 root users  63M Sep  8 10:39 2049054227
-rw-r--r-- 1 root users  63M Sep  8 10:01 2463411087
-rw-r--r-- 1 root users  63M Sep  9 10:44 2726644904
-rw-r--r-- 1 root users  63M Sep  8 10:39 2963478244
-rw-r--r-- 1 root users  63M Sep  9 10:23 3197215412
-rw-r--r-- 1 root users  63M Sep  9 12:58 4045172626
-rw-r--r-- 1 root users  63M Sep  9 11:44 4060345806
-rw-r--r-- 1 root users  63M Sep  9 11:38 4093212594

These are the two blocklists I have installed, and it seems it's the first file that's causing problems somehow.

https://blocklistproject.github.io/Lists/porn.txt
https://adguardteam.github.io/AdGuardSDNSFilter/Filters/filter.txt

FWIW I'm now running v0.107.0-a.157+80548233

@ainar-g ainar-g added needs investigation Needs to be reproduced reliably. P3: Medium and removed waiting for data Waiting for users to provide more data. labels Sep 10, 2021
@ainar-g ainar-g added this to the v0.107.0 milestone Sep 10, 2021
@EugeneOne1 EugeneOne1 added bug and removed needs investigation Needs to be reproduced reliably. labels Sep 14, 2021
@gabrielwhite
Copy link
Author

gabrielwhite commented Sep 14, 2021 •
edited
Loading

As of now the issue has partly resolved itself. The directory is no longer filling up with temporary files.

The problem is now limited to when I trigger a manual update only.

I get this error in the Web UI:

Error: control/filtering/refresh | <html> ,<head><title>504 Gateway Time-out</title></head>, ,<body> ,<center><h1>504 Gateway Time-out</h1></center>, ,<hr />,<center>openresty</center>, </body>, </html> | 504

And a temporary file is created and not deleted in the filter folder:

total 127M
drwxr-S--- 2 root users  12K Sep 14 12:18 .
drwxr-S--- 3 root users 4.0K Sep 13 18:15 ..
-rw-r--r-- 1 root users  63M Sep 13 17:39 1631105697.txt
-rw-r--r-- 1 root users 748K Sep 13 17:48 1631105792.txt
-rw-r--r-- 1 root users  63M Sep 14 12:18 3960263269

I'm now running v0.107.0-a.161+fac574d3

@ainar-g
Copy link
Contributor

ainar-g commented Sep 14, 2021

Thanks for the info, we seem to have found the root cause of the issue and are working on fixing it.

adguard pushed a commit that referenced this issue Sep 15, 2021
@EugeneOne1
Pull request: 3567 filters update
176a344 
Merge in DNS/adguard-home from 3567-old-filters to master

Updates #3567.

Squashed commit of the following:

commit d5cc419
Author: Eugene Burkov <E.Burkov@AdGuard.COM>
Date:   Wed Sep 15 19:22:49 2021 +0300

    home: lock doc

commit 54edba6
Author: Eugene Burkov <E.Burkov@AdGuard.COM>
Date:   Wed Sep 15 14:16:20 2021 +0300

    home: imp code, docs

commit e6dde1d
Author: Eugene Burkov <E.Burkov@AdGuard.COM>
Date:   Wed Sep 15 13:53:50 2021 +0300

    home: imp code, logic

commit b258b62
Author: Eugene Burkov <E.Burkov@AdGuard.COM>
Date:   Tue Sep 14 19:35:14 2021 +0300

    all: imp log of changes

commit 9b66cde
Author: Eugene Burkov <E.Burkov@AdGuard.COM>
Date:   Tue Sep 14 18:56:52 2021 +0300

    home: imp filter upd
@EugeneOne1
Copy link
Member

@gabrielwhite, we've just pushed the latest build to the edge channel. It should fix some logic problems related to the issue. Could you please check, if it works with your filter lists?

@gabrielwhite
Copy link
Author

Upon upgrading, I ran into the same issue again. Additionally it seemed that Adguard crashed and restarted after triggering a manual update. I'm running v0.107.0-a.166+176a344a.

Once it started crashing I thought it may have been a memory issue, so I increased the memory limitation on the docker to 320MB (it had been capped at 256MB), and this stopped the issue.

I'd previously moved to using the edge version because it ran without crashing in 256MB (0.106 wouldn't run with less memory). Adguard had been causing crashes on the OrangePi because it was chewing up too much memory.

@EugeneOne1
Copy link
Member

EugeneOne1 commented Sep 15, 2021 •
edited
Loading

Well, could you please help us to troubleshoot this issue.

We need you to configure Home to collect verbose log into a file. Note that in case of Docker's container the path to log file points to the container's filesystem, so the simplest way is a file in working or configuration directory which are mounted.

Then trigger the manual filter update and post the file here or send it to devteam@adguard.com with the issue number.

@leo15dev
Copy link

leo15dev commented Sep 16, 2021 •
edited
Loading

Hello and thank you for your report. We can't seem to reproduce this issue on our machines. There are filter backup files, but they all must have a .old extension. If you stop AGH, delete the files, and restart it, do they reappear?

@ainar-g It seems to when you take the action Delete in the DNS blocklists page to delete the blocklists, the system will automatically rename the deleted blocklist file with .old extension (e.g. 1631743246.txt --> 1631743246.txt.old ). And when you added the same filter again, the system will create new file (1631743247.txt) not use the backup one (1631743246.txt.old).

-rw-r--r-- 1 root root 65683147 Sep 16 08:30 1631743246.txt.old
-rw-r--r-- 1 root root 65683147 Sep 16 08:36 1631743247.txt

Since the user do the such things which most often means that they don't need those blocklist, if the user want to temporary pause it, they should temporary not enable the blocklist to instead of deleting it. If it is possible not to rename the deleted blocklist but delete the blocklist file when you take the action Delete in the DNS blocklists page? It will save the disk space, especially for those who install Adguard Home in the router. Thank you.

BTW I used the https://blocklistproject.github.io/Lists/porn.txt filter for the example.

@EugeneOne1
Copy link
Member

@gabrielwhite, we've received your logs, thanks! We'll inspect them as soon as possible.

@EugeneOne1
Copy link
Member

@gabrielwhite, we've reproduced the issue and got same results. I'm afraid the amount of RAM allocated for the container is the main cause of these crashes and unremoved temporary files. We've performed a couple of memory tests and concluded that 512 should be enough to handle updating of lists with around 3 million rules. However, you may try to decrease the limitation to some sensible minimum, because it's the only parameter for now which directly affects AdGuard Home stability with regards to updating filters.

We'll close the issue for now, since we cannot do anything about unrecoverable crashes for the moment. We'll consider further optimizations.

heyxkhoa pushed a commit to heyxkhoa/AdGuardHome that referenced this issue Mar 20, 2023
@EugeneOne1 @heyxkhoa
Pull request: 3567 filters update
967fe1e 
Merge in DNS/adguard-home from 3567-old-filters to master

Updates AdguardTeam#3567.

Squashed commit of the following:

commit d5cc419
Author: Eugene Burkov <E.Burkov@AdGuard.COM>
Date:   Wed Sep 15 19:22:49 2021 +0300

    home: lock doc

commit 54edba6
Author: Eugene Burkov <E.Burkov@AdGuard.COM>
Date:   Wed Sep 15 14:16:20 2021 +0300

    home: imp code, docs

commit e6dde1d
Author: Eugene Burkov <E.Burkov@AdGuard.COM>
Date:   Wed Sep 15 13:53:50 2021 +0300

    home: imp code, logic

commit b258b62
Author: Eugene Burkov <E.Burkov@AdGuard.COM>
Date:   Tue Sep 14 19:35:14 2021 +0300

    all: imp log of changes

commit 9b66cde
Author: Eugene Burkov <E.Burkov@AdGuard.COM>
Date:   Tue Sep 14 18:56:52 2021 +0300

    home: imp filter upd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@EugeneOne1 EugeneOne1

Labels
bug P3: Medium
Projects
None yet
Milestone
v0.107.0
Development

No branches or pull requests
4 participants
@gabrielwhite @ainar-g @EugeneOne1 @leo15dev