Add regex filtering support for domains on the Query Log #1611

DL6ER · 2023-08-01T14:59:13Z

What does this implement/fix?

~~Add regex filtering support for domains on the Query Log (new config option webserver.api.excludeRegex accepting an array of strings).~~

updated description: Add regex filtering support for domains on the Query Log as well as the "Top Items" tables on the dashboard. This is implemented by offering regex support for existing webserver.api.excludeClients and webserver.api.excludeDomains settings.

This implements a highly upvoted Discourse feature request (see comment below).

Related issue or feature (if applicable): N/A

Pull request in docs with documentation (if applicable): TBD

By submitting this pull request, I confirm the following:

I have read and understood the contributors guide, as well as this entire template. I understand which branch to base my commits and Pull Requests against.
I have commented my proposed changes within the code.
I am willing to help maintain this change if there are issues with it later.
It is compatible with the EUPL 1.2 license
I have squashed any insignificant commits. (git rebase)

Checklist:

The code change is tested and works locally.
I based my code and PRs against the repositories developmental branch.
I signed off all commits. Pi-hole enforces the DCO for all contributions
I signed all my commits. Pi-hole requires signatures to verify authorship
I have read the above and my PR is ready for review.

pralor-bot · 2023-08-01T14:59:27Z

This pull request has been mentioned on Pi-hole Userspace. There might be relevant details there:

https://discourse.pi-hole.net/t/option-to-ignore-domains-from-appearing-in-the-query-log/6319/68

yubiuser · 2023-08-02T18:46:42Z

There are two related endpoints webserver.api.excludeClients and webserver.api.excludeDomains. I guess there reflect what we have in v5 to exclude from the top lists at the dashboard. I think they should be extended to the query log as well - and maybe excludeRegex to the dashboard' top domain list as well.

yubiuser · 2023-08-02T19:50:33Z

My log is full of errors

023-08-02 21:19:35.597 [31064M] INFO: ########## FTL started on nanopi! ##########
2023-08-02 21:19:35.597 [31064M] INFO: FTL branch: new/queryLogRegex
2023-08-02 21:19:35.597 [31064M] INFO: FTL version: vDev-d3a6a9d
2023-08-02 21:19:35.597 [31064M] INFO: FTL commit: d3a6a9d2
2023-08-02 21:19:35.597 [31064M] INFO: FTL date: 2023-08-01 16:31:35 +0200
2023-08-02 21:19:35.597 [31064M] INFO: FTL user: pihole
2023-08-02 21:19:35.598 [31064M] INFO: Compiled for linux/arm64/v8 (compiled on CI) using cc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924
2023-08-02 21:19:35.604 [31064M] WARNING: Failed to open /etc/pihole/config_backups/pihole.toml.13 for reading: Permission denied
2023-08-02 21:19:35.637 [31064M] INFO: Writing config file
2023-08-02 21:19:35.667 [31064M] ERR: Cannot open /etc/pihole/custom.list for writing, unable to update custom.list: Permission denied
2023-08-02 21:19:35.675 [31064M] INFO: PID of FTL process: 31064
2023-08-02 21:19:35.683 [31064M] INFO: Database version is 13
2023-08-02 21:19:35.687 [31064M] INFO: Database successfully initialized
2023-08-02 21:19:35.737 [31064M] ERR: SQLite3 message: table query_storage has 11 columns but 12 values were supplied in "INSERT INTO query_storage SELECT * FROM disk.query_storage WHERE timestamp >= ?" (1)
2023-08-02 21:19:35.739 [31064M] ERR: import_queries_from_disk(): SQL error prepare: SQL logic error
2023-08-02 21:19:35.739 [31064M] INFO: Parsing queries in database
2023-08-02 21:19:35.743 [31064M] INFO: Imported 0 queries from the long-term database
2023-08-02 21:19:35.744 [31064M] ERR: SQLite3 message: statement aborts at 4: [ATTACH ? AS ?] database disk is already in use (1)
2023-08-02 21:19:35.744 [31064M] ERR: attach_database(): Failed to attach database: SQL logic error
2023-08-02 21:19:35.744 [31064M] INFO:  -> Total DNS queries: 0
2023-08-02 21:19:35.744 [31064M] INFO:  -> Cached DNS queries: 0
2023-08-02 21:19:35.745 [31064M] INFO:  -> Forwarded DNS queries: 0
2023-08-02 21:19:35.745 [31064M] INFO:  -> Blocked DNS queries: 0
2023-08-02 21:19:35.745 [31064M] INFO:  -> Unknown DNS queries: 0
2023-08-02 21:19:35.745 [31064M] INFO:  -> Unique domains: 0
2023-08-02 21:19:35.746 [31064M] INFO:  -> Unique clients: 0
2023-08-02 21:19:35.746 [31064M] INFO:  -> Known forward destinations: 0
2023-08-02 21:19:35.766 [31064M] INFO: listening on 0.0.0.0 port 53
2023-08-02 21:19:35.768 [31064M] INFO: listening on :: port 53
2023-08-02 21:19:35.782 [31064M] INFO: PID of FTL process: 31064
2023-08-02 21:19:35.785 [31064M] INFO: FTL is running as user pihole (UID 999)
2023-08-02 21:19:35.798 [31064M] INFO: Blocking status is enabled
2023-08-02 21:19:35.900 [31064/T31066] INFO: Compiled 1 allow and 2 deny regex for 0 client in 6.0 msec
2023-08-02 21:19:37.141 [31064/T31072] WARNING: API: Unauthorized
2023-08-02 21:19:37.141 [31064/T31071] WARNING: API: Unauthorized
2023-08-02 21:20:00.062 [31064/T31066] ERR: SQLite3 message: statement aborts at 4: [ATTACH ? AS ?] database disk is already in use (1)
2023-08-02 21:20:00.067 [31064/T31066] ERR: attach_database(): Failed to attach database: SQL logic error
2023-08-02 21:21:00.095 [31064/T31066] ERR: SQLite3 message: statement aborts at 4: [ATTACH ? AS ?] database disk is already in use (1)
2023-08-02 21:21:00.095 [31064/T31066] ERR: attach_database(): Failed to attach database: SQL logic error
2023-08-02 21:22:00.138 [31064/T31066] ERR: SQLite3 message: statement aborts at 4: [ATTACH ? AS ?] database disk is already in use (1)
2023-08-02 21:22:00.145 [31064/T31066] ERR: attach_database(): Failed to attach database: SQL logic error
2023-08-02 21:23:00.148 [31064/T31066] ERR: SQLite3 message: statement aborts at 4: [ATTACH ? AS ?] database disk is already in use (1)
2023-08-02 21:23:00.153 [31064/T31066] ERR: attach_database(): Failed to attach database: SQL logic error
2023-08-02 21:24:00.095 [31064/T31066] ERR: SQLite3 message: statement aborts at 4: [ATTACH ? AS ?] database disk is already in use (1)
2023-08-02 21:24:00.100 [31064/T31066] ERR: attach_database(): Failed to attach database: SQL logic error
2023-08-02 21:25:00.100 [31064/T31066] ERR: SQLite3 message: statement aborts at 4: [ATTACH ? AS ?] database disk is already in use (1)
2023-08-02 21:25:00.106 [31064/T31066] ERR: attach_database(): Failed to attach database: SQL logic error
2023-08-02 21:26:00.050 [31064/T31066] ERR: SQLite3 message: statement aborts at 4: [ATTACH ? AS ?] database disk is already in use (1)
2023-08-02 21:26:00.055 [31064/T31066] ERR: attach_database(): Failed to attach database: SQL logic error
2023-08-02 21:27:00.128 [31064/T31066] ERR: SQLite3 message: statement aborts at 4: [ATTACH ? AS ?] database disk is already in use (1)
2023-08-02 21:27:00.133 [31064/T31066] ERR: attach_database(): Failed to attach database: SQL logic error
2023-08-02 21:28:00.077 [31064/T31066] ERR: SQLite3 message: statement aborts at 4: [ATTACH ? AS ?] database disk is already in use (1)
2023-08-02 21:28:00.081 [31064/T31066] ERR: attach_database(): Failed to attach database: SQL logic error
2023-08-02 21:29:00.064 [31064/T31066] ERR: SQLite3 message: statement aborts at 4: [ATTACH ? AS ?] database disk is already in use (1)
2023-08-02 21:29:00.065 [31064/T31066] ERR: attach_database(): Failed to attach database: SQL logic error
2023-08-02 21:30:00.185 [31064/T31066] ERR: SQLite3 message: statement aborts at 4: [ATTACH ? AS ?] database disk is already in use (1)
2023-08-02 21:30:00.189 [31064/T31066] ERR: attach_database(): Failed to attach database: SQL logic error
2023-08-02 21:31:00.095 [31064/T31066] ERR: SQLite3 message: statement aborts at 4: [ATTACH ? AS ?] database disk is already in use (1)
2023-08-02 21:31:00.100 [31064/T31066] ERR: attach_database(): Failed to attach database: SQL logic error
2023-08-02 21:31:47.611 [31064/T31071] WARNING: API: Not found (/api/docs/specs/action.yaml)
2023-08-02 21:32:00.088 [31064/T31066] ERR: SQLite3 message: statement aborts at 4: [ATTACH ? AS ?] database disk is already in use (1)
2023-08-02 21:32:00.093 [31064/T31066] ERR: attach_database(): Failed to attach database: SQL logic error
2023-08-02 21:33:00.073 [31064/T31066] ERR: SQLite3 message: statement aborts at 4: [ATTACH ? AS ?] database disk is already in use (1)
2023-08-02 21:33:00.080 [31064/T31066] ERR: attach_database(): Failed to attach database: SQL logic error
2023-08-02 21:34:00.013 [31064/T31066] ERR: SQLite3 message: statement aborts at 4: [ATTACH ? AS ?] database disk is already in use (1)
2023-08-02 21:34:00.013 [31064/T31066] ERR: attach_database(): Failed to attach database: SQL logic error
2023-08-02 21:35:00.073 [31064/T31066] ERR: SQLite3 message: statement aborts at 4: [ATTACH ? AS ?] database disk is already in use (1)
2023-08-02 21:35:00.077 [31064/T31066] ERR: attach_database(): Failed to attach database: SQL logic error

DL6ER · 2023-08-03T06:38:33Z

There are two related endpoints webserver.api.excludeClients and webserver.api.excludeDomains. I guess there reflect what we have in v5 to exclude from the top lists at the dashboard. I think they should be extended to the query log as well - and maybe excludeRegex to the dashboard' top domain list as well.

This has been discussed in https://discourse.pi-hole.net/t/option-to-ignore-domains-from-appearing-in-the-query-log/6319/65 and following comments. It seems that this is not wanted to be part of the Top Lists. But this can for sure be further discussed (but maybe addressed in a separate PR?).

My log is full of errors

All the errors come from

2023-08-02 21:19:35.737 [31064M] ERR: SQLite3 message: table query_storage has 11 columns but 12 values were supplied in "INSERT INTO query_storage SELECT * FROM disk.query_storage WHERE timestamp >= ?" (1)
2023-08-02 21:19:35.739 [31064M] ERR: import_queries_from_disk(): SQL error prepare: SQL logic error

We have #1615 that tries to avoid the following errors but it will not always succeed. The root cause is that you've tried the regex_id branch which upgraded the FTL database internally but this branch does not contain these changes and, hence, the database schema is different than what FTL expects. This isn't an issue and only happens because the code of #1607 isn't part of this PR.

…option webserver.api.excludeRegex) Signed-off-by: DL6ER <dl6er@dl6er.de>

DL6ER · 2023-09-15T15:38:14Z

Rebased on latest development-v6 @yubiuser the errors you've been seeing should now be gone

DL6ER · 2023-10-07T19:59:13Z

No need to rebase this PR

github-actions · 2023-10-13T18:30:18Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

Signed-off-by: DL6ER <dl6er@dl6er.de>

github-actions · 2023-10-17T19:39:06Z

Conflicts have been resolved.

PromoFaux · 2023-10-28T15:21:46Z

If I use the array provided in the example ([ "(^|\.)\.google\.de$", "\.pi-hole\.net$" ]) I get the following when viewing the query log:

Signed-off-by: DL6ER <dl6er@dl6er.de>

DL6ER · 2023-11-04T06:57:02Z

@PromoFaux You are right, I will adjust the example regex. We need to double-backslash the \ -> \\ because \. on itself is an invalid escape sequence in the TOML file as the error message in the log revealed:

2023-11-04 07:53:58.236 [2248837/T2248851] ERR: Cannot parse config file: line 665: bad escape char

…e TOML file Signed-off-by: DL6ER <dl6er@dl6er.de>

github-actions · 2024-01-06T12:45:36Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

Signed-off-by: DL6ER <dl6er@dl6er.de>

DL6ER · 2024-01-13T09:28:11Z

What I suggested above is now implemented:

webserver.api.excludeRegex is gone
webserver.api.excludeClients and webserver.api.excludeDomains now support regex instead
both settings are automatically converted to exact matching regex on a v5 -> v6 upgrade. There is no upgrade strategy for existing beta users (this would be quite hard to detect as they may want anchored regular expressions)
webserver.api.excludeClients does not affect the client activity graph any longer

martinconroy · 2024-01-15T12:54:45Z

@martinconroy The only thing I can imagine are some spaces somewhere in your input. While being invisible in the screenshot, they will affect matching in a bad way. Please provide the exact output of
pihole-FTL --config webserver.api.excludeRegex
on your Pi-hole.

Sorry for the delay. So, I now have the below in my excludeClients

The output is as follows:
pihole-FTL --config webserver.api.excludeClients
[ "(^|\.)ovoenergy\.com$", "(^|\.)myharmony\.com$" ]

I tried various permutations with and without quotes and nothing seemed to make any difference.

pihole-FTL --config webserver.api.excludeClients
[ (^|\.)ovoenergy\.com$, (^|\.)myharmony\.com$ ]

Neither of these appeared to mask these domains in the query log:

DL6ER · 2024-01-15T15:32:21Z

Wait, excludeClients is definitely the wrong setting if you want to hide domains - this is for, well, hiding clients. In the lastest iteration, excludeDomains can take regex - and please without the surrounding ".

martinconroy · 2024-01-15T16:26:03Z

Wait, excludeClients is definitely the wrong setting if you want to hide domains - this is for, well, hiding clients. In the lastest iteration, excludeDomains can take regex - and please without the surrounding ".

Sorry - I did mean excludeDomains - misread when I saw excludeRegex was replaced.

Seems to have solved for svcs.myharmony.com (I no longer see those), but lrs.ovoenergy.com still gets through

yubiuser · 2024-01-16T20:40:22Z

Works really well. I just noted that query log filtering does not work for ^pi.hole$ and DS/DNSKEY` queries.

__

This PR needs an accompanying web PR to update the descriptions

Signed-off-by: DL6ER <dl6er@dl6er.de>

DL6ER · 2024-01-16T21:36:15Z

I just noted that query log filtering does not work for ^pi.hole$ and DS/DNSKEY` queries.

What you actually found was a bug appearing when you have multiple regular expressions - in your case ^pi.hole$ and ^Pixel-6.*$`. Compiled regular expression were able to overwrite each other (but without any bad side effect that could have triggered a crash (or similar).

I did just push a fix for this. I think this will also solve @martinconroy's observation:

Seems to have solved for svcs.myharmony.com (I no longer see those), but lrs.ovoenergy.com still gets through

It didn't happen all the time so I missed this before.

github-actions · 2024-01-16T21:39:59Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

Signed-off-by: DL6ER <dl6er@dl6er.de>

github-actions · 2024-01-16T21:46:49Z

Conflicts have been resolved.

martinconroy · 2024-01-17T09:53:24Z

I did just push a fix for this. I think this will also solve @martinconroy's observation:

It did! Thanks for your awesome support on this :)

yubiuser

I noticed you have large, identical codeblocks in stats.c and queries.c where you get the regex from the config, check for validity and compile the regex.
Would it increase readability if you factor this part out and re-use it to decrease code duplicity?

src/api/queries.c

src/config/setupVars.c

Signed-off-by: DL6ER <dl6er@dl6er.de>

yubiuser · 2024-01-20T11:18:30Z

Something is not right


2024-01-20 12:15:06.986 [636943M] INFO: Restored 0 API sessions from the database
2024-01-20 12:15:07.007 [636943M] INFO: Blocking status is enabled
2024-01-20 12:15:07.090 [636943/T637019] INFO: Compiled 0 allow and 0 deny regex for 8 clients in 1.1 msec
2024-01-20 12:15:10.009 [636943/T637019] INFO: Gravity database has been updated, reloading now
2024-01-20 12:15:10.016 [636943/T637019] INFO: Compiled 0 allow and 0 deny regex for 8 clients in 1.5 msec
2024-01-20 12:16:04.811 [636943/T637028] INFO: Config file written to /etc/pihole/pihole.toml
2024-01-20 12:16:52.087 [636943/T637031] INFO: Config file written to /etc/pihole/pihole.toml
2024-01-20 12:16:53.746 [636943/T637032] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2024-01-20 12:16:53.747 [636943/T637032] INFO: ---------------------------->  FTL crashed!  <----------------------------
2024-01-20 12:16:53.747 [636943/T637032] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2024-01-20 12:16:53.747 [636943/T637032] INFO: Please report a bug at https://github.com/pi-hole/FTL/issues
2024-01-20 12:16:53.747 [636943/T637032] INFO: and include in your report already the following details:
2024-01-20 12:16:53.747 [636943/T637032] INFO: FTL has been running for 108 seconds
2024-01-20 12:16:53.747 [636943/T637032] INFO: FTL branch: new/queryLogRegex
2024-01-20 12:16:53.747 [636943/T637032] INFO: FTL version: vDev-bafbc78
2024-01-20 12:16:53.747 [636943/T637032] INFO: FTL commit: bafbc780
2024-01-20 12:16:53.748 [636943/T637032] INFO: FTL date: 2024-01-20 09:38:08 +0100
2024-01-20 12:16:53.748 [636943/T637032] INFO: FTL user: started as pihole, ended as pihole
2024-01-20 12:16:53.748 [636943/T637032] INFO: Compiled for linux/arm64/v8 (compiled on CI) using cc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924
2024-01-20 12:16:53.748 [636943/T637032] INFO: Process details: MID: 636943
2024-01-20 12:16:53.748 [636943/T637032] INFO:                  PID: 636943
2024-01-20 12:16:53.748 [636943/T637032] INFO:                  TID: 637032
2024-01-20 12:16:53.749 [636943/T637032] INFO:                  Name: civetweb-worker
2024-01-20 12:16:53.749 [636943/T637032] INFO: Received signal: Segmentation fault
2024-01-20 12:16:53.749 [636943/T637032] INFO:      at address: 0x525493c1050096
2024-01-20 12:16:53.749 [636943/T637032] INFO:      with code:  SEGV_MAPERR (Address not mapped to object)
2024-01-20 12:16:53.749 [636943/T637032] INFO: !!! INFO: pihole-FTL has not been compiled with glibc/backtrace support, not generating one !!!
2024-01-20 12:16:53.749 [636943/T637032] INFO: ------ Listing content of directory /dev/shm ------
2024-01-20 12:16:53.749 [636943/T637032] INFO: File Mode User:Group      Size  Filename
2024-01-20 12:16:53.750 [636943/T637032] INFO: rwxrwxrwx root:root       280  .
2024-01-20 12:16:53.750 [636943/T637032] INFO: rwxr-xr-x root:root         3K  ..
2024-01-20 12:16:53.751 [636943/T637032] INFO: rw------- pihole:pihole   560K  FTL-fifo-log
2024-01-20 12:16:53.751 [636943/T637032] INFO: rw------- pihole:pihole     4K  FTL-per-client-regex
2024-01-20 12:16:53.752 [636943/T637032] INFO: rw------- pihole:pihole    74K  FTL-dns-cache
2024-01-20 12:16:53.752 [636943/T637032] INFO: rw------- pihole:pihole     8K  FTL-overTime
2024-01-20 12:16:53.753 [636943/T637032] INFO: rw------- pihole:pihole     1M  FTL-queries
2024-01-20 12:16:53.753 [636943/T637032] INFO: rw------- pihole:pihole    29K  FTL-upstreams
2024-01-20 12:16:53.754 [636943/T637032] INFO: rw------- pihole:pihole    86K  FTL-clients
2024-01-20 12:16:53.754 [636943/T637032] INFO: rw------- pihole:pihole    37K  FTL-domains
2024-01-20 12:16:53.754 [636943/T637032] INFO: rw------- pihole:pihole    82K  FTL-strings
2024-01-20 12:16:53.755 [636943/T637032] INFO: rw------- pihole:pihole    16  FTL-settings
2024-01-20 12:16:53.755 [636943/T637032] INFO: rw------- pihole:pihole   292  FTL-counters
2024-01-20 12:16:53.756 [636943/T637032] INFO: rw------- pihole:pihole    88  FTL-lock
2024-01-20 12:16:53.756 [636943/T637032] INFO: ---------------------------------------------------
2024-01-20 12:16:53.756 [636943/T637032] INFO: Please also include some lines from above the !!!!!!!!! header.
2024-01-20 12:16:53.756 [636943/T637032] INFO: Thank you for helping us to improve our FTL engine!
2024-01-20 12:16:53.756 [636943/T637032] INFO: Waiting for threads to join
2024-01-20 12:16:53.756 [636943/T637032] INFO: Thread database (0) is idle, terminating it.
2024-01-20 12:16:53.757 [636943/T637032] INFO: Thread housekeeper (1) is idle, terminating it.
2024-01-20 12:16:53.757 [636943/T637032] INFO: Thread DNS client (2) is idle, terminating it.
2024-01-20 12:16:53.759 [636943/T637032] INFO: All threads joined
2024-01-20 12:16:53.759 [636943/T637032] ERR: Error when obtaining outer SHM lock: Resource deadlock would occur
2024-01-20 12:16:53.759 [636943/T637032] ERR: Error when obtaining inner SHM lock: Resource deadlock would occur
2024-01-20 12:16:53.762 [636943/T637026] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2024-01-20 12:16:53.763 [636943/T637026] INFO: ---------------------------->  FTL crashed!  <----------------------------
2024-01-20 12:16:53.763 [636943/T637026] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2024-01-20 12:16:53.763 [636943/T637026] INFO: Please report a bug at https://github.com/pi-hole/FTL/issues
2024-01-20 12:16:53.763 [636943/T637026] INFO: and include in your report already the following details:
2024-01-20 12:16:53.763 [636943/T637026] INFO: FTL has been running for 108 seconds
2024-01-20 12:16:53.763 [636943/T637026] INFO: FTL branch: new/queryLogRegex
2024-01-20 12:16:53.764 [636943/T637026] INFO: FTL version: vDev-bafbc78
2024-01-20 12:16:53.764 [636943/T637026] INFO: FTL commit: bafbc780
2024-01-20 12:16:53.764 [636943/T637026] INFO: FTL date: 2024-01-20 09:38:08 +0100
2024-01-20 12:16:53.764 [636943/T637026] INFO: FTL user: started as pihole, ended as pihole
2024-01-20 12:16:53.764 [636943/T637026] INFO: Compiled for linux/arm64/v8 (compiled on CI) using cc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924
2024-01-20 12:16:53.765 [636943/T637026] INFO: Process details: MID: 636943
2024-01-20 12:16:53.765 [636943/T637026] INFO:                  PID: 636943
2024-01-20 12:16:53.765 [636943/T637026] INFO:                  TID: 637026
2024-01-20 12:16:53.765 [636943/T637026] INFO:                  Name: civetweb-worker
2024-01-20 12:16:53.765 [636943/T637026] INFO: Received signal: Segmentation fault
2024-01-20 12:16:53.765 [636943/T637026] INFO:      at address: 0x525493c1050096
2024-01-20 12:16:53.765 [636943/T637026] INFO:      with code:  SEGV_MAPERR (Address not mapped to object)
2024-01-20 12:16:53.766 [636943/T637026] INFO: !!! INFO: pihole-FTL has not been compiled with glibc/backtrace support, not generating one !!!
2024-01-20 12:16:53.766 [636943/T637026] INFO: ------ Listing content of directory /dev/shm ------
2024-01-20 12:16:53.766 [636943/T637026] INFO: File Mode User:Group      Size  Filename
2024-01-20 12:16:53.767 [636943/T637026] INFO: rwxrwxrwx root:root       280  .
2024-01-20 12:16:53.767 [636943/T637026] INFO: rwxr-xr-x root:root         3K  ..
2024-01-20 12:16:53.768 [636943/T637026] INFO: rw------- pihole:pihole   560K  FTL-fifo-log
2024-01-20 12:16:53.768 [636943/T637032] INFO: Stored 0 API sessions in the database
2024-01-20 12:16:53.768 [636943/T637026] INFO: rw------- pihole:pihole     4K  FTL-per-client-regex
2024-01-20 12:16:53.769 [636943/T637026] INFO: rw------- pihole:pihole    74K  FTL-dns-cache
2024-01-20 12:16:53.769 [636943/T637026] INFO: rw------- pihole:pihole     8K  FTL-overTime
2024-01-20 12:16:53.769 [636943/T637026] INFO: rw------- pihole:pihole     1M  FTL-queries
2024-01-20 12:16:53.770 [636943/T637026] INFO: rw------- pihole:pihole    29K  FTL-upstreams
2024-01-20 12:16:53.771 [636943/T637026] INFO: rw------- pihole:pihole    86K  FTL-clients
2024-01-20 12:16:53.771 [636943/T637026] INFO: rw------- pihole:pihole    37K  FTL-domains
2024-01-20 12:16:53.772 [636943/T637026] INFO: rw------- pihole:pihole    82K  FTL-strings
2024-01-20 12:16:53.772 [636943/T637026] INFO: rw------- pihole:pihole    16  FTL-settings
2024-01-20 12:16:53.772 [636943/T637026] INFO: rw------- pihole:pihole   292  FTL-counters
2024-01-20 12:16:53.773 [636943/T637026] INFO: rw------- pihole:pihole    88  FTL-lock
2024-01-20 12:16:53.773 [636943/T637026] INFO: ---------------------------------------------------
2024-01-20 12:16:53.773 [636943/T637026] INFO: Please also include some lines from above the !!!!!!!!! header.
2024-01-20 12:16:53.773 [636943/T637026] INFO: Thank you for helping us to improve our FTL engine!
2024-01-20 12:16:53.773 [636943/T637026] INFO: Waiting for threads to join
2024-01-20 12:16:53.774 [636943/T637026] INFO: Thread database (0) is idle, terminating it.
2024-01-20 12:16:53.774 [636943/T637026] INFO: Thread housekeeper (1) is idle, terminating it.
2024-01-20 12:16:59.475 [637270M] INFO: ########## FTL started on nanopi! ##########
2024-01-20 12:16:59.476 [637270M] INFO: FTL branch: new/queryLogRegex
2024-01-20 12:16:59.476 [637270M] INFO: FTL version: vDev-bafbc78
2024-01-20 12:16:59.476 [637270M] INFO: FTL commit: bafbc780
2024-01-20 12:16:59.476 [637270M] INFO: FTL date: 2024-01-20 09:38:08 +0100
2024-01-20 12:16:59.476 [637270M] INFO: FTL user: pihole
2024-01-20 12:16:59.476 [637270M] INFO: Compiled for linux/arm64/v8 (compiled on CI) using cc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924
2024-01-20 12:16:59.492 [637270M] INFO: Parsed config file /etc/pihole/pihole.toml successfully
2024-01-20 12:16:59.495 [637270M] INFO: PID of FTL process: 637270
2024-01-20 12:16:59.496 [637270M] INFO: listening on 0.0.0.0 port 53
2024-01-20 12:16:59.496 [637270M] INFO: listening on :: port 53
2024-01-20 12:16:59.501 [637270M] INFO: PID of FTL process: 637270
2024-01-20 12:16:59.503 [637270M] ERR: SQLite3: recovered 13 frames from WAL file /etc/pihole/pihole-FTL.db-wal (283)
2024-01-20 12:16:59.505 [637270M] INFO: Database version is 16
2024-01-20 12:16:59.523 [637270M] INFO: Database successfully initialized
2024-01-20 12:17:00.254 [637270M] INFO: Imported 13457 queries from the on-disk database (it has 383493 rows)
2024-01-20 12:17:00.254 [637270M] INFO: Parsing queries in database
2024-01-20 12:17:01.040 [637270M] INFO:   10000 queries parsed...
2024-01-20 12:17:01.318 [637270M] INFO: Imported 13457 queries from the long-term database
2024-01-20 12:17:01.318 [637270M] INFO:  -> Total DNS queries: 13457
2024-01-20 12:17:01.318 [637270M] INFO:  -> Cached DNS queries: 7208
2024-01-20 12:17:01.318 [637270M] INFO:  -> Forwarded DNS queries: 3328
2024-01-20 12:17:01.318 [637270M] INFO:  -> Blocked DNS queries: 1657
2024-01-20 12:17:01.319 [637270M] INFO:  -> Unknown DNS queries: 2
2024-01-20 12:17:01.319 [637270M] INFO:  -> Unique domains: 1105
2024-01-20 12:17:01.319 [637270M] INFO:  -> Unique clients: 8
2024-01-20 12:17:01.319 [637270M] INFO:  -> DNS cache records: 2605
2024-01-20 12:17:01.319 [637270M] INFO:  -> Known forward destinations: 2
2024-01-20 12:17:01.319 [637270M] INFO: FTL is running as user pihole (UID 999)
2024-01-20 12:17:01.320 [637270M] INFO: Reading certificate from /etc/pihole/tls.pem ...
2024-01-20 12:17:01.320 [637270M] INFO: Using SSL/TLS certificate file /etc/pihole/tls.pem
2024-01-20 12:17:01.323 [637270M] INFO: Restored 0 API sessions from the database
2024-01-20 12:17:01.328 [637270M] INFO: Blocking status is enabled
2024-01-20 12:17:01.423 [637270/T637271] INFO: Compiled 0 allow and 0 deny regex for 8 clients in 0.8 msec
2024-01-20 12:17:06.601 [637270/T637282] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2024-01-20 12:17:06.603 [637270/T637282] INFO: ---------------------------->  FTL crashed!  <----------------------------
2024-01-20 12:17:06.603 [637270/T637282] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2024-01-20 12:17:06.603 [637270/T637282] INFO: Please report a bug at https://github.com/pi-hole/FTL/issues
2024-01-20 12:17:06.603 [637270/T637282] INFO: and include in your report already the following details:
2024-01-20 12:17:06.604 [637270/T637282] INFO: FTL has been running for 7 seconds
2024-01-20 12:17:06.604 [637270/T637282] INFO: FTL branch: new/queryLogRegex
2024-01-20 12:17:06.604 [637270/T637282] INFO: FTL version: vDev-bafbc78
2024-01-20 12:17:06.604 [637270/T637282] INFO: FTL commit: bafbc780
2024-01-20 12:17:06.605 [637270/T637282] INFO: FTL date: 2024-01-20 09:38:08 +0100
2024-01-20 12:17:06.605 [637270/T637282] INFO: FTL user: started as pihole, ended as pihole
2024-01-20 12:17:06.605 [637270/T637282] INFO: Compiled for linux/arm64/v8 (compiled on CI) using cc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924
2024-01-20 12:17:06.606 [637270/T637282] INFO: Process details: MID: 637270
2024-01-20 12:17:06.606 [637270/T637282] INFO:                  PID: 637270
2024-01-20 12:17:06.607 [637270/T637282] INFO:                  TID: 637282
2024-01-20 12:17:06.607 [637270/T637282] INFO:                  Name: civetweb-worker
2024-01-20 12:17:06.607 [637270/T637282] INFO: Received signal: Segmentation fault
2024-01-20 12:17:06.608 [637270/T637282] INFO:      at address: 0x5dd79b0b9c00af
2024-01-20 12:17:06.608 [637270/T637282] INFO:      with code:  SEGV_MAPERR (Address not mapped to object)
2024-01-20 12:17:06.608 [637270/T637282] INFO: !!! INFO: pihole-FTL has not been compiled with glibc/backtrace support, not generating one !!!
2024-01-20 12:17:06.609 [637270/T637282] INFO: ------ Listing content of directory /dev/shm ------
2024-01-20 12:17:06.609 [637270/T637282] INFO: File Mode User:Group      Size  Filename
2024-01-20 12:17:06.610 [637270/T637282] INFO: rwxrwxrwx root:root       280  .
2024-01-20 12:17:06.611 [637270/T637282] INFO: rwxr-xr-x root:root         3K  ..
2024-01-20 12:17:06.612 [637270/T637282] INFO: rw------- pihole:pihole   560K  FTL-fifo-log
2024-01-20 12:17:06.613 [637270/T637282] INFO: rw------- pihole:pihole     4K  FTL-per-client-regex
2024-01-20 12:17:06.613 [637270/T637282] INFO: rw------- pihole:pihole    74K  FTL-dns-cache
2024-01-20 12:17:06.614 [637270/T637282] INFO: rw------- pihole:pihole     8K  FTL-overTime
2024-01-20 12:17:06.615 [637270/T637282] INFO: rw------- pihole:pihole     1M  FTL-queries
2024-01-20 12:17:06.616 [637270/T637282] INFO: rw------- pihole:pihole    29K  FTL-upstreams
2024-01-20 12:17:06.617 [637270/T637282] INFO: rw------- pihole:pihole    86K  FTL-clients
2024-01-20 12:17:06.618 [637270/T637282] INFO: rw------- pihole:pihole    37K  FTL-domains
2024-01-20 12:17:06.618 [637270/T637282] INFO: rw------- pihole:pihole    82K  FTL-strings
2024-01-20 12:17:06.619 [637270/T637282] INFO: rw------- pihole:pihole    16  FTL-settings
2024-01-20 12:17:06.620 [637270/T637282] INFO: rw------- pihole:pihole   292  FTL-counters
2024-01-20 12:17:06.621 [637270/T637282] INFO: rw------- pihole:pihole    88  FTL-lock
2024-01-20 12:17:06.621 [637270/T637282] INFO: ---------------------------------------------------
2024-01-20 12:17:06.621 [637270/T637282] INFO: Please also include some lines from above the !!!!!!!!! header.
2024-01-20 12:17:06.622 [637270/T637282] INFO: Thank you for helping us to improve our FTL engine!
2024-01-20 12:17:06.622 [637270/T637282] INFO: Waiting for threads to join
2024-01-20 12:17:06.622 [637270/T637282] INFO: Thread database (0) is idle, terminating it.
2024-01-20 12:17:06.623 [637270/T637282] INFO: Thread housekeeper (1) is idle, terminating it.
2024-01-20 12:17:06.623 [637270/T637282] INFO: Thread DNS client (2) is idle, terminating it.
2024-01-20 12:17:06.624 [637270/T637282] INFO: All threads joined
2024-01-20 12:17:06.624 [637270/T637282] ERR: Error when obtaining outer SHM lock: Resource deadlock would occur
2024-01-20 12:17:06.625 [637270/T637282] ERR: Error when obtaining inner SHM lock: Resource deadlock would occur
2024-01-20 12:17:06.632 [637270/T637283] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2024-01-20 12:17:06.633 [637270/T637283] INFO: ---------------------------->  FTL crashed!  <----------------------------
2024-01-20 12:17:06.633 [637270/T637283] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2024-01-20 12:17:06.633 [637270/T637283] INFO: Please report a bug at https://github.com/pi-hole/FTL/issues
2024-01-20 12:17:06.634 [637270/T637283] INFO: and include in your report already the following details:
2024-01-20 12:17:06.634 [637270/T637283] INFO: FTL has been running for 7 seconds
2024-01-20 12:17:06.634 [637270/T637283] INFO: FTL branch: new/queryLogRegex
2024-01-20 12:17:06.634 [637270/T637283] INFO: FTL version: vDev-bafbc78
2024-01-20 12:17:06.634 [637270/T637282] INFO: Stored 0 API sessions in the database
2024-01-20 12:17:06.634 [637270/T637283] INFO: FTL commit: bafbc780
2024-01-20 12:17:06.635 [637270/T637283] INFO: FTL date: 2024-01-20 09:38:08 +0100
2024-01-20 12:17:06.635 [637270/T637283] INFO: FTL user: started as pihole, ended as pihole
2024-01-20 12:17:06.635 [637270/T637283] INFO: Compiled for linux/arm64/v8 (compiled on CI) using cc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924
2024-01-20 12:17:06.635 [637270/T637283] INFO: Process details: MID: 637270
2024-01-20 12:17:06.635 [637270/T637283] INFO:                  PID: 637270
2024-01-20 12:17:06.635 [637270/T637283] INFO:                  TID: 637283
2024-01-20 12:17:06.635 [637270/T637283] INFO:                  Name: civetweb-worker
2024-01-20 12:17:06.635 [637270/T637283] INFO: Received signal: Segmentation fault
2024-01-20 12:17:06.636 [637270/T637283] INFO:      at address: 0x5dd79b0b9c00af
2024-01-20 12:17:06.636 [637270/T637283] INFO:      with code:  SEGV_MAPERR (Address not mapped to object)
2024-01-20 12:17:06.636 [637270/T637283] INFO: !!! INFO: pihole-FTL has not been compiled with glibc/backtrace support, not generating one !!!
2024-01-20 12:17:06.636 [637270/T637283] INFO: ------ Listing content of directory /dev/shm ------
2024-01-20 12:17:06.636 [637270/T637283] INFO: File Mode User:Group      Size  Filename
2024-01-20 12:17:06.636 [637270/T637283] INFO: rwxrwxrwx root:root       280  .
2024-01-20 12:17:06.637 [637270/T637283] INFO: rwxr-xr-x root:root         3K  ..
2024-01-20 12:17:06.637 [637270/T637283] INFO: rw------- pihole:pihole   560K  FTL-fifo-log
2024-01-20 12:17:06.637 [637270/T637283] INFO: rw------- pihole:pihole     4K  FTL-per-client-regex
2024-01-20 12:17:06.638 [637270/T637283] INFO: rw------- pihole:pihole    74K  FTL-dns-cache
2024-01-20 12:17:06.638 [637270/T637283] INFO: rw------- pihole:pihole     8K  FTL-overTime
2024-01-20 12:17:06.639 [637270/T637283] INFO: rw------- pihole:pihole     1M  FTL-queries
2024-01-20 12:17:06.639 [637270/T637283] INFO: rw------- pihole:pihole    29K  FTL-upstreams
2024-01-20 12:17:06.639 [637270/T637283] INFO: rw------- pihole:pihole    86K  FTL-clients
2024-01-20 12:17:06.640 [637270/T637283] INFO: rw------- pihole:pihole    37K  FTL-domains
2024-01-20 12:17:06.640 [637270/T637283] INFO: rw------- pihole:pihole    82K  FTL-strings
2024-01-20 12:17:06.641 [637270/T637283] INFO: rw------- pihole:pihole    16  FTL-settings
2024-01-20 12:17:06.641 [637270/T637283] INFO: rw------- pihole:pihole   292  FTL-counters
2024-01-20 12:17:06.641 [637270/T637283] INFO: rw------- pihole:pihole    88  FTL-lock
2024-01-20 12:17:06.641 [637270/T637283] INFO: ---------------------------------------------------
2024-01-20 12:17:06.641 [637270/T637283] INFO: Please also include some lines from above the !!!!!!!!! header.
2024-01-20 12:17:06.642 [637270/T637283] INFO: Thank you for helping us to improve our FTL engine!
2024-01-20 12:17:06.642 [637270/T637283] INFO: Waiting for threads to join
2024-01-20 12:17:06.642 [637270/T637283] INFO: Thread database (0) is idle, terminating it.

yubiuser · 2024-01-20T11:21:34Z

It can be triggert when more then one client regex is added and the web interfaced accessed.

Signed-off-by: DL6ER <dl6er@dl6er.de>

DL6ER · 2024-01-20T11:46:32Z

Ah thanks, this is a clear referencing error that should be solved by the most recent commit. It was introduced by me factoring out this into a subroutine which is now accessing the regex as pointer variable and [i] referencing on a regex_t* (i.e., pointer with size 8 bytes) than on a regex_t variable (with size 16). That's also why it only happens for multiple entries as 0*8 == 0*16 but 1*8 != 1*16.

DL6ER · 2024-01-21T18:51:37Z

@martinconroy Please go back to development-v6 branch, this branch will not receive any further updates. Thanks for being with us and giving valuable feedback!

martinconroy · 2024-01-22T10:31:57Z

@martinconroy Please go back to development-v6 branch, this branch will not receive any further updates. Thanks for being with us and giving valuable feedback!

Thanks @DL6ER for all your help making this an even better product!
Please can you confirm the appropriate ssh command to get me back on track

yubiuser · 2024-01-22T10:37:21Z

pihole checkout ftl development-v6

pralor-bot · 2024-01-29T15:09:44Z

This pull request has been mentioned on Pi-hole Userspace. There might be relevant details there:

https://discourse.pi-hole.net/t/no-query-log-update-anymore-and-no-top-clients-all-and-blocked/67929/13

DL6ER added Discourse Feature Request Pi-hole v6.0 labels Aug 1, 2023

DL6ER requested a review from a team August 1, 2023 14:59

DL6ER mentioned this pull request Aug 5, 2023

DBimport false sql error #1617

Open

Add regex filtering support for domains on the Query Log (new config …

d37c0d1

…option webserver.api.excludeRegex) Signed-off-by: DL6ER <dl6er@dl6er.de>

DL6ER force-pushed the new/queryLogRegex branch from d3a6a9d to d37c0d1 Compare September 15, 2023 15:38

DL6ER added 2 commits September 29, 2023 22:32

Merge branch 'development-v6' into new/queryLogRegex

299850e

Merge branch 'development-v6' into new/queryLogRegex

cc795fb

Merge branch 'development-v6' into new/queryLogRegex

24bacce

github-actions bot added the Merge conflicts label Oct 13, 2023

Merge branch 'development-v6' into new/queryLogRegex

7487e2e

Signed-off-by: DL6ER <dl6er@dl6er.de>

github-actions bot removed the Merge conflicts label Oct 17, 2023

DL6ER enabled auto-merge October 18, 2023 17:05

Merge branch 'development-v6' into new/queryLogRegex

aa55962

Signed-off-by: DL6ER <dl6er@dl6er.de>

DL6ER requested a review from PromoFaux November 4, 2023 06:59

Backslashs need to be escaped to avoid invalid escape sequences in th…

c89a239

…e TOML file Signed-off-by: DL6ER <dl6er@dl6er.de>

DL6ER force-pushed the new/queryLogRegex branch from 754dee3 to c89a239 Compare November 4, 2023 07:00

github-actions bot added the Merge conflicts label Jan 6, 2024

Merge branch 'development-v6' into new/queryLogRegex

ee367a6

Signed-off-by: DL6ER <dl6er@dl6er.de>

DL6ER requested a review from a team January 13, 2024 09:25

DL6ER marked this pull request as ready for review January 13, 2024 09:25

Compile exclude regexes only once, not N^2 times

5c4355f

Signed-off-by: DL6ER <dl6er@dl6er.de>

github-actions bot added the Merge conflicts label Jan 16, 2024

Merge branch 'development-v6' into new/queryLogRegex

862d492

Signed-off-by: DL6ER <dl6er@dl6er.de>

github-actions bot removed the Merge conflicts label Jan 16, 2024

yubiuser reviewed Jan 19, 2024

View reviewed changes

src/api/queries.c Outdated Show resolved Hide resolved

src/api/queries.c Outdated Show resolved Hide resolved

src/config/setupVars.c Outdated Show resolved Hide resolved

yubiuser mentioned this pull request Jan 19, 2024

Add regex hint to domain/client exclusion box pi-hole/web#2934

Merged

1 task

DL6ER added 2 commits January 20, 2024 09:33

Reduce code duplication by factoring out filter regex compilation

24b6df4

Signed-off-by: DL6ER <dl6er@dl6er.de>

Apply review comments

bafbc78

Signed-off-by: DL6ER <dl6er@dl6er.de>

Fix pointer magic going wrong

619a8b1

Signed-off-by: DL6ER <dl6er@dl6er.de>

yubiuser approved these changes Jan 21, 2024

View reviewed changes

DL6ER merged commit 8f4272b into development-v6 Jan 21, 2024
17 checks passed

DL6ER deleted the new/queryLogRegex branch January 21, 2024 18:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add regex filtering support for domains on the Query Log #1611

Add regex filtering support for domains on the Query Log #1611

DL6ER commented Aug 1, 2023 •

edited

pralor-bot commented Aug 1, 2023

yubiuser commented Aug 2, 2023

yubiuser commented Aug 2, 2023

DL6ER commented Aug 3, 2023

DL6ER commented Sep 15, 2023

DL6ER commented Oct 7, 2023

github-actions bot commented Oct 13, 2023

github-actions bot commented Oct 17, 2023

PromoFaux commented Oct 28, 2023

DL6ER commented Nov 4, 2023

github-actions bot commented Jan 6, 2024

DL6ER commented Jan 13, 2024

martinconroy commented Jan 15, 2024

DL6ER commented Jan 15, 2024

martinconroy commented Jan 15, 2024

yubiuser commented Jan 16, 2024

DL6ER commented Jan 16, 2024 •

edited

github-actions bot commented Jan 16, 2024

github-actions bot commented Jan 16, 2024

martinconroy commented Jan 17, 2024

yubiuser left a comment

yubiuser commented Jan 20, 2024

yubiuser commented Jan 20, 2024

DL6ER commented Jan 20, 2024

DL6ER commented Jan 21, 2024

martinconroy commented Jan 22, 2024

yubiuser commented Jan 22, 2024

pralor-bot commented Jan 29, 2024

Add regex filtering support for domains on the Query Log #1611

Add regex filtering support for domains on the Query Log #1611

Conversation

DL6ER commented Aug 1, 2023 • edited

What does this implement/fix?

Checklist:

pralor-bot commented Aug 1, 2023

yubiuser commented Aug 2, 2023

yubiuser commented Aug 2, 2023

DL6ER commented Aug 3, 2023

DL6ER commented Sep 15, 2023

DL6ER commented Oct 7, 2023

github-actions bot commented Oct 13, 2023

github-actions bot commented Oct 17, 2023

PromoFaux commented Oct 28, 2023

DL6ER commented Nov 4, 2023

github-actions bot commented Jan 6, 2024

DL6ER commented Jan 13, 2024

martinconroy commented Jan 15, 2024

DL6ER commented Jan 15, 2024

martinconroy commented Jan 15, 2024

yubiuser commented Jan 16, 2024

DL6ER commented Jan 16, 2024 • edited

github-actions bot commented Jan 16, 2024

github-actions bot commented Jan 16, 2024

martinconroy commented Jan 17, 2024

yubiuser left a comment

Choose a reason for hiding this comment

yubiuser commented Jan 20, 2024

yubiuser commented Jan 20, 2024

DL6ER commented Jan 20, 2024

DL6ER commented Jan 21, 2024

martinconroy commented Jan 22, 2024

yubiuser commented Jan 22, 2024

pralor-bot commented Jan 29, 2024

DL6ER commented Aug 1, 2023 •

edited

DL6ER commented Jan 16, 2024 •

edited