PADD 3.8 crashing pi-hole #252

Gothico74 · 2022-07-27T04:59:06Z

After upgrading from 3.7.1 my pi-hole (raspbian) to 3.8 i get an error: pi hole Disk shortage (/var/log/pihole/FTL.log)
after digging around i was able to find out that the ftl.log was 15Gb big and growing.
raspbian was running out of space and my Raspberry was crashing after two hours.
i deleted the file and it got another two hours.
after reverting padd back to 3.7.1 my system runs now 10hours and still up.

also screen (design) issues

long url's splitt in two lines, breaking the design
after the ipv6 adress is an empty line.
so i am missing two lines in my display

Gothico74 · 2022-07-27T05:00:36Z

forgot. pi-hole is updated to latest version also raspbian (buster)

yubiuser · 2022-07-27T06:45:49Z

What errors or lines fill the ftl.log?

Gothico74 · 2022-07-27T10:57:22Z

i wasn't looking into it. raspbian was very unstable and could freeze any moment, so i deleted it as fast as i could.

jbutters110 · 2022-07-27T16:02:28Z

I am having the same issue FTL.log grows many many GB in several hours. PADD v3.8, latest Pi-hole.

In FTL.log the following line appears approximately 15 times every 0.001 seconds!

An info message about Telnet
WARN: Could not accept() in listener() (/_w/FTL/FTL/src/api/socket.c:267): Socket operation on non-socket

Looks like a communication error between PADD and Pi-hole and Pi-hole is logging the failures.

Perhaps to do with the new API endpoint for the gateway? That uses telnet, right?

ytsejam1138 · 2022-07-27T16:23:23Z

Same issue with my install on RPi Zero W2.

cmdshft · 2022-07-28T04:53:37Z

Interesting that I have this issue on one Pihole but not the other, both were on latest versions of everything at the time of this comment. My Pi3b+ pihole had ftl.log at 20GB, but my Pi4 only about 8Kb. Not sure why there was a difference. However, the pi4 pihole crashed early this morning without my knowledge around 3am according to the 24hr log and my Pi3 pihole crashed 3 hours after that. I am currently downloading the 20GB FTL.log file to see what I can find. I am thankful I have 32GB of RAM now haha

[2022-07-27 15:09:07.399 716/T717] WARN: Could not accept() in listener() (/__w/FTL/FTL/src/api/socket.c:267): Socket operation on non-socket
[2022-07-27 15:09:07.399 716/T717] IPv4 telnet error: Success (0)

[2022-07-28 05:48:14.806 745/T750] WARNING: Disk shortage (/var/log/pihole/FTL.log) ahead: 99% is used (/var/log/pihole: 31.2GB used, 31.2GB total)
[2022-07-28 05:48:25.723 745M] Shutting down...
[2022-07-28 05:48:27.211 745M] Finished final database update (stored 5 queries)
[2022-07-28 05:48:27.212 745M] Waiting for threads to join
[2022-07-28 05:48:27.212 745M] Thread telnet-IPv4 (0) is idle, terminating it.
[2022-07-28 05:48:27.213 745M] Thread telnet-IPv6 (1) is idle, terminating it.
[2022-07-28 05:48:27.213 745M] Thread telnet-socket (2) is idle, terminating it.
[2022-07-28 05:48:27.214 745M] Thread database (3) is idle, terminating it.
[2022-07-28 05:48:27.215 745M] Thread housekeeper (4) is idle, terminating it.
[2022-07-28 05:48:27.215 745M] Thread DNS client (5) is idle, terminating it.
[2022-07-28 05:48:27.216 745M] All threads joined
[2022-07-28 05:48:27.218 745M] WARNING: Unable to remove PORT file: Permission denied
[2022-07-28 05:48:27.218 745M] WARNING: Unable to remove PID file: Permission denied
[2022-07-28 05:48:27.220 745M] ########## FTL terminated after 1h 10m 13s (code 0)! ##########

yubiuser · 2022-07-28T05:37:04Z

Could you please try to manually edit the following line

PADD/padd.sh

Line 113 in 380636a

echo ">$1 >quit" | nc 127.0.0.1 "${ftl_port}"

To

data="$(echo ">$1 >quit" | nc 127.0.0.1 "${ftl_port}")"
echo "${data}“

jbutters110 · 2022-07-28T11:13:34Z

The issue only begins for me after a few hours of running Pi-hole & PADD.. i.e. for an hour or so everything will work fine, but then after an amount of time FTL.log will begin to grow with the message:

WARN: Could not accept() in listener() (/_w/FTL/FTL/src/api/socket.c:267): Socket operation on non-socket

Gothico74 · 2022-07-28T14:59:25Z

Could you please try to manually edit the following line

PADD/padd.sh

Line 113 in 380636a

echo ">$1 >quit" | nc 127.0.0.1 "${ftl_port}"

To
data="$(echo ">$1 >quit" | nc 127.0.0.1 "${ftl_port}")"
echo "${data}“

Well back from work i changed the line.
Waiting, for now the logfile is a few kb small.

mejndp · 2022-07-28T15:06:23Z

Same issue as above. Clean install of raspbian (console only). Couple of hours of removing the FTL.log file and rebooted, Below is what the log is showing before telnet went haywired.

FYI - I made the changes to line 113 above AFTER and will report back if the issue is still ongoing.

--- snippet of FTL.log ----

[2022-07-28 08:40:50.077 834M] Using log file /var/log/pihole/FTL.log
[2022-07-28 08:40:50.078 834M] ########## FTL started on raspberrypi! ##########
[2022-07-28 08:40:50.078 834M] FTL branch: master
[2022-07-28 08:40:50.078 834M] FTL version: v5.16.1
[2022-07-28 08:40:50.078 834M] FTL commit: 5ff5bed
[2022-07-28 08:40:50.078 834M] FTL date: 2022-07-08 08:44:12 +0100
[2022-07-28 08:40:50.078 834M] FTL user: pihole
[2022-07-28 08:40:50.078 834M] Compiled for armv7hf (compiled on CI) using arm-linux-gnueabihf-gcc (Debian 6.3.0-18) 6.3.0 20170516
[2022-07-28 08:40:50.078 834M] Creating mutex
[2022-07-28 08:40:50.078 834M] Creating mutex
[2022-07-28 08:40:50.080 834M] Starting config file parsing (/etc/pihole/pihole-FTL.conf)
[2022-07-28 08:40:50.080 834M] SOCKET_LISTENING: only local
[2022-07-28 08:40:50.080 834M] AAAA_QUERY_ANALYSIS: Show AAAA queries
[2022-07-28 08:40:50.080 834M] MAXDBDAYS: max age for stored queries is 365 days
[2022-07-28 08:40:50.080 834M] RESOLVE_IPV6: Resolve IPv6 addresses
[2022-07-28 08:40:50.080 834M] RESOLVE_IPV4: Resolve IPv4 addresses
[2022-07-28 08:40:50.080 834M] DBINTERVAL: saving to DB file every minute
[2022-07-28 08:40:50.080 834M] DBFILE: Using /etc/pihole/pihole-FTL.db
[2022-07-28 08:40:50.080 834M] MAXLOGAGE: Importing up to 24.0 hours of log data
[2022-07-28 08:40:50.080 834M] PRIVACYLEVEL: Set to 0
[2022-07-28 08:40:50.080 834M] IGNORE_LOCALHOST: Show queries from localhost
[2022-07-28 08:40:50.080 834M] BLOCKINGMODE: Null IPs for blocked domains
[2022-07-28 08:40:50.080 834M] ANALYZE_ONLY_A_AND_AAAA: Disabled. Analyzing all queries
[2022-07-28 08:40:50.080 834M] DBIMPORT: Importing history from database
[2022-07-28 08:40:50.080 834M] PIDFILE: Using /run/pihole-FTL.pid
[2022-07-28 08:40:50.081 834M] PORTFILE: Using /run/pihole-FTL.port
[2022-07-28 08:40:50.081 834M] SOCKETFILE: Using /run/pihole/FTL.sock
[2022-07-28 08:40:50.081 834M] SETUPVARSFILE: Using /etc/pihole/setupVars.conf
[2022-07-28 08:40:50.081 834M] MACVENDORDB: Using /etc/pihole/macvendor.db
[2022-07-28 08:40:50.081 834M] GRAVITYDB: Using /etc/pihole/gravity.db
[2022-07-28 08:40:50.081 834M] PARSE_ARP_CACHE: Active
[2022-07-28 08:40:50.081 834M] CNAME_DEEP_INSPECT: Active
[2022-07-28 08:40:50.081 834M] DELAY_STARTUP: No delay requested.
[2022-07-28 08:40:50.081 834M] BLOCK_ESNI: Enabled, blocking _esni.{blocked domain}
[2022-07-28 08:40:50.081 834M] NICE: Set process niceness to -10 (default)
[2022-07-28 08:40:50.081 834M] MAXNETAGE: Removing IP addresses and host names from network table after 365 days
[2022-07-28 08:40:50.081 834M] NAMES_FROM_NETDB: Enabled, trying to get names from network database
[2022-07-28 08:40:50.081 834M] EDNS0_ECS: Overwrite client from ECS information
[2022-07-28 08:40:50.081 834M] REFRESH_HOSTNAMES: Periodically refreshing IPv4 names
[2022-07-28 08:40:50.081 834M] RATE_LIMIT: Rate-limiting client making more than 1000 queries in 60 seconds
[2022-07-28 08:40:50.081 834M] LOCAL_IPV4: Automatic interface-dependent detection of address
[2022-07-28 08:40:50.081 834M] LOCAL_IPV6: Automatic interface-dependent detection of address
[2022-07-28 08:40:50.081 834M] BLOCK_IPV4: Automatic interface-dependent detection of address
[2022-07-28 08:40:50.082 834M] BLOCK_IPV6: Automatic interface-dependent detection of address
[2022-07-28 08:40:50.082 834M] SHOW_DNSSEC: Enabled, showing automatically generated DNSSEC queries
[2022-07-28 08:40:50.082 834M] MOZILLA_CANARY: Enabled
[2022-07-28 08:40:50.082 834M] PIHOLE_PTR: internal PTR generation enabled (pi.hole)
[2022-07-28 08:40:50.082 834M] ADDR2LINE: Enabled
[2022-07-28 08:40:50.082 834M] REPLY_WHEN_BUSY: Drop queries when the database is busy
[2022-07-28 08:40:50.082 834M] BLOCK_TTL: 2 seconds
[2022-07-28 08:40:50.082 834M] BLOCK_ICLOUD_PR: Enabled
[2022-07-28 08:40:50.082 834M] CHECK_LOAD: Enabled
[2022-07-28 08:40:50.082 834M] CHECK_SHMEM: Warning if shared-memory usage exceeds 90%
[2022-07-28 08:40:50.082 834M] CHECK_DISK: Warning if certain disk usage exceeds 90%
[2022-07-28 08:40:50.082 834M] Finished config file parsing
[2022-07-28 08:40:50.086 834M] Database version is 12
[2022-07-28 08:40:50.086 834M] Resizing "FTL-strings" from 40960 to (81920 * 1) == 81920 (/dev/shm: 1.2MB used, 2.0GB total, FTL uses 1.2MB)
[2022-07-28 08:40:50.086 834M] Imported 0 alias-clients
[2022-07-28 08:40:50.086 834M] Database successfully initialized
[2022-07-28 08:40:50.116 834M] New upstream server: 8.8.4.4:53 (0/1024)
[2022-07-28 08:40:50.117 834M] New upstream server: 8.8.8.8:53 (1/1024)
[2022-07-28 08:40:50.188 834M] Resizing "FTL-queries" from 180224 to (8192 * 44) == 360448 (/dev/shm: 1.3MB used, 2.0GB total, FTL uses 1.3MB)
[2022-07-28 08:40:50.211 834M] Imported 5556 queries from the long-term database
[2022-07-28 08:40:50.211 834M] -> Total DNS queries: 5556
[2022-07-28 08:40:50.211 834M] -> Cached DNS queries: 678
[2022-07-28 08:40:50.211 834M] -> Forwarded DNS queries: 4022
[2022-07-28 08:40:50.211 834M] -> Blocked DNS queries: 856
[2022-07-28 08:40:50.211 834M] -> Unknown DNS queries: 0
[2022-07-28 08:40:50.211 834M] -> Unique domains: 624
[2022-07-28 08:40:50.211 834M] -> Unique clients: 18
[2022-07-28 08:40:50.211 834M] -> Known forward destinations: 2
[2022-07-28 08:40:50.211 834M] Successfully accessed setupVars.conf
[2022-07-28 08:40:50.216 834M] listening on 0.0.0.0 port 53
[2022-07-28 08:40:50.216 834M] listening on :: port 53
[2022-07-28 08:40:50.221 840M] PID of FTL process: 840
[2022-07-28 08:40:50.222 840/T841] Listening on port 4711 for incoming IPv4 telnet connections
[2022-07-28 08:40:50.222 840M] INFO: FTL is running as user pihole (UID 999)
[2022-07-28 08:40:50.223 840/T843] Listening on Unix socket
[2022-07-28 08:40:50.223 840M] Reloading DNS cache
[2022-07-28 08:40:50.224 840/T842] Listening on port 4711 for incoming IPv6 telnet connections
[2022-07-28 08:40:50.331 840/T844] Compiled 0 whitelist and 0 blacklist regex filters for 18 clients in 0.9 msec
[2022-07-28 08:40:50.331 840/T844] Blocking status is disabled
[2022-07-28 08:43:44.946 840M] Received: Real-time signal 0 (34 -> 0)
[2022-07-28 08:43:44.994 840/T844] Compiled 0 whitelist and 0 blacklist regex filters for 18 clients in 0.5 msec
[2022-07-28 08:43:44.994 840/T844] Blocking status is enabled
[2022-07-28 08:47:07.626 840M] Resizing "FTL-dns-cache" from 4096 to (512 * 16) == 8192 (/dev/shm: 1.4MB used, 2.0GB total, FTL uses 1.4MB)
[2022-07-28 08:55:32.212 840M] Resizing "FTL-domains" from 20480 to (2048 * 20) == 40960 (/dev/shm: 1.4MB used, 2.0GB total, FTL uses 1.4MB)
[2022-07-28 09:00:43.576 840M] Resizing "FTL-dns-cache" from 8192 to (768 * 16) == 12288 (/dev/shm: 1.5MB used, 2.0GB total, FTL uses 1.5MB)
[2022-07-28 09:29:07.074 840M] Resizing "FTL-dns-cache" from 12288 to (1024 * 16) == 16384 (/dev/shm: 1.5MB used, 2.0GB total, FTL uses 1.5MB)
[2022-07-28 09:36:27.638 840M] Resizing "FTL-dns-cache" from 16384 to (1280 * 16) == 20480 (/dev/shm: 1.5MB used, 2.0GB total, FTL uses 1.5MB)
[2022-07-28 09:52:08.797 840M] Resizing "FTL-dns-cache" from 20480 to (1536 * 16) == 24576 (/dev/shm: 1.5MB used, 2.0GB total, FTL uses 1.5MB)
[2022-07-28 10:06:16.780 840M] Resizing "FTL-queries" from 360448 to (12288 * 44) == 540672 (/dev/shm: 1.5MB used, 2.0GB total, FTL uses 1.5MB)
[2022-07-28 10:30:01.741 840M] Resizing "FTL-dns-cache" from 24576 to (1792 * 16) == 28672 (/dev/shm: 1.7MB used, 2.0GB total, FTL uses 1.7MB)
[2022-07-28 10:50:21.232 840/T841] WARN: Could not accept() in listener() (/__w/FTL/FTL/src/api/socket.c:267): Socket operation on non-socket
[2022-07-28 10:50:21.232 840/T841] IPv4 telnet error: Success (0)
[2022-07-28 10:50:21.232 840/T841] WARN: Could not accept() in listener() (/__w/FTL/FTL/src/api/socket.c:267): Socket operation on non-socket
[2022-07-28 10:50:21.232 840/T841] IPv4 telnet error: Success (0)
[2022-07-28 10:50:21.232 840/T841] WARN: Could not accept() in listener() (/__w/FTL/FTL/src/api/socket.c:267): Socket operation on non-socket
[2022-07-28 10:50:21.232 840/T841] IPv4 telnet error: Success (0)
[2022-07-28 10:50:21.233 840/T841] WARN: Could not accept() in listener() (/__w/FTL/FTL/src/api/socket.c:267): Socket operation on non-socket
[2022-07-28 10:50:21.233 840/T841] IPv4 telnet error: Success (0)
[2022-07-28 10:50:21.233 840/T841] WARN: Could not accept() in listener() (/__w/FTL/FTL/src/api/socket.c:267): Socket operation on non-socket
[2022-07-28 10:50:21.233 840/T841] IPv4 telnet error: Success (0)
[2022-07-28 10:50:21.233 840/T841] WARN: Could not accept() in listener() (/__w/FTL/FTL/src/api/socket.c:267): Socket operation on non-socket
[2022-07-28 10:50:21.233 840/T841] IPv4 telnet error: Success (0)
[2022-07-28 10:50:21.233 840/T841] WARN: Could not accept() in listener() (/__w/FTL/FTL/src/api/socket.c:267): Socket operation on non-socket

--- end snippet of FTL.log ----

Gothico74 · 2022-07-28T16:16:02Z

pi@raspberrypi-hole:~ $ ls -lha /var/log/pihole*
lrwxrwxrwx 1 pihole pihole 23 Jul 8 13:58 /var/log/pihole-FTL.log -> /var/log/pihole/FTL.log
lrwxrwxrwx 1 pihole pihole 26 Jul 8 13:58 /var/log/pihole.log -> /var/log/pihole/pihole.log
/var/log/pihole:
insgesamt 3,1M
drwxr-xr-x 2 pihole pihole 4,0K Jul 28 16:51 .
drwxr-xr-x 10 root root 4,0K Jul 28 00:00 ..
-rw-r--r-- 1 root root 1,7K Jul 18 03:20 20220718_updatePihole.sh.log
-rw-r--r-- 1 root root 1,7K Jul 25 03:22 20220725_updatePihole.sh.log
-rw-r--r-- 1 pihole pihole 21K Jul 28 18:06 FTL.log
-rw-r--r-- 1 pihole pihole 15K Jul 28 00:00 FTL.log.1
-rw-r--r-- 1 pihole pihole 2,9K Jul 27 00:00 FTL.log.2.gz
-rw-r--r-- 1 pihole pihole 2,4K Jul 26 00:00 FTL.log.3.gz
-rw-r----- 1 pihole pihole 1,2K Jul 28 16:51 pihole.log
-rw-r----- 1 pihole pihole 60 Jul 28 16:51 pihole.log.1
-rw-r----- 1 pihole pihole 353K Jul 27 00:00 pihole.log.2.gz
-rw-r----- 1 pihole pihole 788K Jul 26 00:00 pihole.log.3.gz
-rw-r----- 1 pihole pihole 873K Jul 25 00:00 pihole.log.4.gz
-rw-r----- 1 pihole pihole 1,1M Jul 24 00:00 pihole.log.5.gz
-rw-r--r-- 1 root root 5,1K Jul 24 04:11 pihole_updateGravity.log
-rw-r--r-- 1 root root 1,1K Jul 25 03:22 updatePihole.stats.log

[2022-07-28 16:51:53.527 736/T737] Listening on port 4711 for incoming IPv4 telnet connections
[2022-07-28 16:51:53.527 736/T738] Listening on port 4711 for incoming IPv6 telnet connections
[2022-07-28 16:51:53.528 736/T739] Listening on Unix socket
[2022-07-28 16:51:53.528 736M] INFO: FTL is running as user pihole (UID 999)
[2022-07-28 16:51:53.529 736M] Reloading DNS cache
[2022-07-28 16:51:53.710 736/T740] Compiled 1 whitelist and 37 blacklist regex filters for 15 clients in 73.9 msec
[2022-07-28 16:51:53.710 736/T740] Blocking status is enabled
[2022-07-28 16:51:59.154 736M] Resizing "FTL-domains" from 20480 to (2048 * 20) == 40960 (/dev/shm: 3.6MB used, 484.0MB total, FTL uses 3.6MB)
[2022-07-28 16:52:34.020 736M] Resizing "FTL-queries" from 2523136 to (61440 * 44) == 2703360 (/dev/shm: 3.6MB used, 484.0MB total, FTL uses 3.6MB)
[2022-07-28 16:56:27.434 736M] Resizing "FTL-dns-cache" from 4096 to (512 * 16) == 8192 (/dev/shm: 3.8MB used, 484.0MB total, FTL uses 3.8MB)
[2022-07-28 17:00:00.312 736/T740] Notice: Database size is 1888.31 MB, deleted 774 rows
[2022-07-28 17:10:06.929 736/T740] Notice: Database size is 1888.31 MB, deleted 960 rows
[2022-07-28 17:16:26.391 736M] Resizing "FTL-dns-cache" from 8192 to (768 * 16) == 12288 (/dev/shm: 3.8MB used, 484.0MB total, FTL uses 3.8MB)
[2022-07-28 17:20:05.619 736/T740] Notice: Database size is 1888.31 MB, deleted 1405 rows
[2022-07-28 17:23:12.232 736M] Resizing "FTL-dns-cache" from 12288 to (1024 * 16) == 16384 (/dev/shm: 3.8MB used, 484.0MB total, FTL uses 3.8MB)
[2022-07-28 17:30:02.834 736/T740] Notice: Database size is 1888.31 MB, deleted 1358 rows
[2022-07-28 17:40:04.221 736/T740] Notice: Database size is 1888.31 MB, deleted 1697 rows
[2022-07-28 17:50:02.847 736/T740] Notice: Database size is 1888.31 MB, deleted 1840 rows
[2022-07-28 17:51:25.572 736M] Resizing "FTL-dns-cache" from 16384 to (1280 * 16) == 20480 (/dev/shm: 3.8MB used, 484.0MB total, FTL uses 3.8MB)
[2022-07-28 18:00:06.967 736/T740] Notice: Database size is 1888.31 MB, deleted 1905 rows
[2022-07-28 18:05:51.996 736M] Received: Real-Time Signal 0 (34 -> 0)
[2022-07-28 18:05:52.072 736/T740] Compiled 1 whitelist and 37 blacklist regex filters for 15 clients in 60.5 msec
[2022-07-28 18:05:52.073 736/T740] Blocking status is disabled
[2022-07-28 18:05:56.202 736M] Received: Real-Time Signal 0 (34 -> 0)
[2022-07-28 18:05:56.345 736/T740] Compiled 1 whitelist and 37 blacklist regex filters for 15 clients in 61.0 msec
[2022-07-28 18:05:56.345 736/T740] Blocking status is enabled
[2022-07-28 18:06:02.490 736M] Received: Real-Time Signal 0 (34 -> 0)
[2022-07-28 18:06:08.385 736/T740] Compiled 1 whitelist and 37 blacklist regex filters for 15 clients in 110.6 msec
[2022-07-28 18:06:08.385 736/T740] Blocking status is disabled
[2022-07-28 18:06:17.579 736M] Received: Real-Time Signal 0 (34 -> 0)
[2022-07-28 18:06:17.695 736/T740] Compiled 1 whitelist and 37 blacklist regex filters for 15 clients in 91.2 msec
[2022-07-28 18:06:17.696 736/T740] Blocking status is enabled
[2022-07-28 18:06:25.866 736M] Received: Real-Time Signal 0 (34 -> 0)
[2022-07-28 18:06:26.003 736/T740] Compiled 1 whitelist and 37 blacklist regex filters for 15 clients in 90.1 msec
[2022-07-28 18:06:26.004 736/T740] Blocking status is disabled
[2022-07-28 18:06:50.956 736M] Received: Real-Time Signal 0 (34 -> 0)
[2022-07-28 18:06:51.105 736/T740] Compiled 1 whitelist and 37 blacklist regex filters for 15 clients in 59.4 msec
[2022-07-28 18:06:51.105 736/T740] Blocking status is enabled
[2022-07-28 18:10:02.931 736/T740] Notice: Database size is 1888.31 MB, deleted 2157 rows

after one hour still looking good

Gothico74 · 2022-07-28T17:07:58Z

[2022-07-28 19:02:01.582 736/T737] IPv4 telnet error: Erfolg (0)
[2022-07-28 19:02:01.582 736/T737] WARN: Could not accept() in listener() (/__w/FTL/FTL/src/api/socket.c:267): Ungültiger Dateideskriptor
[2022-07-28 19:02:01.582 736/T737] IPv4 telnet error: Erfolg (0)
[2022-07-28 19:02:01.582 736/T737] WARN: Could not accept() in listener() (/__w/FTL/FTL/src/api/socket.c:267): Ungültiger Dateideskriptor
[2022-07-28 19:02:01.582 736/T737] IPv4 telnet error: Erfolg (0)
[2022-07-28 19:02:01.582 736/T737] WARN: Could not accept() in listener() (/__w/FTL/FTL/src/api/socket.c:267): Ungültiger Dateideskriptor
[2022-07-28 19:02:01.582 736/T737] IPv4 telnet error: Erfolg (0)
[2022-07-28 19:02:01.582 736/T737] WARN: Could not accept() in listener() (/__w/FTL/FTL/src/api/socket.c:267): Ungültiger Dateideskriptor
[2022-07-28 19:02:01.582 736/T737] IPv4 telnet error: Erfolg (0)

here he goes....

pi@raspberrypi-hole:~ $ ls -lha /var/log/pihole*
lrwxrwxrwx 1 pihole pihole 23 Jul 8 13:58 /var/log/pihole-FTL.log -> /var/log/pihole/FTL.log
lrwxrwxrwx 1 pihole pihole 26 Jul 8 13:58 /var/log/pihole.log -> /var/log/pihole/pihole.log
/var/log/pihole:
insgesamt 389M
drwxr-xr-x 2 pihole pihole 4,0K Jul 28 16:51 .
drwxr-xr-x 10 root root 4,0K Jul 28 00:00 ..
-rw-r--r-- 1 root root 1,7K Jul 18 03:20 20220718_updatePihole.sh.log
-rw-r--r-- 1 root root 1,7K Jul 25 03:22 20220725_updatePihole.sh.log
-rw-r--r-- 1 pihole pihole 386M Jul 28 19:06 FTL.log
-rw-r--r-- 1 pihole pihole 15K Jul 28 00:00 FTL.log.1
-rw-r--r-- 1 pihole pihole 2,9K Jul 27 00:00 FTL.log.2.gz
-rw-r--r-- 1 pihole pihole 2,4K Jul 26 00:00 FTL.log.3.gz
-rw-r----- 1 pihole pihole 1,2K Jul 28 16:51 pihole.log
-rw-r----- 1 pihole pihole 60 Jul 28 16:51 pihole.log.1
-rw-r----- 1 pihole pihole 353K Jul 27 00:00 pihole.log.2.gz
-rw-r----- 1 pihole pihole 788K Jul 26 00:00 pihole.log.3.gz
-rw-r----- 1 pihole pihole 873K Jul 25 00:00 pihole.log.4.gz
-rw-r----- 1 pihole pihole 1,1M Jul 24 00:00 pihole.log.5.gz
-rw-r--r-- 1 root root 5,1K Jul 24 04:11 pihole_updateGravity.log
-rw-r--r-- 1 root root 1,1K Jul 25 03:22 updatePihole.stats.log

crashing pi-hole in 5...4....3...
wow, the logfile ist filling within seconds now...

mejndp · 2022-07-28T17:08:56Z

Update: changing line 113 does not fix the issue. I'm still getting the socket/non-socket error message and filling up the FTL.log.

--- snippet of FTL.log ----

[2022-07-28 10:55:21.451 836M] listening on 0.0.0.0 port 53
[2022-07-28 10:55:21.451 836M] listening on :: port 53
[2022-07-28 10:55:21.456 842M] PID of FTL process: 842
[2022-07-28 10:55:21.457 842/T843] Listening on port 4711 for incoming IPv4 telnet connections
[2022-07-28 10:55:21.457 842/T845] Listening on Unix socket
[2022-07-28 10:55:21.457 842M] INFO: FTL is running as user pihole (UID 999)
[2022-07-28 10:55:21.458 842/T844] Listening on port 4711 for incoming IPv6 telnet connections
[2022-07-28 10:55:21.460 842M] Reloading DNS cache
[2022-07-28 10:55:21.566 842/T846] Compiled 0 whitelist and 0 blacklist regex filters for 20 clients in 1.0 msec
[2022-07-28 10:55:21.566 842/T846] Blocking status is enabled
[2022-07-28 11:07:03.312 842M] Resizing "FTL-dns-cache" from 4096 to (512 * 16) == 8192 (/dev/shm: 1.6MB used, 2.0GB total, FTL uses 1.6MB)
[2022-07-28 11:10:37.578 842M] Resizing "FTL-dns-cache" from 8192 to (768 * 16) == 12288 (/dev/shm: 1.7MB used, 2.0GB total, FTL uses 1.6MB)
[2022-07-28 11:22:48.958 842M] Resizing "FTL-dns-cache" from 12288 to (1024 * 16) == 16384 (/dev/shm: 1.7MB used, 2.0GB total, FTL uses 1.6MB)
[2022-07-28 11:46:59.763 842M] Resizing "FTL-dns-cache" from 16384 to (1280 * 16) == 20480 (/dev/shm: 1.7MB used, 2.0GB total, FTL uses 1.6MB)
[2022-07-28 12:08:02.386 842M] Resizing "FTL-dns-cache" from 20480 to (1536 * 16) == 24576 (/dev/shm: 1.7MB used, 2.0GB total, FTL uses 1.7MB)
[2022-07-28 12:28:58.305 842M] Resizing "FTL-strings" from 81920 to (122880 * 1) == 122880 (/dev/shm: 1.7MB used, 2.0GB total, FTL uses 1.7MB)
[2022-07-28 12:31:07.302 842M] Resizing "FTL-dns-cache" from 24576 to (1792 * 16) == 28672 (/dev/shm: 1.7MB used, 2.0GB total, FTL uses 1.7MB)
[2022-07-28 12:41:26.720 842M] Resizing "FTL-dns-cache" from 28672 to (2048 * 16) == 32768 (/dev/shm: 1.7MB used, 2.0GB total, FTL uses 1.7MB)
[2022-07-28 13:04:32.976 842/T843] WARN: Could not accept() in listener() (/__w/FTL/FTL/src/api/socket.c:267): Socket operation on non-socket
[2022-07-28 13:04:32.976 842/T843] IPv4 telnet error: Success (0)
[2022-07-28 13:04:32.976 842/T843] WARN: Could not accept() in listener() (/__w/FTL/FTL/src/api/socket.c:267): Socket operation on non-socket
[2022-07-28 13:04:32.976 842/T843] IPv4 telnet error: Success (0)
[2022-07-28 13:04:32.976 842/T843] WARN: Could not accept() in listener() (/__w/FTL/FTL/src/api/socket.c:267): Socket operation on non-socket
[2022-07-28 13:04:32.976 842/T843] IPv4 telnet error: Success (0)
[2022-07-28 13:04:32.976 842/T843] WARN: Could not accept() in listener() (/__w/FTL/FTL/src/api/socket.c:267): Socket operation on non-socket
[2022-07-28 13:04:32.976 842/T843] IPv4 telnet error: Success (0)
[2022-07-28 13:04:32.977 842/T843] WARN: Could not accept() in listener() (/__w/FTL/FTL/src/api/socket.c:267): Socket operation on non-socket
[2022-07-28 13:04:32.977 842/T843] IPv4 telnet error: Success (0)
[2022-07-28 13:04:32.977 842/T843] WARN: Could not accept() in listener() (/__w/FTL/FTL/src/api/socket.c:267): Socket operation on non-socket
[2022-07-28 13:04:32.977 842/T843] IPv4 telnet error: Success (0)
[2022-07-28 13:04:32.977 842/T843] WARN: Could not accept() in listener() (/__w/FTL/FTL/src/api/socket.c:267): Socket operation on non-socket
[2022-07-28 13:04:32.977 842/T843] IPv4 telnet error: Success (0)
[2022-07-28 13:04:32.977 842/T843] WARN: Could not accept() in listener() (/__w/FTL/FTL/src/api/socket.c:267): Socket operation on non-socket
[2022-07-28 13:04:32.977 842/T843] IPv4 telnet error: Success (0)
[2022-07-28 13:04:32.977 842/T843] WARN: Could not accept() in listener() (/__w/FTL/FTL/src/api/socket.c:267): Socket operation on non-socket
[2022-07-28 13:04:32.977 842/T843] IPv4 telnet error: Success (0)

polyclementc · 2022-07-28T20:44:38Z

Hello, got the same issue as you. May I ask you what command to use to come back to PADD version 3.7.1 please?
Thank you

twerthi · 2022-07-29T00:55:38Z

Same issue here. pihole-FTL Log file grows astronomically quick, turning off PADD seems to resolve this.

mejndp · 2022-07-29T01:31:53Z

turning off PADD seems to resolve this

Hmm... interesting. I went ahead and disabled PADD (by deleting the launch script from .bashrc) then rebooted. We'll see if that is the suspect on my end.

mejndp · 2022-07-29T07:06:45Z

turning off PADD seems to resolve this

Hmm... interesting. I went ahead and disabled PADD (by deleting the launch script from .bashrc) then rebooted. We'll see if that is the suspect on my end.

Confirm. Disabling PADD did the trick. No more non-socket error messages.

yubiuser · 2022-07-29T07:10:31Z

We are still investigating. Cleary, there is an issue here, but we are not sure yet if the problem is within PADD, FTL or both. It's hard too reproduce because it does not affect all installations and usually does occur only after PADD has been running for a longer period of time.
To mitigate, you can run PADD v3.7.1

mejndp · 2022-07-29T07:19:30Z

To mitigate, you can run PADD v3.7.1

What's the proper commands / methods to downgrade and what's the URL? Many of the google search are pointing to a much older version - v3.2.1

yubiuser · 2022-07-29T07:27:22Z

This is the raw file
https://raw.githubusercontent.com/pi-hole/PADD/v3.7.1/padd.sh

Download and replace your PADD.sh

yubiuser · 2022-07-29T16:05:06Z

The only two changes we made regarding the FTL telnet connections are the two new GetFTLData calls

DHCP_ROUTER=$(GetFTLData "gateway" | awk '{ printf $1 }')

def_iface_data=$(GetFTLData "interfaces" | head -n1)

DL6ER · 2022-07-29T18:01:04Z

If you want, you could check my idea to remedy this (especially the "astronomically quick" FTL.log growth) by running

pihole checkout ftl fix/socket_simplification

You can always go back to the previous state by running

pihole checkout ftl master

yubiuser · 2022-07-30T13:44:30Z

pihole checkout ftl fix/socket_simplification

There was an update to that branch, if you are using it, please run pihole -up

howardCodeGit · 2022-08-01T15:46:10Z

Thanks, I am also having this problem and one of the side effect of it is it generated a 27GB FTL.log and forcing my pihole to stop running.

baz069 · 2022-08-01T16:33:14Z

Thanks, I am also having this problem and one of the side effect of it is it generated a 27GB FTL.log and forcing my pihole to stop running.

Exactly the same here, once the disk is full with the massive FTL.log the pi crashes. Rolled back to PADD v3.7.1 as suggested above and the log only reaches a matter of kilobytes.

It would appear that the issue is being worked on, but is currently on hold pending an 'unresolved conversation' (not sure what that means) - pi-hole/FTL#1396

yubiuser · 2022-08-01T17:46:24Z

pi-hole/FTL#1396 together with #254 should prevent crashing of FTL when using PADD 3.8.0.

Once the FTL PR has been merged, it's part of the development branch which will lead to the next released FTL version. We target an release in the not so distant future.

adamjpepper · 2022-08-03T09:27:19Z

Hello, got the same issue as you. May I ask you what command to use to come back to PADD version 3.7.1 please? Thank you

cd ~
wget -N https://raw.githubusercontent.com/pi-hole/PADD/v3.7.1/padd.sh

ChappyEight · 2022-08-04T02:26:09Z

Hello, got the same issue as you. May I ask you what command to use to come back to PADD version 3.7.1 please? Thank you

cd ~ wget -N https://raw.githubusercontent.com/pi-hole/PADD/v3.7.1/padd.sh

I tried this but now I'm getting a "Cannot write to 'padd.sh' (No space left on device)." error.

Anyone else have this?

adamjpepper · 2022-08-04T02:29:51Z

Hello, got the same issue as you. May I ask you what command to use to come back to PADD version 3.7.1 please? Thank you

cd ~ wget -N https://raw.githubusercontent.com/pi-hole/PADD/v3.7.1/padd.sh

I tried this but now I'm getting a "Cannot write to 'padd.sh' (No space left on device)." error.

Anyone else have this?

It's probably because the FTL log is so bit b/c of this bug. Just copy cp /dev/null /var/log/FTL.log to zero it out.

ChappyEight · 2022-08-04T02:31:09Z

Thank you.

Boltsfan91 · 2022-08-04T15:38:31Z

Hi, I am having the same issue of no space left on device, but running the cp /dev/null command above does not seem to allow it to download. Any ideas?

ChappyEight · 2022-08-04T15:40:16Z

This worked for me; (Removes packages and dependencies that are no longer required and deletes query log history)

sudo apt-get clean

sudo apt-get autoremove

sudo service pihole-FTL stop

sudo rm /etc/pihole/pihole-FTL.db

sudo service pihole-FTL start

But, I'm a novice so that may not be advised. Someone else should chime in.

baz069 · 2022-08-04T15:43:28Z

Hi, I am having the same issue of no space left on device, but running the cp /dev/null command above does not seem to allow it to download. Any ideas?

delete the log file completely, it will be recreated automatically.

sudo rm /var/log/pihole/FTL.log

Boltsfan91 · 2022-08-04T15:44:50Z

Hi, I am having the same issue of no space left on device, but running the cp /dev/null command above does not seem to allow it to download. Any ideas?

delete the log file completely, it will be recreated automatically.

sudo rm /var/log/pihole/FTL.log

Thanks, that worked. I was able to download the file, but now what do I do with it? Sorry, new to linux.

baz069 · 2022-08-04T15:47:19Z

Hi, I am having the same issue of no space left on device, but running the cp /dev/null command above does not seem to allow it to download. Any ideas?

delete the log file completely, it will be recreated automatically.
sudo rm /var/log/pihole/FTL.log

Thanks, that worked. I was able to download the file, but now what do I do with it? Sorry, new to linux.

cd ~
wget -N https://raw.githubusercontent.com/pi-hole/PADD/v3.7.1/padd.sh
sudo chmod +x padd.sh

reboot and you should be good to go

Boltsfan91 · 2022-08-04T15:48:12Z

Hi, I am having the same issue of no space left on device, but running the cp /dev/null command above does not seem to allow it to download. Any ideas?

delete the log file completely, it will be recreated automatically.
sudo rm /var/log/pihole/FTL.log

Thanks, that worked. I was able to download the file, but now what do I do with it? Sorry, new to linux.

cd ~ wget -N https://raw.githubusercontent.com/pi-hole/PADD/v3.7.1/padd.sh sudo chmod +x padd.sh

reboot and you should be good to go

You're a goddamn genius. Thank you so much!

baz069 · 2022-08-04T15:49:59Z

Hi, I am having the same issue of no space left on device, but running the cp /dev/null command above does not seem to allow it to download. Any ideas?

delete the log file completely, it will be recreated automatically.
sudo rm /var/log/pihole/FTL.log

Thanks, that worked. I was able to download the file, but now what do I do with it? Sorry, new to linux.

cd ~ wget -N https://raw.githubusercontent.com/pi-hole/PADD/v3.7.1/padd.sh sudo chmod +x padd.sh
reboot and you should be good to go

You're a goddamn genius. Thank you so much!

Dunno about that, i've just been through the same pain! 🤣

eric-pierce · 2022-08-04T19:31:12Z

Where is padd.sh located in the docker image? I took a look at the dockerfile and have looked in the usual places within the image itself and haven't been able to find it. I'm running into similar freezing issues, my current FTL log isn't too large, but I'd like to verify the padd version and see if downgrading resolves my freezing issues.

yubiuser · 2022-08-04T19:46:15Z

Where is padd.sh located in the docker image?

It's not part of the docker image (yet: pi-hole/docker-pi-hole#1105)

zangaby · 2022-08-09T07:32:05Z

[2022-08-08 21:46:46.035 23451/T23452] WARN: Could not accept() in listener() (/__w/FTL/FTL/src/api/socket.c:267): Bad file descriptor
[2022-08-08 21:46:46.035 23451/T23452] IPv4 telnet error: Success (0)
[2022-08-08 21:46:46.035 23451/T23452] WARN: Could not accept() in listener() (/__w/FTL/FTL/src/api/socket.c:267): Bad file descriptor
[2022-08-08 21:46:46.035 23451/T23452] IPv4 telnet error: Success (0)
[2022-08-08 21:46:46.035 23451/T23452] WARN: Could not accept() in listener() (/__w/FTL/FTL/src/api/socket.c:267): Bad file descriptor
[2022-08-08 21:46:46.035 23451/T23452] IPv4 telnet error: Success (0)
[2022-08-08 21:46:46.035 23451/T23452] WARN: Could not accept() in listener() (/__w/FTL/FTL/src/api/socket.c:267): Bad file descriptor
[2022-08-08 21:46:46.035 23451/T23452] IPv4 telnet error: Success (0)
[2022-08-08 21:46:46.035 23451/T23452] WARN: Could not accept() in listener() (/__w/FTL/FTL/src/api/socket.c:267): Bad file descriptor

Pi-hole v5.11.4 FTL v5.16.1 Web Interface v5.13
PADD 3.80

cmdshft · 2022-08-16T06:22:38Z

21 days... Any progress on this?

yubiuser · 2022-08-16T06:38:21Z

It's fixed already. The next Pi-hole version will include the fix.

turbodlr · 2022-08-17T17:43:39Z

So if I do pihole -up and it updates pihole I can go ahead and update PADD to latest version?

yubiuser · 2022-08-17T17:48:33Z

If your're still on v3.71 you should not upgrade PADD now. Once there is a new version of Pihole via pihole -up, it's safe to upgrade PADD as well. I have a number of PRs for PADD as well, so expect a v3.8.1 soon after the next Pi-hole update.

yubiuser · 2022-08-18T11:08:28Z

A FTL hotfix (v5.16.2) has been released just now. This specifically addressed the PADD v3.8.0 crashes. You can now upgrade your Pi-hole version and should be able to run latest PADD withouth crashes. The log spam should also be reduced, the remaining warnings like

[2022-07-29 18:30:59.146 6051/T8643] WARN: Could not write() everything in ssend() [/__w/FTL/FTL/src/api/socket.c:245]: Connection reset by peer
[2022-07-29 18:30:59.147 6051/T8643] WARN: Could not write() everything in ssend() [/__w/FTL/FTL/src/api/socket.c:245]: Broken pipe
[2022-07-29 18:30:59.147 6051/T8643] WARN: Could not write() everything in ssend() [/__w/FTL/FTL/src/api/socket.c:245]: Broken pipe
[2022-07-29 18:30:59.147 6051/T8643] WARN: Could not write() everything in ssend() [/__w/FTL/FTL/src/api/socket.c:245]: Broken pipe
[2022-07-29 18:31:14.307 6051/T8695] WARN: Could not write() everything in ssend() [/__w/FTL/FTL/src/api/socket.c:245]: Broken pipe

will be addressed with the upcoming PADD v3.8.1

yubiuser added the Bug label Jul 28, 2022

DL6ER mentioned this issue Jul 31, 2022

Major rewrite of the socket processing routines pi-hole/FTL#1396

Merged

5 tasks

DL6ER added the WIP label Jul 31, 2022

eiddor mentioned this issue Aug 8, 2022

WARN: Could not accept() in listener() (/__w/FTL/FTL/src/api/socket.c:267): Bad file descriptor klutchell/balena-pihole#162

Closed

This was referenced Aug 17, 2022

Hotfix/padd issues pi-hole/FTL#1401

Closed

Hotfix padd issues pi-hole/FTL#1402

Merged

yubiuser closed this as completed Aug 18, 2022

PADD 3.8 crashing pi-hole #252

PADD 3.8 crashing pi-hole #252

Comments

Gothico74 commented Jul 27, 2022

Gothico74 commented Jul 27, 2022

yubiuser commented Jul 27, 2022

Gothico74 commented Jul 27, 2022

jbutters110 commented Jul 27, 2022 • edited

ytsejam1138 commented Jul 27, 2022

cmdshft commented Jul 28, 2022 • edited

yubiuser commented Jul 28, 2022

jbutters110 commented Jul 28, 2022

Gothico74 commented Jul 28, 2022

mejndp commented Jul 28, 2022

Gothico74 commented Jul 28, 2022

Gothico74 commented Jul 28, 2022

mejndp commented Jul 28, 2022

polyclementc commented Jul 28, 2022

twerthi commented Jul 29, 2022

mejndp commented Jul 29, 2022

mejndp commented Jul 29, 2022

yubiuser commented Jul 29, 2022 • edited

mejndp commented Jul 29, 2022

yubiuser commented Jul 29, 2022

yubiuser commented Jul 29, 2022

DL6ER commented Jul 29, 2022

yubiuser commented Jul 30, 2022

howardCodeGit commented Aug 1, 2022

baz069 commented Aug 1, 2022 • edited

yubiuser commented Aug 1, 2022

adamjpepper commented Aug 3, 2022

ChappyEight commented Aug 4, 2022

adamjpepper commented Aug 4, 2022

ChappyEight commented Aug 4, 2022

Boltsfan91 commented Aug 4, 2022

ChappyEight commented Aug 4, 2022

baz069 commented Aug 4, 2022

Boltsfan91 commented Aug 4, 2022

baz069 commented Aug 4, 2022 • edited

Boltsfan91 commented Aug 4, 2022

baz069 commented Aug 4, 2022

eric-pierce commented Aug 4, 2022

yubiuser commented Aug 4, 2022

zangaby commented Aug 9, 2022 • edited

cmdshft commented Aug 16, 2022

yubiuser commented Aug 16, 2022

turbodlr commented Aug 17, 2022

yubiuser commented Aug 17, 2022

yubiuser commented Aug 18, 2022

jbutters110 commented Jul 27, 2022 •

edited

cmdshft commented Jul 28, 2022 •

edited

yubiuser commented Jul 29, 2022 •

edited

baz069 commented Aug 1, 2022 •

edited

baz069 commented Aug 4, 2022 •

edited

zangaby commented Aug 9, 2022 •

edited