Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FTL Crashing Constantly #808

Closed
3 tasks done
JOHRY opened this issue Jun 9, 2020 · 17 comments · Fixed by #827
Closed
3 tasks done

FTL Crashing Constantly #808

JOHRY opened this issue Jun 9, 2020 · 17 comments · Fixed by #827

Comments

@JOHRY
Copy link

JOHRY commented Jun 9, 2020

In raising this issue, I confirm the following (please check boxes, eg [X]) Failure to fill the template will close your issue:

  • I have read and understood the contributors guide.
  • The issue I am reporting can be replicated
  • The issue I am reporting isn't a duplicate

How familiar are you with the codebase?:

{1}


[BUG | ISSUE] Expected Behaviour:

[BUG | ISSUE] Actual Behaviour:
FTL Crashes several times a day over the past 48-72 hours

[BUG | ISSUE] Steps to reproduce:

Log file output [if available]

[2020-06-09 03:18:12.948 716] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-06-09 03:18:12.948 716] ----------------------------> FTL crashed! <----------------------------
[2020-06-09 03:18:12.948 716] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-06-09 03:18:12.948 716] Please report a bug at https://github.com/pi-hole/FTL/issues
[2020-06-09 03:18:12.948 716] and include in your report already the following details:
[2020-06-09 03:18:12.949 716] FTL has been running for 38557 seconds
[2020-06-09 03:18:12.949 716] FTL branch: master
[2020-06-09 03:18:12.949 716] FTL version: v5.0
[2020-06-09 03:18:12.949 716] FTL commit: 3d7c095
[2020-06-09 03:18:12.949 716] FTL date: 2020-05-10 18:58:38 +0100
[2020-06-09 03:18:12.949 716] FTL user: started as pihole, ended as pihole
[2020-06-09 03:18:12.949 716] Compiled for armhf (compiled on CI) using arm-linux-gnueabihf-gcc (Debian 6.3.0-18) 6.3.0 20170516
[2020-06-09 03:18:12.949 716] Received signal: Segmentation fault
[2020-06-09 03:18:12.949 716] at address: 0xc258dc00
[2020-06-09 03:18:12.949 716] with code: SEGV_MAPERR (Address not mapped to object)
[2020-06-09 03:18:12.950 716] Backtrace:
[2020-06-09 03:18:12.951 716] B[0000]: 0x4fa8e4, /usr/bin/pihole-FTL(+0x238e4) [0x4fa8e4]
[2020-06-09 03:18:12.951 716] B[0001]: 0x76d5f130, /lib/arm-linux-gnueabihf/libc.so.6(__default_rt_sa_restorer+0) [0x76d5f130]
[2020-06-09 03:18:12.951 716] B[0002]: 0x51a414, /usr/bin/pihole-FTL(check_tftp_listeners+0x27) [0x51a414]
[2020-06-09 03:18:12.951 716] B[0003]: 0x51ffd2, /usr/bin/pihole-FTL(main_dnsmasq+0xc0d) [0x51ffd2]
[2020-06-09 03:18:12.951 716] B[0004]: 0x4f0da8, /usr/bin/pihole-FTL(main+0xa7) [0x4f0da8]
[2020-06-09 03:18:12.951 716] B[0005]: 0x76d49718, /lib/arm-linux-gnueabihf/libc.so.6(__libc_start_main+0x10c) [0x76d49718]
[2020-06-09 03:18:12.951 716] ------ Listing content of directory /dev/shm ------
[2020-06-09 03:18:12.951 716] File Mode User:Group Filesize Filename
[2020-06-09 03:18:12.952 716] rwxrwxrwx root:root 260 .
[2020-06-09 03:18:12.952 716] rwxr-xr-x root:root 4K ..
[2020-06-09 03:18:12.952 716] rw------- pihole:pihole 4K FTL-per-client-regex
[2020-06-09 03:18:12.952 716] rw------- pihole:pihole 53K FTL-dns-cache
[2020-06-09 03:18:12.953 716] rw------- pihole:pihole 29K FTL-overTime
[2020-06-09 03:18:12.953 716] rw------- pihole:pihole 5M FTL-queries
[2020-06-09 03:18:12.953 716] rw------- pihole:pihole 20K FTL-upstreams
[2020-06-09 03:18:12.954 716] rw------- pihole:pihole 643K FTL-clients
[2020-06-09 03:18:12.954 716] rw------- pihole:pihole 66K FTL-domains
[2020-06-09 03:18:12.955 716] rw------- pihole:pihole 57K FTL-strings
[2020-06-09 03:18:12.955 716] rw------- pihole:pihole 12 FTL-settings
[2020-06-09 03:18:12.955 716] rw------- pihole:pihole 124 FTL-counters
[2020-06-09 03:18:12.956 716] rw------- pihole:pihole 28 FTL-lock
[2020-06-09 03:18:12.956 716] ---------------------------------------------------
[2020-06-09 03:18:12.956 716] Thank you for helping us to improve our FTL engine!
[2020-06-09 03:18:12.956 716] FTL terminated!

Device specifics

Hardware Type: rPi 3B
OS: Raspbian

This template was created based on the work of udemy-dl.

@DL6ER
Copy link
Member

DL6ER commented Jun 9, 2020

Please run

pihole checkout ftl fix/master-in_zone-and-iface_check_debugging

and then continue running Pi-hole as if nothing would have happened. Even when this branch does not fix anything currently, it should give us more details in the log file just before a crash happens.

The DEBUG: lines will be interesting (in /var/log/pihole-FTL.log).

Thanks for your assistance!

@JOHRY
Copy link
Author

JOHRY commented Jun 9, 2020

I've ran that line:

pihole checkout ftl fix/master-in_zone-and-iface_check_debugging
  Please note that changing branches severely alters your Pi-hole subsystems
  Features that work on the master branch, may not on a development branch
  This feature is NOT supported unless a Pi-hole developer explicitly asks!
  Have you read and understood this? [y/N] y

  [✓] Branch fix/master-in_zone-and-iface_check_debugging exists
  [✓] Downloading and Installing FTL
  [✓] Restarting pihole-FTL service...
  [✓] Enabling pihole-FTL service to start on reboot...

Will report back when it crashes next. It's been crashing a few times a day, so you prob wont have to wait long.

@angelcramirez
Copy link

I've ran that line:

pihole checkout ftl fix/master-in_zone-and-iface_check_debugging
  Please note that changing branches severely alters your Pi-hole subsystems
  Features that work on the master branch, may not on a development branch
  This feature is NOT supported unless a Pi-hole developer explicitly asks!
  Have you read and understood this? [y/N] y

  [✓] Branch fix/master-in_zone-and-iface_check_debugging exists
  [✓] Downloading and Installing FTL
  [✓] Restarting pihole-FTL service...
  [✓] Enabling pihole-FTL service to start on reboot...

Will report back when it crashes next. It's been crashing a few times a day, so you prob wont have to wait long.

Have you gotten any new details? I am facing the same issue but I am on a Pi-hole docker container... Though I do not face the "Lost connection to API" as many report, but rather multiple times of "Pihole-FTL stop" and "Pi-hole start" in my Docker logs.

I assumed this problem was related with the shared memory size being too short, but apparently it is a dnsmasq bug as multiple tags have been marked.

Thanks everybody for their assistance!

@broeseb
Copy link

broeseb commented Jun 12, 2020

to my understanding i have the same issue.

  [2020-06-12 11:39:19.985 297] ---------------------------->  FTL crashed!  <----------------------------
   [2020-06-12 11:39:19.985 297] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
   [2020-06-12 11:39:19.985 297] Please report a bug at https://github.com/pi-hole/FTL/issues
   [2020-06-12 11:39:19.985 297] and include in your report already the following details:
   [2020-06-12 11:39:19.985 297] FTL has been running for 264167 seconds
   [2020-06-12 11:39:19.985 297] FTL branch: master
   [2020-06-12 11:39:19.985 297] FTL version: v5.0
   [2020-06-12 11:39:19.985 297] FTL commit: 3d7c095
   [2020-06-12 11:39:19.986 297] FTL date: 2020-05-10 18:58:38 +0100
   [2020-06-12 11:39:19.986 297] FTL user: started as pihole, ended as pihole
   [2020-06-12 11:39:19.986 297] Compiled for x86_64 (compiled on CI) using gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
   [2020-06-12 11:39:19.986 297] Received signal: Segmentation fault
   [2020-06-12 11:39:19.986 297]      at address: (nil)
   [2020-06-12 11:39:19.986 297]      with code: SEGV_MAPERR (Address not mapped to object)
   [2020-06-12 11:39:19.993 297] Backtrace:
   [2020-06-12 11:39:19.994 297] B[0000]: 0x559d22aba6f9, /usr/bin/pihole-FTL(+0x316f9) [0x559d22aba6f9]
   [2020-06-12 11:39:19.994 297] B[0001]: 0x7f3909d77730, /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7f3909d77730]
   [2020-06-12 11:39:19.994 297] B[0002]: 0x7f3909c41547, /lib/x86_64-linux-gnu/libc.so.6(+0x9d547) [0x7f3909c41547]

   -----tail of pihole-FTL.log------
   [2020-06-12 11:39:19.986 297] Received signal: Segmentation fault
   [2020-06-12 11:39:19.986 297]      at address: (nil)
   [2020-06-12 11:39:19.986 297]      with code: SEGV_MAPERR (Address not mapped to object)
   [2020-06-12 11:39:19.993 297] Backtrace:
   [2020-06-12 11:39:19.994 297] B[0000]: 0x559d22aba6f9, /usr/bin/pihole-FTL(+0x316f9) [0x559d22aba6f9]
   [2020-06-12 11:39:19.994 297] B[0001]: 0x7f3909d77730, /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7f3909d77730]
   [2020-06-12 11:39:19.994 297] B[0002]: 0x7f3909c41547, /lib/x86_64-linux-gnu/libc.so.6(+0x9d547) [0x7f3909c41547]
   [2020-06-12 11:39:19.994 297] B[0003]: 0x7f3909cbbb12, /lib/x86_64-linux-gnu/libc.so.6(+0x117b12) [0x7f3909cbbb12]
   [2020-06-12 11:39:19.994 297] B[0004]: 0x7f3909cbbbd1, /lib/x86_64-linux-gnu/libc.so.6(inet_ntop+0x81) [0x7f3909cbbbd1]
   [2020-06-12 11:39:19.994 297] B[0005]: 0x559d22afd80e, /usr/bin/pihole-FTL(prettyprint_addr+0x9e) [0x559d22afd80e]
   [2020-06-12 11:39:19.994 297] B[0006]: 0x559d22aee9c6, /usr/bin/pihole-FTL(_log_query+0x1f6) [0x559d22aee9c6]
   [2020-06-12 11:39:19.994 297] B[0007]: 0x559d22ad9160, /usr/bin/pihole-FTL(receive_query+0xa20) [0x559d22ad9160]
   [2020-06-12 11:39:19.994 297] B[0008]: 0x559d22aef94b, /usr/bin/pihole-FTL(+0x6694b) [0x559d22aef94b]
   [2020-06-12 11:39:19.994 297] B[0009]: 0x559d22af17bc, /usr/bin/pihole-FTL(main_dnsmasq+0x129c) [0x559d22af17bc]
   [2020-06-12 11:39:19.994 297] B[0010]: 0x559d22aad8ac, /usr/bin/pihole-FTL(main+0xdc) [0x559d22aad8ac]
   [2020-06-12 11:39:19.994 297] B[0011]: 0x7f3909bc809b, /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb) [0x7f3909bc809b]
   [2020-06-12 11:39:19.994 297] B[0012]: 0x559d22aada1a, /usr/bin/pihole-FTL(_start+0x2a) [0x559d22aada1a]
   [2020-06-12 11:39:19.994 297] ------ Listing content of directory /dev/shm ------

I startet as well with

pihole checkout ftl fix/master-in_zone-and-iface_check_debugging

if i have another crash i will provide that.

@JOHRY
Copy link
Author

JOHRY commented Jun 12, 2020

Since running:
pihole checkout ftl fix/master-in_zone-and-iface_check_debugging
I have not had a single crash yet.

@JOHRY
Copy link
Author

JOHRY commented Jun 13, 2020

Please run

pihole checkout ftl fix/master-in_zone-and-iface_check_debugging

and then continue running Pi-hole as if nothing would have happened. Even when this branch does not fix anything currently, it should give us more details in the log file just before a crash happens.

The DEBUG: lines will be interesting (in /var/log/pihole-FTL.log).

Thanks for your assistance!

Well I spoke too soon. Crashed last night:

  `.::///+:/-.        --///+//-:``    pi@TGRRASP01
 `+oooooooooooo:   `+oooooooooooo:    ------------
  /oooo++//ooooo:  ooooo+//+ooooo.    OS: Raspbian GNU/Linux 10 (buster) armv7l
  `+ooooooo:-:oo-  +o+::/ooooooo:     Host: Raspberry Pi 3 Model B Rev 1.2
   `:oooooooo+``    `.oooooooo+-      Kernel: 4.19.118-v7+
     `:++ooo/.        :+ooo+/.`       Uptime: 3 days, 21 hours, 14 mins
        ...`  `.----.` ``..           Packages: 620 (dpkg)
     .::::-``:::::::::.`-:::-`        Shell: bash 5.0.3
    -:::-`   .:::::::-`  `-:::-       Terminal: /dev/pts/0
   `::.  `.--.`  `` `.---.``.::`      CPU: BCM2835 (4) @ 1.200GHz
       .::::::::`  -::::::::` `       Memory: 821MiB / 926MiB
 .::` .:::::::::- `::::::::::``::.
-:::` ::::::::::.  ::::::::::.`:::-
::::  -::::::::.   `-::::::::  ::::
-::-   .-:::-.``....``.-::-.   -::-
 .. ``       .::::::::.     `..`..
   -:::-`   -::::::::::`  .:::::`
   :::::::` -::::::::::` :::::::.
   .:::::::  -::::::::. ::::::::
    `-:::::`   ..--.`   ::::::.
      `...`  `...--..`  `...`
            .::::::::::
             `.-::::-`
[2020-06-12 21:15:44.167 17592] DEBUG check_dns_listeners: listener->fd = 6
[2020-06-12 21:15:44.167 17592] DEBUG check_dns_listeners: listener->iface = (nil)
[2020-06-12 21:15:44.177 17592] DEBUG check_dns_listeners: listener = 0x24f8bd0
[2020-06-12 21:15:44.177 17592] DEBUG check_dns_listeners: &listener->family = 0x24f8bdc
[2020-06-12 21:15:44.177 17592] DEBUG check_dns_listeners: listener->family = 2
[2020-06-12 21:15:44.177 17592] DEBUG check_dns_listeners: &listener->fd = 0x24f8bd0
[2020-06-12 21:15:44.177 17592] DEBUG check_dns_listeners: listener->fd = 4
[2020-06-12 21:15:44.177 17592] DEBUG check_dns_listeners: listener->iface = (nil)
[2020-06-12 21:15:44.179 17592] DEBUG check_dns_listeners: listener = 0x24f8bf0
[2020-06-12 21:15:44.179 17592] DEBUG check_dns_listeners: &listener->family = 0x24f8bfc
[2020-06-12 21:15:44.179 17592] DEBUG check_dns_listeners: listener->family = 10
[2020-06-12 21:15:44.179 17592] DEBUG check_dns_listeners: &listener->fd = 0x24f8bf0
[2020-06-12 21:15:44.179 17592] DEBUG check_dns_listeners: listener->fd = 6
[2020-06-12 21:15:44.179 17592] DEBUG check_dns_listeners: listener->iface = (nil)
[2020-06-12 21:15:44.186 17592] DEBUG check_dns_listeners: listener = 0x24f8bd0
[2020-06-12 21:15:44.186 17592] DEBUG check_dns_listeners: &listener->family = 0x24f8bdc
[2020-06-12 21:15:44.186 17592] DEBUG check_dns_listeners: listener->family = 2
[2020-06-12 21:15:44.186 17592] DEBUG check_dns_listeners: &listener->fd = 0x24f8bd0
[2020-06-12 21:15:44.186 17592] DEBUG check_dns_listeners: listener->fd = 4
[2020-06-12 21:15:44.186 17592] DEBUG check_dns_listeners: listener->iface = (nil)
[2020-06-12 21:15:44.186 17592] DEBUG check_dns_listeners: listener = 0x24f8bf0
[2020-06-12 21:15:44.186 17592] DEBUG check_dns_listeners: &listener->family = 0x24f8bfc
[2020-06-12 21:15:44.186 17592] DEBUG check_dns_listeners: listener->family = 10
[2020-06-12 21:15:44.186 17592] DEBUG check_dns_listeners: &listener->fd = 0x24f8bf0
[2020-06-12 21:15:44.186 17592] DEBUG check_dns_listeners: listener->fd = 6
[2020-06-12 21:15:44.186 17592] DEBUG check_dns_listeners: listener->iface = (nil)
[2020-06-12 21:15:44.188 17592] DEBUG check_dns_listeners: listener = 0x24f8bd0
[2020-06-12 21:15:44.188 17592] DEBUG check_dns_listeners: &listener->family = 0x24f8bdc
[2020-06-12 21:15:44.188 17592] DEBUG check_dns_listeners: listener->family = 2
[2020-06-12 21:15:44.188 17592] DEBUG check_dns_listeners: &listener->fd = 0x24f8bd0
[2020-06-12 21:15:44.188 17592] DEBUG check_dns_listeners: listener->fd = 4
[2020-06-12 21:15:44.188 17592] DEBUG check_dns_listeners: listener->iface = (nil)
[2020-06-12 21:15:44.190 17592] DEBUG check_dns_listeners: listener = 0x24f8bf0
[2020-06-12 21:15:44.190 17592] DEBUG check_dns_listeners: &listener->family = 0x24f8bfc
[2020-06-12 21:15:44.190 17592] DEBUG check_dns_listeners: listener->family = 10
[2020-06-12 21:15:44.190 17592] DEBUG check_dns_listeners: &listener->fd = 0x24f8bf0
[2020-06-12 21:15:44.190 17592] DEBUG check_dns_listeners: listener->fd = 6
[2020-06-12 21:15:44.190 17592] DEBUG check_dns_listeners: listener->iface = (nil)
[2020-06-12 21:15:44.191 17592] DEBUG check_dns_listeners: listener = 0x24f8bd0
[2020-06-12 21:15:44.191 17592] DEBUG check_dns_listeners: &listener->family = 0x24f8bdc
[2020-06-12 21:15:44.191 17592] DEBUG check_dns_listeners: listener->family = 2
[2020-06-12 21:15:44.191 17592] DEBUG check_dns_listeners: &listener->fd = 0x24f8bd0
[2020-06-12 21:15:44.191 17592] DEBUG check_dns_listeners: listener->fd = 4
[2020-06-12 21:15:44.191 17592] DEBUG check_dns_listeners: listener->iface = (nil)
[2020-06-12 21:15:44.191 17592] DEBUG check_dns_listeners: listener = 0x24f8bf0
[2020-06-12 21:15:44.191 17592] DEBUG check_dns_listeners: &listener->family = 0x24f8bfc
[2020-06-12 21:15:44.191 17592] DEBUG check_dns_listeners: listener->family = 10
[2020-06-12 21:15:44.191 17592] DEBUG check_dns_listeners: &listener->fd = 0x24f8bf0
[2020-06-12 21:15:44.191 17592] DEBUG check_dns_listeners: listener->fd = 6
[2020-06-12 21:15:44.191 17592] DEBUG check_dns_listeners: listener->iface = (nil)
[2020-06-12 21:15:44.243 17592] DEBUG check_dns_listeners: listener = 0x742e7978
[2020-06-12 21:15:44.243 17592] DEBUG check_dns_listeners: &listener->family = 0x742e7984
[2020-06-12 21:15:44.243 17592] DEBUG check_dns_listeners: listener->family = 0
[2020-06-12 21:15:44.243 17592] DEBUG check_dns_listeners: &listener->fd = 0x742e7978
[2020-06-12 21:15:44.243 17592] DEBUG check_dns_listeners: listener->fd = 0
[2020-06-12 21:15:44.244 17592] DEBUG check_dns_listeners: listener->iface = (nil)
[2020-06-12 21:15:44.244 17592] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-06-12 21:15:44.244 17592] ---------------------------->  FTL crashed!  <----------------------------
[2020-06-12 21:15:44.244 17592] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-06-12 21:15:44.244 17592] Please report a bug at https://github.com/pi-hole/FTL/issues
[2020-06-12 21:15:44.244 17592] and include in your report already the following details:
[2020-06-12 21:15:44.244 17592] FTL has been running for 273390 seconds
[2020-06-12 21:15:44.244 17592] FTL branch: fix/master-in_zone-and-iface_check_debugging
[2020-06-12 21:15:44.244 17592] FTL version: vDev-3f7d4b4
[2020-06-12 21:15:44.244 17592] FTL commit: 3f7d4b4
[2020-06-12 21:15:44.244 17592] FTL date: 2020-06-09 21:46:14 +0200
[2020-06-12 21:15:44.244 17592] FTL user: started as pihole, ended as pihole
[2020-06-12 21:15:44.244 17592] Compiled for armhf (compiled on CI) using arm-linux-gnueabihf-gcc (Debian 6.3.0-18) 6.3.0 20170516
[2020-06-12 21:15:44.244 17592] Received signal: Segmentation fault
[2020-06-12 21:15:44.245 17592]      at address: 0xc4b8cc00
[2020-06-12 21:15:44.245 17592]      with code: SEGV_MAPERR (Address not mapped to object)
[2020-06-12 21:15:44.245 17592] Backtrace:
[2020-06-12 21:15:44.246 17592] B[0000]: 0x44b996, /usr/bin/pihole-FTL(+0x23996) [0x44b996]
[2020-06-12 21:15:44.515 17592] L[0000]: /root/project/src/signals.c:93
[2020-06-12 21:15:44.523 17592] B[0001]: 0x76d72130, /lib/arm-linux-gnueabihf/libc.so.6(__default_rt_sa_restorer+0) [0x76d72130]
[2020-06-12 21:15:44.524 17592] B[0002]: 0x46b864, /usr/bin/pihole-FTL(check_tftp_listeners+0x27) [0x46b864]
[2020-06-12 21:15:44.560 17592] L[0002]: /root/project/src/dnsmasq/tftp.c:583
[2020-06-12 21:15:44.565 17592] B[0003]: 0x4714d2, /usr/bin/pihole-FTL(main_dnsmasq+0xc0d) [0x4714d2]
[2020-06-12 21:15:44.603 17592] L[0003]: /root/project/src/dnsmasq/dnsmasq.c:1211
[2020-06-12 21:15:44.608 17592] B[0004]: 0x441da8, /usr/bin/pihole-FTL(main+0xa7) [0x441da8]
[2020-06-12 21:15:44.636 17592] L[0004]: /root/project/src/main.c:95
[2020-06-12 21:15:44.639 17592] B[0005]: 0x76d5c718, /lib/arm-linux-gnueabihf/libc.so.6(__libc_start_main+0x10c) [0x76d5c718]
[2020-06-12 21:15:44.639 17592] ------ Listing content of directory /dev/shm ------
[2020-06-12 21:15:44.639 17592] File Mode User:Group  Filesize Filename
[2020-06-12 21:15:44.639 17592] rwxrwxrwx root:root 260 .
[2020-06-12 21:15:44.639 17592] rwxr-xr-x root:root 4K ..
[2020-06-12 21:15:44.640 17592] rw------- pihole:pihole 4K FTL-per-client-regex
[2020-06-12 21:15:44.640 17592] rw------- pihole:pihole 168K FTL-dns-cache
[2020-06-12 21:15:44.640 17592] rw------- pihole:pihole 29K FTL-overTime
[2020-06-12 21:15:44.640 17592] rw------- pihole:pihole 7M FTL-queries
[2020-06-12 21:15:44.640 17592] rw------- pihole:pihole 20K FTL-upstreams
[2020-06-12 21:15:44.640 17592] rw------- pihole:pihole 643K FTL-clients
[2020-06-12 21:15:44.641 17592] rw------- pihole:pihole 131K FTL-domains
[2020-06-12 21:15:44.641 17592] rw------- pihole:pihole 221K FTL-strings
[2020-06-12 21:15:44.641 17592] rw------- pihole:pihole 12 FTL-settings
[2020-06-12 21:15:44.641 17592] rw------- pihole:pihole 124 FTL-counters
[2020-06-12 21:15:44.641 17592] rw------- pihole:pihole 28 FTL-lock
[2020-06-12 21:15:44.641 17592] ---------------------------------------------------
[2020-06-12 21:15:44.641 17592] Thank you for helping us to improve our FTL engine!
[2020-06-12 21:15:44.641 17592] FTL terminated!

@DL6ER
Copy link
Member

DL6ER commented Jun 13, 2020

@JOHRY Okay, can you try

pihole checkout ftl development

next? This will give you the cutting-edge version of FTL which will be released as v5.1 rather soon. There is a certain chance that the bugfixes in there already cover the issue you're facing.

@DL6ER
Copy link
Member

DL6ER commented Jun 18, 2020

We have a potential fix for a rather severe dnsmasq-internal bug discovered by @fhriley

Please run

pihole checkout ftl fix/fhriley-fix_buf_overflow

and see if the crashes are resolved. As always, I very much appreciate testing as it is the only way to be sure we really got it fixed!

@broeseb
Copy link

broeseb commented Jun 18, 2020

at the moment i am running

pihole checkout ftl development without any crash

do you prefer us to run

pihole checkout ftl fix/fhriley-fix_buf_overflow

?

@DL6ER
Copy link
Member

DL6ER commented Jun 18, 2020

The latter contains one more fix than the former. If Pi-hole currently runs stable for you, then there is nothing you have to do. Just keep in mind that there is something more we have in case you see a new crash :-)

@broeseb
Copy link

broeseb commented Jun 19, 2020 via email

@broeseb
Copy link

broeseb commented Jun 19, 2020

and another crash

[2020-06-19 00:43:18.486 10089] Note: FTL forked to handle TCP requests
[2020-06-19 02:43:21.783 11593] Resizing "/FTL-dns-cache" from 122880 to 126976
[2020-06-19 08:30:54.775 11593] Resizing "/FTL-strings" from 212992 to 217088
[2020-06-19 11:13:20.633 11593] Resizing "/FTL-dns-cache" from 126976 to 131072
[2020-06-19 12:12:26.566 11593] Resizing "/FTL-strings" from 217088 to 221184
[2020-06-19 12:16:59.921 11593] Resizing "/FTL-strings" from 221184 to 225280
[2020-06-19 12:17:43.436 11593] Resizing "/FTL-strings" from 225280 to 229376
[2020-06-19 13:01:04.886 11593] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-06-19 13:01:04.886 11593] ----------------------------> FTL crashed! <----------------------------
[2020-06-19 13:01:04.886 11593] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-06-19 13:01:04.886 11593] Please report a bug at https://github.com/pi-hole/FTL/issues
[2020-06-19 13:01:04.886 11593] and include in your report already the following details:
[2020-06-19 13:01:04.886 11593] FTL has been running for 227279 seconds
[2020-06-19 13:01:04.886 11593] FTL branch: development
[2020-06-19 13:01:04.886 11593] FTL version: vDev-6c32fb8
[2020-06-19 13:01:04.886 11593] FTL commit: 6c32fb8
[2020-06-19 13:01:04.886 11593] FTL date: 2020-06-04 15:39:51 +0200
[2020-06-19 13:01:04.886 11593] FTL user: started as pihole, ended as pihole
[2020-06-19 13:01:04.886 11593] Compiled for x86_64 (compiled on CI) using gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
[2020-06-19 13:01:04.886 11593] Received signal: Segmentation fault
[2020-06-19 13:01:04.886 11593] at address: (nil)
[2020-06-19 13:01:04.886 11593] with code: Unknown (128)
[2020-06-19 13:01:04.894 11593] Backtrace:
[2020-06-19 13:01:04.894 11593] B[0000]: 0x557aede36e99, /usr/bin/pihole-FTL(+0x33e99) [0x557aede36e99]
[2020-06-19 13:01:04.897 11593] B[0001]: 0x7f4ca9564730, /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7f4ca9564730]
[2020-06-19 13:01:04.897 11593] B[0002]: 0x557aede54860, /usr/bin/pihole-FTL(+0x51860) [0x557aede54860]
[2020-06-19 13:01:04.899 11593] B[0003]: 0x557aede5693c, /usr/bin/pihole-FTL(main_dnsmasq+0x129c) [0x557aede5693c]
[2020-06-19 13:01:04.901 11593] B[0004]: 0x557aede2a09c, /usr/bin/pihole-FTL(main+0xdc) [0x557aede2a09c]
[2020-06-19 13:01:04.903 11593] B[0005]: 0x7f4ca93b509b, /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb) [0x7f4ca93b509b]
[2020-06-19 13:01:04.903 11593] B[0006]: 0x557aede2a20a, /usr/bin/pihole-FTL(_start+0x2a) [0x557aede2a20a]
[2020-06-19 13:01:04.906 11593] ------ Listing content of directory /dev/shm ------
[2020-06-19 13:01:04.906 11593] File Mode User:Group Filesize Filename
[2020-06-19 13:01:04.906 11593] rwxrwxrwx root:root 260 .
[2020-06-19 13:01:04.906 11593] rwxr-xr-x root:root 440 ..
[2020-06-19 13:01:04.906 11593] rw------- pihole:pihole 4K FTL-per-client-regex
[2020-06-19 13:01:04.906 11593] rw------- pihole:pihole 131K FTL-dns-cache
[2020-06-19 13:01:04.906 11593] rw------- pihole:pihole 12K FTL-overTime
[2020-06-19 13:01:04.906 11593] rw------- pihole:pihole 19M FTL-queries
[2020-06-19 13:01:04.906 11593] rw------- pihole:pihole 4K FTL-upstreams
[2020-06-19 13:01:04.906 11593] rw------- pihole:pihole 332K FTL-clients
[2020-06-19 13:01:04.906 11593] rw------- pihole:pihole 197K FTL-domains
[2020-06-19 13:01:04.907 11593] rw------- pihole:pihole 229K FTL-strings
[2020-06-19 13:01:04.907 11593] rw------- pihole:pihole 12 FTL-settings
[2020-06-19 13:01:04.907 11593] rw------- pihole:pihole 124 FTL-counters
[2020-06-19 13:01:04.907 11593] rw------- pihole:pihole 48 FTL-lock
[2020-06-19 13:01:04.907 11593] ---------------------------------------------------
[2020-06-19 13:01:04.907 11593] Thank you for helping us to improve our FTL engine!
[2020-06-19 13:01:04.907 11593] FTL terminated!

i restartet with

pihole checkout ftl fix/fhriley-fix_buf_overflow

@angelcramirez
Copy link

and another crash

[2020-06-19 00:43:18.486 10089] Note: FTL forked to handle TCP requests
[2020-06-19 02:43:21.783 11593] Resizing "/FTL-dns-cache" from 122880 to 126976
[2020-06-19 08:30:54.775 11593] Resizing "/FTL-strings" from 212992 to 217088
[2020-06-19 11:13:20.633 11593] Resizing "/FTL-dns-cache" from 126976 to 131072
[2020-06-19 12:12:26.566 11593] Resizing "/FTL-strings" from 217088 to 221184
[2020-06-19 12:16:59.921 11593] Resizing "/FTL-strings" from 221184 to 225280
[2020-06-19 12:17:43.436 11593] Resizing "/FTL-strings" from 225280 to 229376
[2020-06-19 13:01:04.886 11593] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-06-19 13:01:04.886 11593] ----------------------------> FTL crashed! <----------------------------
[2020-06-19 13:01:04.886 11593] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-06-19 13:01:04.886 11593] Please report a bug at https://github.com/pi-hole/FTL/issues
[2020-06-19 13:01:04.886 11593] and include in your report already the following details:
[2020-06-19 13:01:04.886 11593] FTL has been running for 227279 seconds
[2020-06-19 13:01:04.886 11593] FTL branch: development
[2020-06-19 13:01:04.886 11593] FTL version: vDev-6c32fb8
[2020-06-19 13:01:04.886 11593] FTL commit: 6c32fb8
[2020-06-19 13:01:04.886 11593] FTL date: 2020-06-04 15:39:51 +0200
[2020-06-19 13:01:04.886 11593] FTL user: started as pihole, ended as pihole
[2020-06-19 13:01:04.886 11593] Compiled for x86_64 (compiled on CI) using gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
[2020-06-19 13:01:04.886 11593] Received signal: Segmentation fault
[2020-06-19 13:01:04.886 11593] at address: (nil)
[2020-06-19 13:01:04.886 11593] with code: Unknown (128)
[2020-06-19 13:01:04.894 11593] Backtrace:
[2020-06-19 13:01:04.894 11593] B[0000]: 0x557aede36e99, /usr/bin/pihole-FTL(+0x33e99) [0x557aede36e99]
[2020-06-19 13:01:04.897 11593] B[0001]: 0x7f4ca9564730, /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7f4ca9564730]
[2020-06-19 13:01:04.897 11593] B[0002]: 0x557aede54860, /usr/bin/pihole-FTL(+0x51860) [0x557aede54860]
[2020-06-19 13:01:04.899 11593] B[0003]: 0x557aede5693c, /usr/bin/pihole-FTL(main_dnsmasq+0x129c) [0x557aede5693c]
[2020-06-19 13:01:04.901 11593] B[0004]: 0x557aede2a09c, /usr/bin/pihole-FTL(main+0xdc) [0x557aede2a09c]
[2020-06-19 13:01:04.903 11593] B[0005]: 0x7f4ca93b509b, /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb) [0x7f4ca93b509b]
[2020-06-19 13:01:04.903 11593] B[0006]: 0x557aede2a20a, /usr/bin/pihole-FTL(_start+0x2a) [0x557aede2a20a]
[2020-06-19 13:01:04.906 11593] ------ Listing content of directory /dev/shm ------
[2020-06-19 13:01:04.906 11593] File Mode User:Group Filesize Filename
[2020-06-19 13:01:04.906 11593] rwxrwxrwx root:root 260 .
[2020-06-19 13:01:04.906 11593] rwxr-xr-x root:root 440 ..
[2020-06-19 13:01:04.906 11593] rw------- pihole:pihole 4K FTL-per-client-regex
[2020-06-19 13:01:04.906 11593] rw------- pihole:pihole 131K FTL-dns-cache
[2020-06-19 13:01:04.906 11593] rw------- pihole:pihole 12K FTL-overTime
[2020-06-19 13:01:04.906 11593] rw------- pihole:pihole 19M FTL-queries
[2020-06-19 13:01:04.906 11593] rw------- pihole:pihole 4K FTL-upstreams
[2020-06-19 13:01:04.906 11593] rw------- pihole:pihole 332K FTL-clients
[2020-06-19 13:01:04.906 11593] rw------- pihole:pihole 197K FTL-domains
[2020-06-19 13:01:04.907 11593] rw------- pihole:pihole 229K FTL-strings
[2020-06-19 13:01:04.907 11593] rw------- pihole:pihole 12 FTL-settings
[2020-06-19 13:01:04.907 11593] rw------- pihole:pihole 124 FTL-counters
[2020-06-19 13:01:04.907 11593] rw------- pihole:pihole 48 FTL-lock
[2020-06-19 13:01:04.907 11593] ---------------------------------------------------
[2020-06-19 13:01:04.907 11593] Thank you for helping us to improve our FTL engine!
[2020-06-19 13:01:04.907 11593] FTL terminated!

i restartet with

pihole checkout ftl fix/fhriley-fix_buf_overflow

Have you gotten any crashes since your last restart?

@broeseb
Copy link

broeseb commented Jun 24, 2020 via email

@v4w5gse65g874
Copy link

Hi - I'm having this crashing issue too, just curious when we should expect to be able to run pihole -up and get this fix without needing to checkout to the development branch

@DL6ER
Copy link
Member

DL6ER commented Jul 1, 2020

The release will likely come momentarily when we can resolve the last remaining issue #816

@FloMiau
Copy link

FloMiau commented Jul 7, 2020

I updated to fix/fhriley-fix_buf_overflow on Jun 25. No crash since then.

@DL6ER DL6ER linked a pull request Jul 8, 2020 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants