Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FTL crashed! #1068

Closed
libertyx82 opened this issue Feb 18, 2021 · 20 comments
Closed

FTL crashed! #1068

libertyx82 opened this issue Feb 18, 2021 · 20 comments
Labels

Comments

@libertyx82
Copy link

libertyx82 commented Feb 18, 2021

Versions

  • Pi-hole: v5.2.4
  • FTL: v5.6

Platform

  • OS and version: Debian 6.3.0-18+deb9u1
  • Platform: VM (vmware)
[2021-02-18 00:10:39.349 23134/F650] Resizing "FTL-per-client-regex" from 4096 to (187 * 22) == 4114 (/dev/shm: 1.1MB used, 1.0GB total)
[2021-02-18 00:10:42.056 23110/F650] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2021-02-18 00:10:42.056 23110/F650] ---------------------------->  FTL crashed!  <----------------------------
[2021-02-18 00:10:42.056 23110/F650] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2021-02-18 00:10:42.056 23110/F650] Please report a bug at https://github.com/pi-hole/FTL/issues
[2021-02-18 00:10:42.056 23110/F650] and include in your report already the following details:
[2021-02-18 00:10:42.056 23110/F650] FTL has been running for 520839 seconds
[2021-02-18 00:10:42.056 23110/F650] FTL branch: master
[2021-02-18 00:10:42.056 23110/F650] FTL version: v5.6
[2021-02-18 00:10:42.056 23110/F650] FTL commit: da89cc6
[2021-02-18 00:10:42.056 23110/F650] FTL date: 2021-01-27 17:55:33 +0000
[2021-02-18 00:10:42.056 23110/F650] FTL user: started as pihole, ended as pihole
[2021-02-18 00:10:42.056 23110/F650] Compiled for x86_64 (compiled on CI) using gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
[2021-02-18 00:10:42.056 23110/F650] Process details: MID: 650
[2021-02-18 00:10:42.057 23110/F650]                  PID: 23110
[2021-02-18 00:10:42.057 23110/F650]                  TID: 23110
[2021-02-18 00:10:42.057 23110/F650]                  Name: pihole-FTL
[2021-02-18 00:10:42.057 23110/F650] Received signal: Segmentation fault
[2021-02-18 00:10:42.057 23110/F650]      at address: 0x7fafc4e1f000
[2021-02-18 00:10:42.057 23110/F650]      with code:  SEGV_MAPERR (Address not mapped to object)
[2021-02-18 00:10:42.057 23110/F650] Backtrace:
[2021-02-18 00:10:42.057 23110/F650] B[0000]: /usr/bin/pihole-FTL(+0x58f35) [0x5559ccaaff35]
[2021-02-18 00:10:42.060 23110/F650] L[0000]: N/A (0x58f35)
[2021-02-18 00:10:42.060 23110/F650] B[0001]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7fafc4c80730]
[2021-02-18 00:10:42.060 23110/F650] B[0002]: /usr/bin/pihole-FTL(get_per_client_regex+0x34) [0x5559ccaaf8e4]
[2021-02-18 00:10:42.061 23110/F650] L[0002]: N/A (0x588e4)
[2021-02-18 00:10:42.061 23110/F650] B[0003]: /usr/bin/pihole-FTL(match_regex+0x13e) [0x5559ccaa9ffe]
[2021-02-18 00:10:42.062 23110/F650] L[0003]: N/A (0x52ffe)
[2021-02-18 00:10:42.062 23110/F650] B[0004]: /usr/bin/pihole-FTL(+0x4b526) [0x5559ccaa2526]
[2021-02-18 00:10:42.063 23110/F650] L[0004]: N/A (0x4b526)
[2021-02-18 00:10:42.063 23110/F650] B[0005]: /usr/bin/pihole-FTL(_FTL_new_query+0x800) [0x5559ccaa3a30]
[2021-02-18 00:10:42.064 23110/F650] L[0005]: N/A (0x4ca30)
[2021-02-18 00:10:42.064 23110/F650] B[0006]: /usr/bin/pihole-FTL(tcp_request+0x38b) [0x5559ccadd02b]
[2021-02-18 00:10:42.065 23110/F650] L[0006]: N/A (0x8602b)
[2021-02-18 00:10:42.065 23110/F650] B[0007]: /usr/bin/pihole-FTL(+0x7c30d) [0x5559ccad330d]
[2021-02-18 00:10:42.065 23110/F650] L[0007]: N/A (0x7c30d)
[2021-02-18 00:10:42.066 23110/F650] B[0008]: /usr/bin/pihole-FTL(main_dnsmasq+0x127c) [0x5559ccad52ac]
[2021-02-18 00:10:42.067 23110/F650] L[0008]: N/A (0x7e2ac)
[2021-02-18 00:10:42.067 23110/F650] B[0009]: /usr/bin/pihole-FTL(main+0xec) [0x5559cca9b96c]
[2021-02-18 00:10:42.068 23110/F650] L[0009]: N/A (0x4496c)
[2021-02-18 00:10:42.068 23110/F650] B[0010]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb) [0x7fafc4ad109b]
[2021-02-18 00:10:42.068 23110/F650] B[0011]: /usr/bin/pihole-FTL(_start+0x2a) [0x5559cca9bb3a]
[2021-02-18 00:10:42.068 23110/F650] L[0011]: N/A (0x44b3a)
[2021-02-18 00:10:42.068 23110/F650] ------ Listing content of directory /dev/shm ------
[2021-02-18 00:10:42.069 23110/F650] File Mode User:Group  Filesize Filename
[2021-02-18 00:10:42.069 23110/F650] rwxrwxrwx root:root 260 .
[2021-02-18 00:10:42.069 23110/F650] rwxr-xr-x root:root 3K ..
[2021-02-18 00:10:42.069 23110/F650] rw------- pihole:pihole 4K FTL-per-client-regex
[2021-02-18 00:10:42.069 23110/F650] rw------- pihole:pihole 205K FTL-dns-cache
[2021-02-18 00:10:42.069 23110/F650] rw------- pihole:pihole 25K FTL-overTime
[2021-02-18 00:10:42.069 23110/F650] rw------- pihole:pihole 524K FTL-queries
[2021-02-18 00:10:42.069 23110/F650] rw------- pihole:pihole 20K FTL-upstreams
[2021-02-18 00:10:42.069 23110/F650] rw------- pihole:pihole 176K FTL-clients
[2021-02-18 00:10:42.069 23110/F650] rw------- pihole:pihole 98K FTL-domains
[2021-02-18 00:10:42.069 23110/F650] rw------- pihole:pihole 82K FTL-strings
[2021-02-18 00:10:42.069 23110/F650] rw------- pihole:pihole 12 FTL-settings
[2021-02-18 00:10:42.069 23110/F650] rw------- pihole:pihole 152 FTL-counters
[2021-02-18 00:10:42.069 23110/F650] rw------- pihole:pihole 48 FTL-lock
[2021-02-18 00:10:42.069 23110/F650] ---------------------------------------------------
[2021-02-18 00:10:42.069 23110/F650] Please also include some lines from above the !!!!!!!!! header.
[2021-02-18 00:10:42.069 23110/F650] Thank you for helping us to improve our FTL engine!
[2021-02-18 00:10:42.069 23110/F650] Asking parent pihole-FTL (PID 650) to shut down
[2021-02-18 00:10:42.069 23110/F650] FTL fork terminated!
[2021-02-18 00:10:42.069 650M] Received: Real-time signal 2 (36 -> 2)
[2021-02-18 00:10:42.069 650M] Shutting down...
[2021-02-18 00:10:42.087 650M] Finished final database update
[2021-02-18 00:10:42.089 650M] ########## FTL terminated after 6d 40m 38s ! ##########
@ndupont-net
Copy link

Hi, I have a similar feedback with the same version.
Raspbian buster on a Pi Zero W. Full-upgrade yesterday evening.

The crashed instance was not even under load as my router was directing the requests to a second raspberry pi since 8am.

[2021-02-19 00:59:00.207 466/T470] Notice: Database size is 5.53 MB, deleted 257 rows
[2021-02-19 01:59:00.308 466/T470] Notice: Database size is 5.53 MB, deleted 294 rows
[2021-02-19 02:59:00.186 466/T470] Notice: Database size is 5.53 MB, deleted 251 rows
[2021-02-19 03:59:00.157 466/T470] Notice: Database size is 5.54 MB, deleted 256 rows
[2021-02-19 04:59:00.191 466/T470] Notice: Database size is 5.55 MB, deleted 240 rows
[2021-02-19 05:37:18.752 466M] Resizing "FTL-queries" from 917504 to (20480 * 56) == 1146880 (/dev/shm: 1.5MB used, 250.5MB total)
[2021-02-19 05:59:00.185 466/T470] Notice: Database size is 5.57 MB, deleted 395 rows
[2021-02-19 06:05:54.726 466M] Resizing "FTL-strings" from 73728 to (77824 * 1) == 77824 (/dev/shm: 1.8MB used, 250.5MB total)
[2021-02-19 06:22:47.772 466M] Resizing "FTL-dns-cache" from 36864 to (2560 * 16) == 40960 (/dev/shm: 1.8MB used, 250.5MB total)
[2021-02-19 06:32:11.755 466M] Resizing "FTL-strings" from 77824 to (81920 * 1) == 81920 (/dev/shm: 1.8MB used, 250.5MB total)
[2021-02-19 06:34:18.500 466M] Resizing "FTL-strings" from 81920 to (86016 * 1) == 86016 (/dev/shm: 1.8MB used, 250.5MB total)
[2021-02-19 06:34:21.186 466M] Resizing "FTL-strings" from 86016 to (90112 * 1) == 90112 (/dev/shm: 1.8MB used, 250.5MB total)
[2021-02-19 06:34:23.091 466M] Resizing "FTL-dns-cache" from 40960 to (2816 * 16) == 45056 (/dev/shm: 1.8MB used, 250.5MB total)
[2021-02-19 06:34:28.360 466M] Resizing "FTL-strings" from 90112 to (94208 * 1) == 94208 (/dev/shm: 1.8MB used, 250.5MB total)
[2021-02-19 07:00:00.128 466/T470] Notice: Database size is 5.62 MB, deleted 533 rows
[2021-02-19 08:00:00.084 466/T470] Notice: Database size is 5.62 MB, deleted 814 rows
[2021-02-19 08:59:00.109 466/T470] Notice: Database size is 5.62 MB, deleted 517 rows
[2021-02-19 09:59:00.141 466/T470] Notice: Database size is 5.62 MB, deleted 963 rows
[2021-02-19 10:59:00.256 466/T470] Notice: Database size is 5.62 MB, deleted 1040 rows
[2021-02-19 11:59:00.135 466/T470] Notice: Database size is 5.62 MB, deleted 1207 rows
[2021-02-19 12:27:00.026 466/T470] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2021-02-19 12:27:00.027 466/T470] ---------------------------->  FTL crashed!  <----------------------------
[2021-02-19 12:27:00.027 466/T470] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2021-02-19 12:27:00.027 466/T470] Please report a bug at https://github.com/pi-hole/FTL/issues
[2021-02-19 12:27:00.028 466/T470] and include in your report already the following details:
[2021-02-19 12:27:00.028 466/T470] FTL has been running for 68015 seconds
[2021-02-19 12:27:00.028 466/T470] FTL branch: master
[2021-02-19 12:27:00.029 466/T470] FTL version: v5.6
[2021-02-19 12:27:00.029 466/T470] FTL commit: da89cc6
[2021-02-19 12:27:00.029 466/T470] FTL date: 2021-01-27 17:55:33 +0000
[2021-02-19 12:27:00.030 466/T470] FTL user: started as pihole, ended as pihole
[2021-02-19 12:27:00.030 466/T470] Compiled for armv6hf (compiled on CI) using arm-linux-gnueabihf-gcc (crosstool-NG crosstool-ng-1.22.0-88-g8460611) 4.9.3
[2021-02-19 12:27:00.031 466/T470] Process details: MID: 466
[2021-02-19 12:27:00.031 466/T470]                  PID: 466
[2021-02-19 12:27:00.031 466/T470]                  TID: 470
[2021-02-19 12:27:00.032 466/T470]                  Name: database
[2021-02-19 12:27:00.032 466/T470] Received signal: Segmentation fault
[2021-02-19 12:27:00.033 466/T470]      at address: 0xe2c33000
[2021-02-19 12:27:00.033 466/T470]      with code:  SEGV_MAPERR (Address not mapped to object)
[2021-02-19 12:27:00.034 466/T470] Backtrace:
[2021-02-19 12:27:00.038 466/T470] B[0000]: /usr/bin/pihole-FTL(+0x4f8c8) [0x4cf8c8]
[2021-02-19 12:27:00.453 466/T470] L[0000]: /root/project/src/signals.c:192
[2021-02-19 12:27:00.462 466/T470] B[0001]: /lib/arm-linux-gnueabihf/libc.so.6(__default_rt_sa_restorer+0) [0xb6da4130]
[2021-02-19 12:27:00.462 466/T470] B[0002]: /usr/bin/pihole-FTL(+0x33e40) [0x4b3e40]
[2021-02-19 12:27:00.974 466/T470] L[0002]: /root/project/src/database/sqlite3.c:120471
[2021-02-19 12:27:01.024 466/T470] B[0003]: /usr/bin/pihole-FTL(+0x19ef84) [0x61ef84]
[2021-02-19 12:27:01.538 466/T470] L[0003]: /root/project/src/database/sqlite3.c:122301
[2021-02-19 12:27:01.582 466/T470] B[0004]: /usr/bin/pihole-FTL(+0x181d14) [0x601d14]
[2021-02-19 12:27:02.080 466/T470] L[0004]: /root/project/src/database/sqlite3.c:158985
[2021-02-19 12:27:02.125 466/T470] B[0005]: /usr/bin/pihole-FTL(+0x1861c4) [0x6061c4]
[2021-02-19 12:27:02.622 466/T470] L[0005]: /root/project/src/database/sqlite3.c:129510
[2021-02-19 12:27:02.668 466/T470] B[0006]: /usr/bin/pihole-FTL(+0x1865cc) [0x6065cc]
[2021-02-19 12:27:03.168 466/T470] L[0006]: /root/project/src/database/sqlite3.c:129585
[2021-02-19 12:27:03.213 466/T470] B[0007]: /usr/bin/pihole-FTL(sqlite3_prepare_v2+0x24) [0x6067b4]
[2021-02-19 12:27:03.715 466/T470] L[0007]: /root/project/src/database/sqlite3.c:129671
[2021-02-19 12:27:03.759 466/T470] B[0008]: /usr/bin/pihole-FTL(DB_save_queries+0xd8) [0x4e12e4]
[2021-02-19 12:27:03.833 466/T470] L[0008]: /root/project/src/database/query-table.c:72
[2021-02-19 12:27:03.840 466/T470] B[0009]: /usr/bin/pihole-FTL(DB_thread+0x264) [0x4d8f44]
[2021-02-19 12:27:03.913 466/T470] L[0009]: /root/project/src/database/database-thread.c:54
[2021-02-19 12:27:03.920 466/T470] ------ Listing content of directory /dev/shm ------
[2021-02-19 12:27:03.921 466/T470] File Mode User:Group  Filesize Filename
[2021-02-19 12:27:03.921 466/T470] rwxrwxrwx root:root 260 .
[2021-02-19 12:27:03.922 466/T470] rwxr-xr-x root:root 3K ..
[2021-02-19 12:27:03.922 466/T470] rw------- pihole:pihole 4K FTL-per-client-regex
[2021-02-19 12:27:03.923 466/T470] rw------- pihole:pihole 45K FTL-dns-cache
[2021-02-19 12:27:03.923 466/T470] rw------- pihole:pihole 45K FTL-overTime
[2021-02-19 12:27:03.924 466/T470] rw------- pihole:pihole 1M FTL-queries
[2021-02-19 12:27:03.925 466/T470] rw------- pihole:pihole 29K FTL-upstreams
[2021-02-19 12:27:03.926 466/T470] rw------- pihole:pihole 340K FTL-clients
[2021-02-19 12:27:03.926 466/T470] rw------- pihole:pihole 66K FTL-domains
[2021-02-19 12:27:03.927 466/T470] rw------- pihole:pihole 94K FTL-strings
[2021-02-19 12:27:03.928 466/T470] rw------- pihole:pihole 12 FTL-settings
[2021-02-19 12:27:03.928 466/T470] rw------- pihole:pihole 152 FTL-counters
[2021-02-19 12:27:03.929 466/T470] rw------- pihole:pihole 28 FTL-lock
[2021-02-19 12:27:03.929 466/T470] ---------------------------------------------------
[2021-02-19 12:27:03.929 466/T470] Please also include some lines from above the !!!!!!!!! header.
[2021-02-19 12:27:03.930 466/T470] Thank you for helping us to improve our FTL engine!
[2021-02-19 12:27:03.930 466/T470] FTL terminated!

I see this warning appearing dozens of times after restart :

[2021-02-19 14:22:39.181 480M] Warning: Found unknown status 14 in long term database!
[2021-02-19 14:22:39.202 480M] Warning: Found unknown status 14 in long term database!
[2021-02-19 14:22:39.203 480M] Warning: Found unknown status 14 in long term database!

@DL6ER
Copy link
Member

DL6ER commented Feb 20, 2021

You two are describing different crashes. Are they in any way reproducible? How often does this happen?

Concerning the "unknown status 14": This is entirely harmless and fixed in the current version (FTL v5.7) by #1048

@DL6ER DL6ER added the Bug label Feb 20, 2021
@libertyx82
Copy link
Author

Are they in any way reproducible? How often does this happen?

No i can not reproduce it, it was the first time i have noticed this crash and it does not appear again till today

@nebakanezzer
Copy link

nebakanezzer commented Feb 21, 2021

also had a random crash this morning, running pihole in hyperv VM, same debian version. this is tough to reproduce, so I have not posted about it before, but it seems to happen maybe once every month or so, for no apparent reason. the only things in the log from the same time period of it happening is reloading dns cache.

[2021-02-21 00:59:01.409 52604/T52609] WARN: moveOverTimeMemory(): overTime time index correction failed (188: 5 / 6)
[2021-02-21 00:59:01.879 52604/T52608] Notice: Database size is 2186.40 MB, deleted 4680 rows
[2021-02-21 01:59:01.694 52604/T52608] Notice: Database size is 2186.40 MB, deleted 5272 rows
[2021-02-21 02:13:23.387 52604M] Resizing "FTL-dns-cache" from 958464 to (60160 * 16) == 962560 (/dev/shm: 7.8MB used, 219.7MB total)
[2021-02-21 02:59:01.672 52604/T52608] Notice: Database size is 2186.40 MB, deleted 4771 rows
[2021-02-21 03:59:01.533 52604/T52608] Notice: Database size is 2186.40 MB, deleted 5600 rows
[2021-02-21 04:46:43.477 52604M] Reloading DNS cache
[2021-02-21 04:46:43.526 52604M] Blocking status is enabled
[2021-02-21 04:46:43.537 52604M] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2021-02-21 04:46:43.537 52604M] ----------------------------> FTL crashed! <----------------------------
[2021-02-21 04:46:43.537 52604M] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2021-02-21 04:46:43.537 52604M] Please report a bug at https://github.com/pi-hole/FTL/issues
[2021-02-21 04:46:43.537 52604M] and include in your report already the following details:
[2021-02-21 04:46:43.537 52604M] FTL has been running for 2395462 seconds
[2021-02-21 04:46:43.537 52604M] FTL branch: master
[2021-02-21 04:46:43.537 52604M] FTL version: v5.3.2
[2021-02-21 04:46:43.537 52604M] FTL commit: 0790cf7
[2021-02-21 04:46:43.537 52604M] FTL date: 2020-12-02 21:28:39 +0000
[2021-02-21 04:46:43.538 52604M] FTL user: started as pihole, ended as pihole
[2021-02-21 04:46:43.538 52604M] Compiled for x86_64 (compiled on CI) using gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
[2021-02-21 04:46:43.538 52604M] Process details: MID: 52604
[2021-02-21 04:46:43.538 52604M] PID: 52604
[2021-02-21 04:46:43.538 52604M] TID: 52604
[2021-02-21 04:46:43.538 52604M] Name: pihole-FTL
[2021-02-21 04:46:43.538 52604M] Received signal: Segmentation fault
[2021-02-21 04:46:43.538 52604M] at address: 0x7
[2021-02-21 04:46:43.538 52604M] with code: SEGV_MAPERR (Address not mapped to object)
[2021-02-21 04:46:43.569 52604M] Backtrace:
[2021-02-21 04:46:43.569 52604M] B[0000]: /usr/bin/pihole-FTL(+0x534e5) [0x5646ccb7d4e5]
[2021-02-21 04:46:43.649 52604M] L[0000]: N/A (0x534e5)
[2021-02-21 04:46:43.649 52604M] B[0001]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x110c0) [0x7f6d636e00c0]
[2021-02-21 04:46:43.649 52604M] B[0002]: /lib/x86_64-linux-gnu/libc.so.6(+0x83f66) [0x7f6d633b3f66]
[2021-02-21 04:46:43.649 52604M] B[0003]: /lib/x86_64-linux-gnu/libc.so.6(getdelim+0x1b8) [0x7f6d633982f8]
[2021-02-21 04:46:43.649 52604M] B[0004]: /usr/bin/pihole-FTL(+0x41698) [0x5646ccb6b698]
[2021-02-21 04:46:43.651 52604M] L[0004]: N/A (0x41698)
[2021-02-21 04:46:43.651 52604M] B[0005]: /usr/bin/pihole-FTL(get_blocking_mode+0x32) [0x5646ccb6bb32]
[2021-02-21 04:46:43.652 52604M] L[0005]: N/A (0x41b32)
[2021-02-21 04:46:43.652 52604M] B[0006]: /usr/bin/pihole-FTL(FTL_dnsmasq_reload+0x4a) [0x5646ccb71f5a]
[2021-02-21 04:46:43.653 52604M] L[0006]: N/A (0x47f5a)
[2021-02-21 04:46:43.653 52604M] B[0007]: /usr/bin/pihole-FTL(clear_cache_and_reload+0x22) [0x5646ccba0ab2]
[2021-02-21 04:46:43.654 52604M] L[0007]: N/A (0x76ab2)
[2021-02-21 04:46:43.654 52604M] B[0008]: /usr/bin/pihole-FTL(main_dnsmasq+0x1c37) [0x5646ccba2ac7]
[2021-02-21 04:46:43.656 52604M] L[0008]: N/A (0x78ac7)
[2021-02-21 04:46:43.656 52604M] B[0009]: /usr/bin/pihole-FTL(main+0xf1) [0x5646ccb6a301]
[2021-02-21 04:46:43.657 52604M] L[0009]: N/A (0x40301)
[2021-02-21 04:46:43.658 52604M] B[0010]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7f6d633502b1]
[2021-02-21 04:46:43.658 52604M] B[0011]: /usr/bin/pihole-FTL(_start+0x2a) [0x5646ccb6a4da]
[2021-02-21 04:46:43.660 52604M] L[0011]: N/A (0x404da)
[2021-02-21 04:46:43.660 52604M] ------ Listing content of directory /dev/shm ------
[2021-02-21 04:46:43.660 52604M] File Mode User:Group Filesize Filename
[2021-02-21 04:46:43.660 52604M] rwxrwxrwx root:root 260 .
[2021-02-21 04:46:43.660 52604M] rwxr-xr-x root:root 3K ..
[2021-02-21 04:46:43.660 52604M] rw------- pihole:pihole 4K FTL-per-client-regex
[2021-02-21 04:46:43.660 52604M] rw------- pihole:pihole 963K FTL-dns-cache
[2021-02-21 04:46:43.660 52604M] rw------- pihole:pihole 45K FTL-overTime
[2021-02-21 04:46:43.660 52604M] rw------- pihole:pihole 6M FTL-queries
[2021-02-21 04:46:43.660 52604M] rw------- pihole:pihole 20K FTL-upstreams
[2021-02-21 04:46:43.660 52604M] rw------- pihole:pihole 176K FTL-clients
[2021-02-21 04:46:43.660 52604M] rw------- pihole:pihole 393K FTL-domains
[2021-02-21 04:46:43.660 52604M] rw------- pihole:pihole 377K FTL-strings
[2021-02-21 04:46:43.660 52604M] rw------- pihole:pihole 12 FTL-settings
[2021-02-21 04:46:43.661 52604M] rw------- pihole:pihole 152 FTL-counters
[2021-02-21 04:46:43.661 52604M] rw------- pihole:pihole 48 FTL-lock
[2021-02-21 04:46:43.661 52604M] ---------------------------------------------------
[2021-02-21 04:46:43.661 52604M] Please also include some lines from above the !!!!!!!!! header.
[2021-02-21 04:46:43.661 52604M] Thank you for helping us to improve our FTL engine!
[2021-02-21 04:46:43.661 52604M] FTL terminated!
[2021-02-21 09:26:29.889 33042M] Using log file /var/log/pihole-FTL.log
[2021-02-21 09:26:29.897 33042M] ########## FTL started! ##########

@DL6ER
Copy link
Member

DL6ER commented Feb 21, 2021

@nebakanezzer v5.3.2 is a bit outdated and I seem to recall that we fixed this exact issue (crash in getdelim) sometime in December 2020. In fact, I see now that this was the exact reason for the v5.3.3 release :) (#967)

@nebakanezzer
Copy link

@nebakanezzer v5.3.2 is a bit outdated and I seem to recall that we fixed this exact issue (crash in getdelim) sometime in December 2020. In fact, I see now that this was the exact reason for the v5.3.3 release :) (#967)

sweet. thank you for that. I will update and patiently await next month to see if this pops up again haha

@tom-daley
Copy link

I installed pihole on Friday and have been getting an FTL crash once a day.

[2021-03-03 00:49:37.671 17843M] Resizing "FTL-dns-cache" from 28672 to (2048 * 16) == 32768 (/dev/shm: 4.8MB used, 93.5MB total, FTL uses 4.8MB)
[2021-03-03 07:36:33.768 17843M] Resizing "FTL-dns-cache" from 32768 to (2304 * 16) == 36864 (/dev/shm: 4.8MB used, 93.5MB total, FTL uses 4.8MB)
[2021-03-03 08:15:05.265 17843M] Resizing "FTL-strings" from 53248 to (57344 * 1) == 57344 (/dev/shm: 4.8MB used, 93.5MB total, FTL uses 4.8MB)
[2021-03-03 13:18:00.032 17843/T17847] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2021-03-03 13:18:00.040 17843/T17847] ----------------------------> FTL crashed! <----------------------------
[2021-03-03 13:18:00.040 17843/T17847] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2021-03-03 13:18:00.040 17843/T17847] Please report a bug at https://github.com/pi-hole/FTL/issues
[2021-03-03 13:18:00.041 17843/T17847] and include in your report already the following details:
[2021-03-03 13:18:00.041 17843/T17847] FTL has been running for 87900 seconds
[2021-03-03 13:18:00.042 17843/T17847] FTL branch: master
[2021-03-03 13:18:00.042 17843/T17847] FTL version: v5.7
[2021-03-03 13:18:00.042 17843/T17847] FTL commit: 2999e2b
[2021-03-03 13:18:00.042 17843/T17847] FTL date: 2021-02-16 19:36:43 +0000
[2021-03-03 13:18:00.043 17843/T17847] FTL user: started as pihole, ended as pihole
[2021-03-03 13:18:00.044 17843/T17847] Compiled for armv6hf (compiled on CI) using arm-linux-gnueabihf-gcc (crosstool-NG crosstool-ng-1.22.0-88-g8460611) 4.9.3
[2021-03-03 13:18:00.044 17843/T17847] Process details: MID: 17843
[2021-03-03 13:18:00.044 17843/T17847] PID: 17843
[2021-03-03 13:18:00.045 17843/T17847] TID: 17847
[2021-03-03 13:18:00.045 17843/T17847] Name: database
[2021-03-03 13:18:00.045 17843/T17847] Received signal: Segmentation fault
[2021-03-03 13:18:00.046 17843/T17847] at address: 0x34
[2021-03-03 13:18:00.046 17843/T17847] with code: SEGV_MAPERR (Address not mapped to object)
[2021-03-03 13:18:00.060 17843/T17847] Backtrace:
[2021-03-03 13:18:00.065 17843/T17847] B[0000]: /usr/bin/pihole-FTL(+0x4ff40) [0x495f40]
[2021-03-03 13:18:00.631 17843/T17847] L[0000]: /root/project/src/signals.c:192
[2021-03-03 13:18:00.643 17843/T17847] B[0001]: /lib/arm-linux-gnueabihf/libc.so.6(__default_rt_sa_restorer+0) [0xb6d3e130]
[2021-03-03 13:18:00.643 17843/T17847] B[0002]: /usr/bin/pihole-FTL(+0xf610c) [0x53c10c]
[2021-03-03 13:18:01.554 17843/T17847] L[0002]: /root/project/src/database/sqlite3.c:50586
[2021-03-03 13:18:01.961 17843/T17847] B[0003]: /usr/bin/pihole-FTL(+0x14a13c) [0x59013c]
[2021-03-03 13:18:02.873 17843/T17847] L[0003]: /root/project/src/database/sqlite3.c:57398
[2021-03-03 13:18:02.949 17843/T17847] B[0004]: /usr/bin/pihole-FTL(+0x110f6c) [0x556f6c]
[2021-03-03 13:18:03.833 17843/T17847] L[0004]: /root/project/src/database/sqlite3.c:66847
[2021-03-03 13:18:03.905 17843/T17847] B[0005]: /usr/bin/pihole-FTL(+0x14d6ac) [0x5936ac]
[2021-03-03 13:18:04.761 17843/T17847] L[0005]: /root/project/src/database/sqlite3.c:70937
[2021-03-03 13:18:04.832 17843/T17847] B[0006]: /usr/bin/pihole-FTL(+0x1536c8) [0x5996c8]
[2021-03-03 13:18:05.679 17843/T17847] L[0006]: /root/project/src/database/sqlite3.c:72034
[2021-03-03 13:18:05.750 17843/T17847] B[0007]: /usr/bin/pihole-FTL(+0x15438c) [0x59a38c]
[2021-03-03 13:18:06.598 17843/T17847] L[0007]: /root/project/src/database/sqlite3.c:73613
[2021-03-03 13:18:06.672 17843/T17847] B[0008]: /usr/bin/pihole-FTL(+0x18b424) [0x5d1424]
[2021-03-03 13:18:07.547 17843/T17847] L[0008]: /root/project/src/database/sqlite3.c:90708
[2021-03-03 13:18:07.617 17843/T17847] B[0009]: /usr/bin/pihole-FTL(+0x194be8) [0x5dabe8]
[2021-03-03 13:18:08.480 17843/T17847] L[0009]: /root/project/src/database/sqlite3.c:83942
[2021-03-03 13:18:08.550 17843/T17847] B[0010]: /usr/bin/pihole-FTL(DB_save_queries+0x3cc) [0x4a7c50]
[2021-03-03 13:18:08.670 17843/T17847] L[0010]: /root/project/src/database/query-table.c:195
[2021-03-03 13:18:08.682 17843/T17847] B[0011]: /usr/bin/pihole-FTL(DB_thread+0x264) [0x49f5bc]
[2021-03-03 13:18:08.798 17843/T17847] L[0011]: /root/project/src/database/database-thread.c:54
[2021-03-03 13:18:08.810 17843/T17847] ------ Listing content of directory /dev/shm ------
[2021-03-03 13:18:08.810 17843/T17847] File Mode User:Group Filesize Filename
[2021-03-03 13:18:08.821 17843/T17847] rwxrwxrwx root:root 260 .
[2021-03-03 13:18:08.822 17843/T17847] rwxr-xr-x root:root 4K ..
[2021-03-03 13:18:08.823 17843/T17847] rw------- pihole:pihole 4K FTL-per-client-regex
[2021-03-03 13:18:08.824 17843/T17847] rw------- pihole:pihole 37K FTL-dns-cache
[2021-03-03 13:18:08.825 17843/T17847] rw------- pihole:pihole 45K FTL-overTime
[2021-03-03 13:18:08.826 17843/T17847] rw------- pihole:pihole 4M FTL-queries
[2021-03-03 13:18:08.827 17843/T17847] rw------- pihole:pihole 29K FTL-upstreams
[2021-03-03 13:18:08.828 17843/T17847] rw------- pihole:pihole 684K FTL-clients
[2021-03-03 13:18:08.829 17843/T17847] rw------- pihole:pihole 66K FTL-domains
[2021-03-03 13:18:08.830 17843/T17847] rw------- pihole:pihole 57K FTL-strings
[2021-03-03 13:18:08.831 17843/T17847] rw------- pihole:pihole 12 FTL-settings
[2021-03-03 13:18:08.832 17843/T17847] rw------- pihole:pihole 152 FTL-counters
[2021-03-03 13:18:08.833 17843/T17847] rw------- pihole:pihole 28 FTL-lock
[2021-03-03 13:18:08.833 17843/T17847] ---------------------------------------------------
[2021-03-03 13:18:08.834 17843/T17847] Please also include some lines from above the !!!!!!!!! header.
[2021-03-03 13:18:08.834 17843/T17847] Thank you for helping us to improve our FTL engine!
[2021-03-03 13:18:08.834 17843/T17847] FTL terminated!

Please let me know if I can provide any more info. Thanks

@DL6ER
Copy link
Member

DL6ER commented Mar 3, 2021

Can you try if

pihole checkout ftl development

improves the situation?

@tom-daley
Copy link

I have updated to the development version and will report back.
Thanks for the quick response!

@tom-daley
Copy link

With the development code I have new crashes:

[2021-03-03 17:47:05.303 30901M] Resizing "FTL-dns-cache" from 12288 to (1024 * 16) == 16384 (/dev/shm: 2.0MB used, 93.5MB total, FTL uses 2.0MB)
[2021-03-03 18:12:13.684 30901M] Resizing "FTL-strings" from 36864 to (40960 * 1) == 40960 (/dev/shm: 2.0MB used, 93.5MB total, FTL uses 2.0MB)
[2021-03-03 19:20:41.533 30901/T32390] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2021-03-03 19:20:41.535 30901/T32390] ----------------------------> FTL crashed! <----------------------------
[2021-03-03 19:20:41.535 30901/T32390] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2021-03-03 19:20:41.535 30901/T32390] Please report a bug at https://github.com/pi-hole/FTL/issues
[2021-03-03 19:20:41.536 30901/T32390] and include in your report already the following details:
[2021-03-03 19:20:41.536 30901/T32390] FTL has been running for 10567 seconds
[2021-03-03 19:20:41.536 30901/T32390] FTL branch: development
[2021-03-03 19:20:41.537 30901/T32390] FTL version: vDev-2485ac9
[2021-03-03 19:20:41.537 30901/T32390] FTL commit: 2485ac9
[2021-03-03 19:20:41.537 30901/T32390] FTL date: 2021-03-02 06:00:10 +0100
[2021-03-03 19:20:41.538 30901/T32390] FTL user: started as pihole, ended as pihole
[2021-03-03 19:20:41.538 30901/T32390] Compiled for armv6hf (compiled on CI) using arm-linux-gnueabihf-gcc (crosstool-NG crosstool-ng-1.22.0-88-g8460611) 4.9.3
[2021-03-03 19:20:41.539 30901/T32390] Process details: MID: 30901
[2021-03-03 19:20:41.539 30901/T32390] PID: 30901
[2021-03-03 19:20:41.539 30901/T32390] TID: 32390
[2021-03-03 19:20:41.540 30901/T32390] Name: telnet-IPv4
[2021-03-03 19:20:41.540 30901/T32390] Received signal: Illegal instruction
[2021-03-03 19:20:41.541 30901/T32390] at address: 0x45ca08
[2021-03-03 19:20:41.541 30901/T32390] with code: ILL_ILLOPC (Illegal opcode)
[2021-03-03 19:20:41.543 30901/T32390] Backtrace:
[2021-03-03 19:20:41.552 30901/T32390] B[0000]: /usr/bin/pihole-FTL(+0x4fe40) [0x456e40]
[2021-03-03 19:20:44.112 30901/T32390] L[0000]: /root/project/src/signals.c:192
[2021-03-03 19:20:44.125 30901/T32390] B[0001]: /lib/arm-linux-gnueabihf/libc.so.6(__default_rt_sa_restorer+0) [0xb6d07130]
[2021-03-03 19:20:44.126 30901/T32390] B[0002]: /usr/bin/pihole-FTL(+0x55a08) [0x45ca08]
[2021-03-03 19:20:44.276 30901/T32390] L[0002]: /root/project/src/api/socket.c:305
[2021-03-03 19:20:44.303 30901/T32390] ------ Listing content of directory /dev/shm ------
[2021-03-03 19:20:44.303 30901/T32390] File Mode User:Group Size Filename
[2021-03-03 19:20:44.305 30901/T32390] rwxrwxrwx root:root 260 .
[2021-03-03 19:20:44.305 30901/T32390] rwxr-xr-x root:root 4K ..
[2021-03-03 19:20:44.307 30901/T32390] rw------- pihole:pihole 4K FTL-per-client-regex
[2021-03-03 19:20:44.308 30901/T32390] rw------- pihole:pihole 16K FTL-dns-cache
[2021-03-03 19:20:44.309 30901/T32390] rw------- pihole:pihole 45K FTL-overTime
[2021-03-03 19:20:44.309 30901/T32390] rw------- pihole:pihole 1M FTL-queries
[2021-03-03 19:20:44.321 30901/T32390] rw------- pihole:pihole 29K FTL-upstreams
[2021-03-03 19:20:44.323 30901/T32390] rw------- pihole:pihole 684K FTL-clients
[2021-03-03 19:20:44.324 30901/T32390] rw------- pihole:pihole 66K FTL-domains
[2021-03-03 19:20:44.325 30901/T32390] rw------- pihole:pihole 41K FTL-strings
[2021-03-03 19:20:44.331 30901/T32390] rw------- pihole:pihole 12 FTL-settings
[2021-03-03 19:20:44.333 30901/T32390] rw------- pihole:pihole 156 FTL-counters
[2021-03-03 19:20:44.334 30901/T32390] rw------- pihole:pihole 56 FTL-lock
[2021-03-03 19:20:44.334 30901/T32390] ---------------------------------------------------
[2021-03-03 19:20:44.334 30901/T32390] Please also include some lines from above the !!!!!!!!! header.
[2021-03-03 19:20:44.335 30901/T32390] Thank you for helping us to improve our FTL engine!

Then tried "systemctl restart pihole-FTL" and it crashed again:

[2021-03-03 19:20:44.359 30901/T32390] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2021-03-03 19:20:44.360 30901/T32390] ----------------------------> FTL crashed! <----------------------------
[2021-03-03 19:20:44.360 30901/T32390] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2021-03-03 19:20:44.369 30901/T32390] Please report a bug at https://github.com/pi-hole/FTL/issues
[2021-03-03 19:20:44.369 30901/T32390] and include in your report already the following details:
[2021-03-03 19:20:44.369 30901/T32390] FTL has been running for 10570 seconds
[2021-03-03 19:20:44.372 30901/T32390] FTL branch: development
[2021-03-03 19:20:44.373 30901/T32390] FTL version: vDev-2485ac9
[2021-03-03 19:20:44.373 30901/T32390] FTL commit: 2485ac9
[2021-03-03 19:20:44.373 30901/T32390] FTL date: 2021-03-02 06:00:10 +0100
[2021-03-03 19:20:44.374 30901/T32390] FTL user: started as pihole, ended as pihole
[2021-03-03 19:20:44.375 30901/T32390] Compiled for armv6hf (compiled on CI) using arm-linux-gnueabihf-gcc (crosstool-NG crosstool-ng-1.22.0-88-g8460611) 4.9.3
[2021-03-03 19:20:44.375 30901/T32390] Process details: MID: 30901
[2021-03-03 19:20:44.375 30901/T32390] PID: 30901
[2021-03-03 19:20:44.376 30901/T32390] TID: 32390
[2021-03-03 19:20:44.376 30901/T32390] Name: telnet-IPv4
[2021-03-03 19:20:44.376 30901/T32390] Received signal: Segmentation fault
[2021-03-03 19:20:44.377 30901/T32390] at address: 0xfd000000
[2021-03-03 19:20:44.377 30901/T32390] with code: SEGV_MAPERR (Address not mapped to object)
[2021-03-03 19:20:44.377 30901/T32390] Backtrace:
[2021-03-03 19:20:44.381 30901/T32390] B[0000]: /usr/bin/pihole-FTL(+0x4fe40) [0x456e40]
[2021-03-03 19:20:44.496 30901/T32390] L[0000]: /root/project/src/signals.c:192
[2021-03-03 19:20:44.508 30901/T32390] B[0001]: /lib/arm-linux-gnueabihf/libc.so.6(__default_rt_sa_restorer+0) [0xb6d07130]
[2021-03-03 19:20:44.508 30901/T32390] B[0002]: /usr/bin/pihole-FTL(+0x55fc4) [0x45cfc4]
[2021-03-03 19:20:44.626 30901/T32390] L[0002]: /root/project/src/api/socket.c:286
[2021-03-03 19:20:44.638 30901/T32390] B[0003]: /usr/bin/pihole-FTL(+0x50204) [0x457204]
[2021-03-03 19:20:44.754 30901/T32390] L[0003]: /root/project/src/signals.c:240
[2021-03-03 19:20:44.765 30901/T32390] B[0004]: /lib/arm-linux-gnueabihf/libc.so.6(__default_rt_sa_restorer+0) [0xb6d07130]
[2021-03-03 19:20:44.766 30901/T32390] B[0005]: /usr/bin/pihole-FTL(+0x55a08) [0x45ca08]
[2021-03-03 19:20:44.883 30901/T32390] L[0005]: /root/project/src/api/socket.c:305
[2021-03-03 19:20:44.894 30901/T32390] ------ Listing content of directory /dev/shm ------
[2021-03-03 19:20:44.895 30901/T32390] File Mode User:Group Size Filename
[2021-03-03 19:20:44.896 30901/T32390] rwxrwxrwx root:root 260 .
[2021-03-03 19:20:44.897 30901/T32390] rwxr-xr-x root:root 4K ..
[2021-03-03 19:20:44.898 30901/T32390] rw------- pihole:pihole 4K FTL-per-client-regex
[2021-03-03 19:20:44.899 30901/T32390] rw------- pihole:pihole 16K FTL-dns-cache
[2021-03-03 19:20:44.900 30901/T32390] rw------- pihole:pihole 45K FTL-overTime
[2021-03-03 19:20:44.901 30901/T32390] rw------- pihole:pihole 1M FTL-queries
[2021-03-03 19:20:44.903 30901/T32390] rw------- pihole:pihole 29K FTL-upstreams
[2021-03-03 19:20:44.903 30901/T32390] rw------- pihole:pihole 684K FTL-clients
[2021-03-03 19:20:44.904 30901/T32390] rw------- pihole:pihole 66K FTL-domains
[2021-03-03 19:20:44.905 30901/T32390] rw------- pihole:pihole 41K FTL-strings
[2021-03-03 19:20:44.907 30901/T32390] rw------- pihole:pihole 12 FTL-settings
[2021-03-03 19:20:44.908 30901/T32390] rw------- pihole:pihole 156 FTL-counters
[2021-03-03 19:20:44.909 30901/T32390] rw------- pihole:pihole 56 FTL-lock
[2021-03-03 19:20:44.909 30901/T32390] ---------------------------------------------------
[2021-03-03 19:20:44.909 30901/T32390] Please also include some lines from above the !!!!!!!!! header.
[2021-03-03 19:20:44.910 30901/T32390] Thank you for helping us to improve our FTL engine!

@DL6ER
Copy link
Member

DL6ER commented Mar 3, 2021

ILL_ILLOPC (Illegal opcode)

Oh, interesting. This is a CPU issue. Which board are you using?

Then tried "systemctl restart pihole-FTL" and it crashed again

The two reports were from the same process, there was no restart in between.

@tom-daley
Copy link

Raspberry Pi Model B Rev 1

@peterd550

This comment has been minimized.

@DL6ER
Copy link
Member

DL6ER commented Mar 15, 2021

Sorry for the delay in replying. I checked the code around the crash you're seeing carefully and cannot find anything odd. In fact, the second crash even happened in code where a const static stack variable is used. Together with the fact that we haven't seen such a report before, I think we are concerned either with a SD card corruption or maybe a hardware (memory) fault of some kind. Reason for saying this is that, otherwise, all Pi-hole's would be affected in the same way and we'd have thousands of issue tickets concerning this.

@peterd550
Copy link

I found the issue - for me anyway

I have my own docker image - I have webhooks to block and unblock sites using 'pihole --regex'

As part of the build it does a git pull to the versions.

I noticed that there was updates to the install.sh, s6 folder and bash_functions.sh

When I updated my repo with your new updates it worked and I am up running

@tom-daley
Copy link

Since I had this problem, I have installed docker and am now running pi-hole in a container instead of on a pi.

Thanks for the feedback!

@DL6ER
Copy link
Member

DL6ER commented Mar 22, 2021

Is this still relevant? We have quite a few fixes in development by now.

@peterd550
Copy link

I think so
I went to clear my logs and pihole crashed
Looking on the docker-compose logs it was full of TfL issues
I stopped my container and restart it and that seemed to fix it

I haven't had a chance to see if I can replicate the issue

@DL6ER
Copy link
Member

DL6ER commented Mar 27, 2021

There is a new development version available that should fix bugs associated with high TCP query load (or bad timing). Does this work for you?

@DL6ER
Copy link
Member

DL6ER commented Apr 14, 2021

The next version of FTL has been released.

If you have any issues, please either reopen this ticket or (preferably) create a new ticket describing the issues in further detail and only reference this ticket. This will help us to help you best.

@DL6ER DL6ER closed this as completed Apr 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

6 participants