pihole-FTL segmentation fault when calling getallqueries-client via Telnet API

[davidsuart]

In raising this issue, I confirm the following

• I have read and understood the contributors guide.
• The issue I am reporting can be replicated
• The issue I am reporting isn't a duplicate

Well .. It looks similar to these other unresolved issues that have not yet been repeatable (As this one is):

• pi-hole/FTL/issues/257
• pi-hole/FTL/issues/323

How familiar are you with the codebase?:

2 on a scale of 1 to 4.

---

Expected Behaviour:

pihole-FTL doesn't crash

Actual Behaviour:

pihole-FTL crashes

Steps to reproduce:

With either an existing install or a clean install from master + FTLDNS

Navigate to http://<pi-hole-server>/admin/queries.php?client=<raw-ip-address>

When you hit this page for a client that has been PTR'd and resolved, no error. When you hit that page for a client with a fresh IP address that has not been PTR'd, pihole-FTL crashes.

So;

• Fresh install, invoke getAllQueries for an IP address (EG: click on link in "Top clients"). Crash.
• Wait several minutes until IP addresses have been looked up to hostnames, click on the same link (Literally, the same URL with the IP address in it) in top clients, invoke getAllQueries for the same IP address, data returned, no crash.

Calling pages that make other Telnet API queries such as "top domains" does not appear to invoke the same problem, so it appears that just this one command may have an issue.

Log file output # 1 (from /var/log/pihole-FTL.log)

[2018-07-25 12:09:59.932] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2018-07-25 12:09:59.932] ---------------------------->  FTL crashed!  <----------------------------
[2018-07-25 12:09:59.932] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2018-07-25 12:09:59.932] Please report a bug at https://github.com/pi-hole/FTL/issues
[2018-07-25 12:09:59.932] and include in your report already the following details:

[2018-07-25 12:09:59.932] FTL has been running for 51813 seconds
[2018-07-25 12:09:59.932] FTL branch: FTLDNS
[2018-07-25 12:09:59.932] FTL version: 
[2018-07-25 12:09:59.932] FTL commit: 5ecab0a
[2018-07-25 12:09:59.932] FTL date: 2018-05-18 10:24:07 +0100
[2018-07-25 12:09:59.932] FTL user: pihole
[2018-07-25 12:09:59.932] Received signal: Segmentation fault
[2018-07-25 12:09:59.932]      at address: 0
[2018-07-25 12:09:59.932]      with code: SEGV_MAPERR (Address not mapped to object)
[2018-07-25 12:09:59.945] Backtrace:
[2018-07-25 12:09:59.945] B[0000]: /usr/bin/pihole-FTL(+0x22963) [0x55c18f35a963]
[2018-07-25 12:09:59.945] B[0001]: /usr/glibc-compat/lib/libpthread.so.0(+0x11ef0) [0x7f44d3c9eef0]
[2018-07-25 12:09:59.945] B[0002]: /usr/glibc-compat/lib/libc.so.6(+0x12c2d6) [0x7f44d3a042d6]
[2018-07-25 12:09:59.945] B[0003]: /usr/bin/pihole-FTL(getAllQueries+0x538) [0x55c18f362108]
[2018-07-25 12:09:59.945] B[0004]: /usr/bin/pihole-FTL(process_request+0x255) [0x55c18f35c115]
[2018-07-25 12:09:59.945] B[0005]: /usr/bin/pihole-FTL(telnet_connection_handler_thread+0x10b) [0x55c18f35ac8b]
[2018-07-25 12:09:59.945] B[0006]: /usr/glibc-compat/lib/libpthread.so.0(+0x7587) [0x7f44d3c94587]
[2018-07-25 12:09:59.945] B[0007]: /usr/glibc-compat/lib/libc.so.6(clone+0x3f) [0x7f44d39c997f]
[2018-07-25 12:09:59.945] Memory usage (structs): 680728
[2018-07-25 12:09:59.945] Memory usage (dynamic): 19211

[2018-07-25 12:09:59.945] Thank you for helping us to improve our FTL engine!
[2018-07-25 12:09:59.945] FTL terminated!

Log file output # 2 (as per "the debugging instructions")

...
[LWP 23151 exited]
[New LWP 23152]
[LWP 23152 exited]
[New LWP 23159]

Thread 28 "telnet-15" received signal SIGSEGV, Segmentation fault.
[Switching to LWP 23159]
0x00007f9a747e52d6 in ?? () from /usr/glibc-compat/lib/libc.so.6

(gdb) backtrace
#0  0x00007f9a747e52d6 in ?? () from /usr/glibc-compat/lib/libc.so.6
#1  0x00005598ffce3108 in getAllQueries (client_message=client_message@entry=0x7f9a64000bb0 ">getallqueries-client <redacted-ipv4-address>", 
    sock=sock@entry=0x7f9a71ca0abc) at api.c:649
#2  0x00005598ffcdd115 in process_request (client_message=client_message@entry=0x7f9a64000bb0 ">getallqueries-client <redacted-ipv4-address>", 
    sock=sock@entry=0x7f9a71ca0abc) at request.c:63
#3  0x00005598ffcdbc8b in telnet_connection_handler_thread (socket_desc=0x7f9a6c000b20) at socket.c:326
#4  0x00007f9a74a75587 in ?? () from /usr/glibc-compat/lib/libpthread.so.0
#5  0x00007f9a747aa97f in clone () from /usr/glibc-compat/lib/libc.so.6
(gdb) 

Device specifics

Hardware Type: VM
OS: Alpine Linux.

Additional information

Pi-hole is not a DHCP server in this scenario. It is referring all reverse lookup queries to an upstream server, hence there is a small delay between new clients arriving in pi-hole (As IP addresses) vs being PTR resolved to hostnames.

[DL6ER]

Your version of FTL is from May. Can you update to the most recent beta version and see if the issues is still there?

[DL6ER]

Ignore my previous post. Using the debugging output you provided I was able to isolate the cause of this failure. It still exists in the most recent version of the beta testing branches. I'm in the train and will push a fix soon.

[davidsuart]

I don't mind testing @DL6ER

When you say "update to the most recent beta version" ... are you implying to pull the binary from an alternate branch or that the FTLDNS branch should already have a newer binary? ... I pulled this one from FTLDNS about ten days ago.

[DL6ER]

Yesterday, we updated the FTLDNS branch. However, I discovered the bug you're affected by. It is not yet resolved in the current beta binary, but will be solved in the next one (see #329).

Thanks for reporting this and providing very useful debugging information!