Error 500 on api.php?getAllQueries

[Windyo]

Expected Behaviour:

api.php?getAllQueries should return a list of all queries

Actual Behaviour:

an error 500 is thrown.
the other calls to api.php actually work.

Steps to reproduce this issue:

go to admin
go to "all queries"
an AJAX error is thrown.

Try to go to /api.php?getAllQueries
an error 500 is returned.

Mahalaka list used, but the other parts of the web interface work fine:

The only thing failing is "getAllQueries" which shouldn't be impacted by the presence of the list.

Setup:

Pi-hole Version v2.5.3 [clean install]
Web Interface Version v1.1.7
Raspberry Pi 3
Mahalaka list

[CDeeRON]

Works fine for me..check the size of your log file.

2.6.2 1.1.7

On Apr 13, 2016, at 3:40 PM, G notifications@github.com wrote:

/api.php?getAllQueries

[7h3ju57]

was an issue with php, since the mahalaka list is huge. its been fixed in newer versions.

time to update :)

[Windyo]

@7h3ju57 Duly noted, updating :)

[Windyo]

Update done, error stays exactly the same.
Pi-hole Version v2.6.2 Web Interface Version v1.1.7

[7h3ju57]

Only other thing i can think of is increase the php memory limit and
restart lighttpd service.

On Wed, Apr 13, 2016, 4:24 PM G notifications@github.com wrote:

Update done, error stays exactly the same.
Pi-hole Version v2.6.2 Web Interface Version v1.1.7

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#465 (comment)

[CDeeRON]

what is the size of your pihole.log?
-rw-r--r-- 1 dnsmasq root 11M Apr 13 16:29 /var/log/pihole.log

[Windyo]

@CDeeRON 492463.
@7h3ju57 noob questsion, but where ? There doesn't seem to be a memory_limit line in lighttpd.conf

[CDeeRON]

/etc/php5/*/php.ini

[CDeeRON]

curious perform pihole -f then try to reproduce issue

[Windyo]

@CDeeRON ran the command.
Admin console now displays 0 queries today.
Error is still present.

[7h3ju57]

Did you find php.ini?

You can try "whereis php" also

[Windyo]

Set the available memory to 256 in /cgi/php.ini from 128, and to 256 in /cli/php.ini from -1.

Error still present. :(

[Windyo]

Running apt-get update && apt-get upgrade. We'll see if that solves anything.

[7h3ju57]

I'm going to say wasnt the right location, should be under /etc/php...

On Wed, Apr 13, 2016, 4:49 PM G notifications@github.com wrote:

Set the available memory to 256 in /cgi/php.ini from 128, and to 256 in
/cli/php.ini from -1.

Error still present. :(

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#465 (comment)

[Windyo]

full path was /etc/php5/cli/php.ini and /etc/php5/cgi/php.ini

[CDeeRON]

i receive the errors below when using the api
Apr 13 16:52:23 dnsmasq[13808]: bad name at /etc/pihole/gravity.list line 373216 Apr 13 16:52:33 dnsmasq[13808]: bad name at /etc/pihole/gravity.list line 617001 Apr 13 16:52:33 dnsmasq[13808]: bad name at /etc/pihole/gravity.list line 617002 Apr 13 16:52:33 dnsmasq[13808]: bad name at /etc/pihole/gravity.list line 617011 Apr 13 16:52:33 dnsmasq[13808]: bad name at /etc/pihole/gravity.list line 617012 Apr 13 16:52:38 dnsmasq[13808]: bad name at /etc/pihole/gravity.list line 771305 Apr 13 16:52:38 dnsmasq[13808]: bad name at /etc/pihole/gravity.list line 771306 Apr 13 16:53:07 dnsmasq[13808]: bad name at /etc/pihole/gravity.list line 1516163 Apr 13 16:53:07 dnsmasq[13808]: bad name at /etc/pihole/gravity.list line 1516164

[CDeeRON]

when i remove the Mahalaka list and run gravity the API becomes functional and no errors in log

[Windyo]

I just realized that /etc/pihole/gravity.list is listed as an advertizer whith Mahalaka... Maybe that's the source of the problem ?

Let me take a look.

---

I can't seem to find a reason for which gravity.list would be detected as a pi-holable domain with mahalaka. I'll try and search deeper sometime this week. If anyone wants to dig further, please do.

[Windyo]

@CDeeRON could you give me the procedure to generate the log you just did ? I don't find the same messages in /var/log/

[AzureMarker]

The reason localhost and gravity show up as blocked is because they are being loaded off of the pi. All queries redirected to the pi are listed as "blocked" although some are just from normal web interface usage. The php error is most likely coming from memory limits, which can't be easily solved by us, since we are not actually paid to do this, until we switch to using a database. We are looking for ways to limit this from occurring until then, and have made a few optimizations already. :)

[Windyo]

@Mcat12 I am not totally sure it's an out of memory error, seeing as the query log is empty as of writing.

If I can just understand how @CDeeRON generated his log, I'll take a look at it myself and report back. Do you have any idea about that ?

[CDeeRON]

@Windyo Sorry I was away from my computer. The log from which I reported the errors should be located /var/log/pihole.log. If you don't see it then try enabling debug for pihole with pihole -d.
-d, debug Start a debugging session if having trouble

[Windyo]

@CDeeRON no problem at all, it's already cool you're helping me out.

To stop the debug session should I just run the command again ? I'll report back once / if I find anything

[CDeeRON]

I found the entries listed as "bad name" are prefixed with . example below, in /etc/pihole/gravity.list
.blogspot.l.googleusercontent.com

It outputs to screen what you need to do, CRTL C

[Windyo]

I found multiple entries that had a strange format for the moment :
URLS starting with points, with hypens, with hypen-minus.... I'll try and sanitize it, see what happens.

[CDeeRON]

If the log is flushed daily why do we need the api call getAllQueries vs dns_queries_today. They both should show the same right?

[Windyo]

@CDeeRON my log is empty after running it and trying to access the api. I'll try to dig more later on.

[7h3ju57]

have you checked the lighttpd log?

[CDeeRON]

@7h3ju57 see error below

2016-04-15 10:35:18: (mod_fastcgi.c.2562) unexpected end-of-file (perhaps the fastcgi process died): pid: 2982 socket: unix:/var/run/lighttpd/php.socket-0 
2016-04-15 10:35:18: (mod_fastcgi.c.3346) response not received, request sent: 868 on socket: unix:/var/run/lighttpd/php.socket-0 for /admin/api.php?getAllQueries&_=1460730938591, closing connection 

[CDeeRON]

Try this
in file /var/www/html/admin/data.php

comment out the following lines like have below

//        $fileName = '/etc/pihole/gravity.list';
//        //Turn gravity.list into an array
//        $lines = explode("\n", file_get_contents($fileName));

        //Create a new array and set domain name as index instead of value, with value as 1
//        foreach(array_values($lines) as $v){
//            $new_lines[trim(strstr($v, ' '))] = 1;
//        }

[CDeeRON]

Also increase memory for php in file etc/php5/cgi/php.ini
memory_limit = 256M

[CDeeRON]

I can't reproduce the issue after these changes with a clean install

[PromoFaux]

OK, trying to catch up here. I'm confused @CDeeRON, why are you commenting out that section of data.php?

[CDeeRON]

With the Mahalaka list enabled it makes the gravity.list to much for PHP to handle it seems. Also the end result for both the API admin/api.php?getAllQueries and url /admin/queries.php the function does seem to be needed. Correct me if I'm wrong.

[PromoFaux]

That section of the code is to match domains on the queries page with gravity.list so that we can display the status (pi-holed or ok)

I'm wondering if over a million domains is too much to load into an array....

[CDeeRON]

Maybe that info could be added the the query log? To prevent from having to pull from both files?

[PromoFaux]

/var/log/pihole.log is just an output of dnsmasq, and it logs them in quite an annoying way for parsing:

Apr 15 16:22:25 dnsmasq[30147]: query[A] vortex.data.microsoft.com from 192.168.217.38
Apr 15 16:22:25 dnsmasq[30147]: /etc/pihole/gravity.list vortex.data.microsoft.com is 192.168.216.235
Apr 15 16:23:08 dnsmasq[30147]: query[A] accounts.google.com from 192.168.217.38
Apr 15 16:23:08 dnsmasq[30147]: forwarded accounts.google.com to 8.8.8.8
Apr 15 16:23:08 dnsmasq[30147]: reply accounts.google.com is 216.58.212.237

So the top query is blocked, and the second is forwarded onto google.

I guess maybe (not tried this as I'm at work) we could change the logic up when reading in the log to say something like (warning... pseudo-code ahead!):

for each line in log
     if line contains 'query['
          get domain name, client IP, time for json object
          read next line
          if next line contains '/etc/pihole/gravity.list'
               set status as pi-holed in json
          else
               set status as OK in json
          end if
    end if

[Windyo]

@CDeeRON confirmed working after commenting out the lines.
Thank you 👍

Also refreshed the hosts file to integrate the "bad domains" I removed before : still works. Seems "bad domains" weren't an issue after all.

I'll give the pihole a spin this week-end and see if anything deviates from normal behavior.

[Windyo]

Quick testing seems to indicate that while addresses are being pi-holed, they are no longer displaying as such in the queries.php view., as predicted by @PromoFaux. I would rather that than an error, so I'll keep it that way :)

Thanks to everyone involved, I'll continue digging and see if I can find a solution to the underlying cause.

[PromoFaux]

I've just sat down at home with a beer and my favourite editor... so I'll have a bash at translating that pseudo-code above to something that actually works!

[PromoFaux]

I'm going to move this issue over to the AdminLTE repo for easier tracking, as it's an issue with the PHP and not the bash scripts!