Advanced stats

[bricej13]

Let's try this again.

I merged the changes into the devel branch changes. I ripped out the datatables piece as data.php isn't creating the proper data to support it.

Here's the original pull request info:

Added the following UI elements:

• A graph of queries over time
• Top chart of valid queries
• Top chart of advertisement queries
• Chart of most recent queries
• Removed 'More Info ->' links from top boxes to reduce confusion

Architecture changes

• sed/awk/wc replaced with PHP
• Moved data generation to separate file data.php file
• Data.php is used both in index.html and api.php

[AzureMarker]

You have a merge conflict in api.php

[DonLexos]

Played with this and it looks great! Not entirely sure what is causing the conflict (getting used to how github and the github software works), noticed your version on the bottom is 2.2 and the master is at 2.4 so could be you based this on that code?

[AzureMarker]

The current versioning system is based on the Pi-hole, not the web interface. I'm currently changing it so that there will be separate versions for both.

@bricej13 If you look in api.php, you'll see two versions of it, separated by a divider. You need to choose one or combine them. You do this by just modifying the file to how you want it to look (removing the stuff git put in). You can read more about it here.

[AzureMarker]

You should move your JS code to index, after php loads in the footer. Only the index needs that code, and it will error on the list pages.

[AzureMarker]

You may want to look into using ajax (through jQuery) to populate the tables. This will lower the load time of the page (which is a little long right now).

[bricej13]

I fixed the merge conflict in api.php and moved the JS code to index.php.

Using Ajax is a good idea. This would make it feasible to load entire lists instead of the first few elements.

@Mcat12 would you rather I create a new branch for adding the Ajax functionality or tack it on to this one?

[AzureMarker]

You can work on it in this one. Also, maybe you should separate out the original api from the other stuff. This will allow you to load the original stats with the page and use ajax for the lists of domains and such.

[AzureMarker]

Are your calculations taking into account ipv6? It's doubling the amount. Remember that users could have either ipv4 or both.

[PromoFaux]

Ah yeah, you need to test for IPv6, which is pretty much as simple as checking for the existence of /etc/pihole/.useIPv6

From chronometer.sh:

function CalcBlockedDomains(){
    CheckIPv6
    if [ -e "$gravity" ]; then
        #Are we IPV6 or IPV4?
        if [[ -n $piholeIPv6 ]];then
            #We are IPV6
            blockedDomainsTotal=$(wc -l /etc/pihole/gravity.list | awk '{print $1/2}')
        else
            #We are IPV4
            blockedDomainsTotal=$(wc -l /etc/pihole/gravity.list | awk '{print $1}')
        fi
    else
        blockedDomainsTotal="Err."
    fi
}

function CheckIPv6(){
    piholeIPv6file="/etc/pihole/.useIPv6"
    if [[ -f $piholeIPv6file ]];then
        # If the file exists, then the user previously chose to use IPv6 in the automated installer
        piholeIPv6=$(ip -6 route get 2001:4860:4860::8888 | awk -F " " '{ for(i=1;i<=NF;i++) if ($i == "src") print $(i+1) }')
    fi
}

[DonLexos]

I am not sure if I have the latest files but if I do the 2.5 release breaks a few things in the top domains/top advertisers.

[PromoFaux]

More than possible, do you have @bricej13 's version checked out on your pi? Because the changes aren't merged into the main site yet..

[DonLexos]

I grabbed it yesterday so I believe I have, but I am aware there is a merge conflict with master. Just wanted to test it out, looks good ! :-)

[PromoFaux]

I agree, I like the graphs a lot!

[PromoFaux]

But yeah, we may have broken functionality on it during the 2.5 update

[DonLexos]

Some files are moved, not sure if the logfile is filled differently with more information but instead of domain names now query[A] is shown. And my top advertiser is /etc/pihole/gravity.list :-)

Hope @bricej13 can resolve this.

[bricej13]

Made some updates:

• Updated API to allow data to be retrieved independently
• Moved most data calls to ajax
• Fixed total blocked domains number for ipv6 users
• Fixed Top Ads & Top Queries tables to show domains properly
• Fixed time column on Recent Queries table

[AzureMarker]

Good work. I'll test as soon as I get home!

[jmacdowall]

This is easy to fix. Edit /var/www/html/admin/data.php:

Just change a couple of lines:
OLD:
$domain = trim($exploded[5]);
NEW:
$domain = trim($exploded[6]);

OLD:
$queryArray['domain'] = trim($exploded[5]);
$queryArray['ip'] = trim($exploded[7]);
NEW:
$queryArray['domain'] = trim($exploded[6]);
$queryArray['ip'] = trim($exploded[8]);

@bricej13 - sweet graph!!

[AzureMarker]

What are you fixing @jmacdowall ?

[bricej13]

More Charts!

• Query Types: Shows what types of queries are being made (i.e. MX, AA, AAA, SOA, etc.)
• Top Clients: Shows all clients making requests and the amount of requests being made
• Forward Destinations: External DNS servers being forwarded to (i.e. 8.8.8.8, 8.8.4.4)

[jacobsalmela]

Beautiful...We'll check this out. Thanks!

[AzureMarker]

Looks good. If there's any way to have all the hours of the day show up on the graph, that would be nice. Everything looks so nice!

[bricej13]

@Mcat12 I fixed the issue where hours would be missing from the graph. It should now show all hours between the first query of the day and the last.

I also added a new page (the Query Log) that has a table of all of the queries for the day.

[bricej13]

@jacobsalmela How are you coming on getting the whitelist/blacklist files updated? Could we strip out the editing piece for now? That would allow us to get the new dashboard out while a solution is worked on for editing the lists.

[jacobsalmela]

@bricej13 I'm good with that. Do you want to update this PR with the white and blacklist links removed from the sidebar (and also fix the conflicts).

[AzureMarker]

@jacobsalmela we can merge this and when melting into master just not merge the whitelist stuff. I think there's an option to allow you to edit the merge commit before merging.

[AzureMarker]

Checkout --no-commit here.

[AzureMarker]

Shoot! This touchscreen is wacky!

[AzureMarker]

I was trying to comment, not close! -.-

[bricej13]

It should be ready to merge now.

[AzureMarker]

Serbs stable enough here. I'll merge soon if no one objects.

[jacobsalmela]

I'm OK with it. It's going into devel so we can work out any final issues there.

[PromoFaux]

Wow, this page is looking flashy as hell now. 👍

[jacobsalmela]

+1

[DonLexos]

Loving this! One (tiny) thing, Query log time stamps are shown like 2016-02-15T02:01:04 but its Mon 15 Feb 01:00:59 CET 2016 here now. They are 1 hour off.. and I can't figure out why. (date taken from date at the pi (reckoning it uses that date).

[jmacdowall]

My timestamps are off by 4 hours—same as my time zone offset. Coincidence?

I set the time “wrong” on my raspberry to fix it…

John MacDowall
UDP SysAdmin [Contractor]
Building 50, Room 5523
National Institutes of Heath
Bethesda, MD
301-827-4433

From: Alex [mailto:notifications@github.com]
Sent: Sunday, February 14, 2016 7:03 PM
To: pi-hole/AdminLTE
Cc: Macdowall, John (NIH/NHGRI) [C]
Subject: Re: [AdminLTE] Advanced stats (#25)

Loving this! One (tiny) thing, Query log time stamps are shown like 2016-02-15T02:01:04 but its Mon 15 Feb 01:00:59 CET 2016 here now. They are 1 hour off.. and I can't figure out why.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/25#issuecomment-184008940.

[DonLexos]

Oh that must be it, CET is GMT+1 .. Funny thing is, I checked the logfiles, there the time is perfect and ok, and I ran raspi-config to make sure it is set correctly to begin with (and it was).