New admin features (idea)

[DonLexos]

Hi,

I am very keen on all the great work people are doing to enhance the admin interface. Right now there are several items like #23 and #20 that show different information both on the index. I also saw the whitelist/blacklist addition that was available from the left navigation menu. And MySQL stats, someone creating a SQLite version of it . . .cool things!

I think it might help to set some guidance (by @jacobsalmela ?) on what would be the best way to add features. For example I think it would be impractical to combine both #23 and #21 on the same index. They do display different information so having them both would be great, just not sure if that should be on 1 page (load/processing time).

Wouldn't it help if pages like these would move to the navigation on the left and the index is simply a summary? That way (if they become separate pages) they might not get stuck when master/dev branches become different since they are in different parts.

I can also imagine that when the pihole.log is parsed into a 'standard' database type/format the stats can be run through php instead of fetching all the data live from the file itself (especially since that is purged once a day). Not sure what would be the best way to set that up would be, but I thought maybe expressing the idea would get a conversation regarding this started.

Regards Alex

---

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[AzureMarker]

This would be helpful. I started a small contributor section in the readme about the git side of adding features, but we also need a practical guide for where to add things and how to format it.

[jacobsalmela]

I would like the main page to show the stats. The white/blacklist can be on separate pages with a link in the sidebar.

[DonLexos]

Well I opened this since some guidance might be helpful for people that are submitting ideas and code. There are 2 additions on the homepage that are both nice, do not show the exact same information but might be strange if you put them on the same page (if for no other reason only for the load of the page). In my humble opinion the first page should show a summary of the statistics and load quickly, and could then provide a link to a page with more detailed information.

If the date would be added in a 'standard' method in SQLite if would make sense to pull it from there, if not only because the log file might be parsed every 15 minutes instead of daily.

[ghost]

Would people be interested in an admin panel re-written in python? Possibly pip-installable, with a bunch of other graphs? I've made (rewriting and) extending the pi-hole admin page as a bit of a hobby. So far my admin page looks like:

(Note: I spliced three screenshots to get the three different tool-tips to show what the underlying data is)

Pretty easy to extend since it's all written in python, using mako templates for generating the pages and chart.js for the charts, and kept running by supervisor - updates whenever you read the page with a log tailer, so it's not constantly re-reading the log file.

[AzureMarker]

Right now there's a pr open for some graphs (#25), but when we move the graphs over to a different page on the interface, we can look into adding some of these graphs as well.

[jacobsalmela]

@Mcat12 mentioned the other PR, but your screenshot looks very impressive. The only suggestion I would have is to match the color scheme of the four boxes on top so they correlate (i.e. a blue line on the graph would represent ads blocked).

[AzureMarker]

Yes, it is impressive.

[ghost]

Finally got the hang of drilldown in Highcharts...

If you click the 17 on the Axis it drills down all series for that hour:

And if you click any data point on the minute-ly graph, (e.g. 17:48 for total):

Kinda cool... Drilling down into rackcdn.com twice:

Back to the main graph, clicking on a single data point, not on an axis, drills down solely into that point:

Into (same numbers as before for that hour, for green:

Pretty cool; Highcharts also supports exporting the graph to a bunch of formats:

As for your point about the colours, good idea! Just by complete fluke the default colours for Highcharts is pretty close, but when I had charts.js, I was generating distinct colours myself based on the number of series/values (hsv->rgba).

Anyway, considering this is entirely a re-write (doesn't use chronometer.sh, or lighttpd for the webserving), it'd probably make more sense for it to be its own repo, but I figured i'd post some updates here with ideas i've had for visualising the data.

[jacobsalmela]

This is pretty insane! I like it...just wondering how we could integrate it since you mentioned it was an entire revamp.

[PromoFaux]

There is also the issue of Highcharts being on one of the blocklists. Cant remember where I saw that, but we'd have to hardcode it to be whitelisted.

Should probably find out why it was whitelisted. Is Highcharts free to use? I recall from looking at it for something at work that there is a license cost to it...

[PromoFaux]

Ah, it was on reddit

[ghost]

Nah - code.highcharts.com (where the js is served from) isn't blocked, only the www subdomain (which is dumb anyway):

Regarding the licensing - Ah. I'm sort of new to opensource, so everything i've ever done would fall under 'personal use'. Oh well, for the time being I suppose i'll continue this as a personal hobby. ;)

[PromoFaux]

Ah right OK then.

Also, we should be good. It's covered by creative commons v3.0, but I'm a bit flaky with licences. @jacobsalmela , any thoughts?

[jacobsalmela]

If we ever want to commercialize anything, that license might cause issues.

[dschaper]

There's a package that another project I helped out on, the RPIMonitor, and it used http://www.flotcharts.org/ and I think it's open licensed. Maybe that could help out? I know with RPIMon it used RRD's for the backend database and I haven't had a chance to dig to deep, but HighCharts does have a pretty restrictive license when it comes to commercial usage...

[rkallos]

Hey guys. Great work on the 2.5.3 release! I really like the new stats on index.php, and I see some interesting avenues for possible enhancements.

The features I've been thinking of implementing are:

• Keeping pihole.log in RAM and flushing it more frequently, decreasing SD card deterioration and improving processing speed. This should only be done once some long-term stats solution is implemented.
• Storing long-term stats in a SQLite database. I got this mostly working before 2.5.3 was released with the new dashboard. Nowadays I'm mostly thinking of what data should be kept in the database so that it scales gracefully over time.

My current thoughts for a database schema include:

• Summary statistics every time piholeLogFlush.sh is run: Date, # of ads blocked, # of DNS queries. We could use these to calculate summary statistics for any desired time range, which could be handy. If piholeLogFlush is run multiple times per day (e.g. every 15 minutes), it would also be possible to see trends in daily use, which could be fun! We could also accumulate statistics in a single row, once they reach a certain age, to keep the size of the database constant.
• Summary statistics on domains and/or clients: This doesn't really interest me very much, mainly for privacy reasons, but I know others may feel different. Perhaps storing these statistics can be opt-in?

I'm very interested to hear thoughts about what we should store in a DB. Meanwhile, I hope to get a PR up sometime soon.

[jacobsalmela]

I made a mysql database that would track long-term changes and save them to a database--it works, but it isn't very good. It would also need to be ported to some lighter-weight DB for better performance.

[sachilles]

Hi,

if mysql is to heavy why you don't think of using RRD as suggested? Such databases are used for a lot of applications (in example for system montoring using Munin or Monitorix installed on the raspberry).

[dschaper]

Benefits of RRD would also be that we know ahead of time what the file sizes are going to be, since the are created at full size, and I know you can do a lot with RRD's tools to import and export the data in just about any format you could want. The downsides are that the version of RRD that ships with Jessie's repos is kind of old... (Another project that I use uses collectd to dump data to RRD's and then plots system performance stats...)

[DL6ER]

This issue is very old and it not clear if there are any feature requests left. If so, please open a ticket on Discourse.