Include tool for apache log stats/analysis in provision?

[mcheck]

Sorry if this is a bit off topic:

I have 2-3 sites running production with evolution and have come across a need to provide stats and apache log analysis for traffic, more than what google analytics can give us.

So, I'm looking for a recommendation/discussion for a web stats analyzer (awstats, webalyzer, etc) to integrate in the core provision to work with as the logs get pumping in production. Does that make sense to run it on the server? As a separately accessible service perhaps, running for staging and production only?

I see that the New Relic monitor can be installed, could that be used for web traffic with their New Relic Browser service, do any of you have experience with it?

Or perhaps a differenct remote service? Any suggestions?

Thanks

[EvanK]

New Relic is more geared toward app stability, resource and error monitoring...as far as I know, it doesn't have meaningful log analysis capabilities.

I have thought about web front-ends for some of the server-side tools (like fail2web), and how those could be exposed remotely via (for example) ssh port forwarding. That said, I'd be hesitant to pile too much on top of a production server, without knowing how it could affect performance of the site.

It might be feasible to, per your suggestion, configure & run a log stats tool on local, and sync logs down from staging/prod for analysis.

TL;DR: worth looking into, probably best to limit to a local stage for performance reasons

[EvanK]

From what preliminary reading I've done, it looks like awstats is the best candidate. It's widely used and more user friendly than webalizer or analog.

[mcheck]

Awstats was my pick as well of those.

I installed a few today and my favorite of them is piwik. It's not an Apache log analyzer of course, but has most of the stats I want from an awstats type system, I think. I installed on a small digital ocean droplet and had it running in less than 30 minutes. Plus being remote, there's no hit to server performance (other than outbound http).

Because it's browser side js it can be blocked, so of course it's not a true picture of service. How do you feel about that difference?

Regardless, thanks for noodling it.

[EvanK]

Piwik looks like it would require an always running and web accessible server to which stats would be reporting, aside of the production stage itself. It looks like a pretty useful tool, but might best be used with their existing wordpress plugin. Browser side integrations seem a little out of scope for evolution to manage.

Running awstats on the local stage may require little to no extra setup on the user's behalf (beyond what we can provision through ansible), and it could "pull" the logs from production at regular intervals (while the vm is up) or on demand (via a capistrano task, I'm thinking).

[EvanK]

I've got awstats more or less usable in #96.

I'd welcome a second set of eyes and have added a note to the readme on testing from a branch.

[EvanK]

This has been merged & tagged for release in 1.3.5