Add additional reverse proxy setup metrics from Varnish logs

[LeeNX]

Having a reverse proxy in front of a web server, means that your reverse proxy would have more accurate visitor details compared to your web server, unlike #78, I have details that would be interesting to see per url, like the number of cache hits, age, times.

[allinurl]

@LeeNX, I'm not sure I understand how this differs from #78? Thanks

[LeeNX]

Meant to documenting the log format, but close the set of windows that had the comment, now redone. ;-)

Items of interest might be object cache hits, age and maybe how often same items are missed, or fall out of the cache.

Log Output:

123.12.1.12 - 2014-12-04 13:56:53 +0200 "GET http://www.somesite.tld/tools/widget?limit=1 HTTP/1.1" 200 1751 "http://www.somesite.tld/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36" GET www.somesite.tld /tools/widget ?limit=1 varnish-server vcl-a default_director webheadA 0.000097275 0 0.000224 hit 3 91 120.000 172800.000 200 Healthy PC -
123.12.1.12 - 2014-12-04 13:57:24 +0200 "GET http://www.somesite.tld/tools/widget?limit=1 HTTP/1.1" 200 1751 "http://www.somesite.tld/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36" GET www.somesite.tld /tools/widget ?limit=1 varnish-server vcl-a default_director webheadB 0.111841917 0 0.111945 miss - 0 120.000 172800.000 200 Healthy PC -
123.12.1.12 - 2014-12-04 13:57:54 +0200 "GET http://www.somesite.tld/tools/widget?limit=1 HTTP/1.1" 200 1751 "http://www.somesite.tld/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36" GET www.somesite.tld /tools/widget ?limit=1 varnish-server vcl-a default_director webheadB 0.000109673 0 0.000263 hit 1 30 120.000 172800.000 200 Healthy PC -
123.12.1.12 - 2014-12-04 13:58:24 +0200 "GET http://www.somesite.tld/tools/widget?limit=1 HTTP/1.1" 200 1751 "http://www.somesite.tld/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36" GET www.somesite.tld /tools/widget ?limit=1 varnish-server vcl-a default_director webheadB 0.000094652 0 0.000217 hit 2 61 120.000 172800.000 200 Healthy PC -
123.12.1.12 - 2014-12-04 13:58:55 +0200 "GET http://www.somesite.tld/tools/widget?limit=1 HTTP/1.1" 200 1751 "http://www.somesite.tld/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36" GET www.somesite.tld /tools/widget ?limit=1 varnish-server vcl-a default_director webheadB 0.000091553 0 0.000220 hit 3 91 120.000 172800.000 200 Healthy PC -

Log Format:

• Client IP
• username
• Date
• Time
• Time Zone
• URL encode " method complete URL and HTTP request type"
• HTTP responce code
• Object size
• Hostname
• Full User-Agent
• HTTP Method
• HTTP Hostname
• HTTP URL
• HTTP Query
• Varnish Server
• Varnish VCL
• Varnish Director
• Web Server
• Time to the first byte from the backend arrived
• Time taken to serve the request, in seconds
• Time taken to serve the request, in microseconds
• How the request was handled, whether it was a cache hit, miss, pass, pipe or error
• Number of Cache Hits
• Age of Object
• TTL for object
• Grace Mode Time
• HTTP Status returned from Web Server
• Backend Healthy
• Mobile Detect (UA)
• Compression type (gzip/none/deflate)

[allinurl]

I have a branch that I've been experimenting with, it allows users to add custom metrics to certain panels, though, I must say that allowing custom panels/modules might be ideal. Please keep this open so I can look into it in more detail.

Also, can you expand on how some of the items of interest should get displayed, e.g., a metric within a panel, new panel?

[LeeNX]

Really not sure how it would be best displayed, thinking about metric that would be useful to me. Some of the metric don't provide info all the time, so I might need to expend the logging to add a metric, so that reports can be more accurately displayed.

URL's sorted with highest cache hit rate, miss rate. Another would be to sort URL by Age?

Not sure how to represent the data, but resources with return code 200 comes out of cache, verse return 304 would be client side cached, yet unchanged. URLs with columns for return code count? ie ...

URL	Total hits	code 200 hits	code 304 hits
/favicon.ico	567	98	469
/robots.txt	123	23	100

I have a few other ideas, about time to server and time from backed, but not sure how much info you might be looking for. I have a few logs that might be worth sending offline that could give you more insight?

[allinurl]

Thanks for posting this in. I still think this should be configured at the user level. However, it definitely adds a good level of complexity since metrics would need to be defined by the user. The other option is to hard-code them if they are pretty standard. Are you looking to display these metrics on the request panel only? or could they go somewhere else too?

It's also worth mentioning that adding more metrics would increase the parsing time and memory consumption.

Feel free to post any other ideas... Thanks.

[LeeNX]

Thank you for entertaining some of our crazy ideas. And a bigger thank you for time spend on goaccess!!

Been able to create custom metrics at a user level would make goaccess even more useful tool, but I can see how that could get out of hand. Something I like, is the easy with which I can use and of late, make reports with.

Above counters would make sense in the request panel as extra columns..

Any additional code/complexity/metrics would require more resources, which seeing that I am the only person interested in these added metric, might not be worth extra resources for all users?