Skip to content
Collection of PowerShell scripts for log parsing, data analysis and analytics
PowerShell
Branch: master
Clone or download
Latest commit 0673fd1 Dec 4, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
AccessLogParser Fixed bugs with AccessLogParser and Job server Jun 24, 2018
Job Server Bug fixes Dec 4, 2018
LICENSE Created a basic Powershell based Job server. Mar 24, 2018
readme.md Fixed bugs with AccessLogParser and Job server Jun 24, 2018

readme.md

Power Scripts

This repo is a collection PowerShell scripts to parse the various forms of plain text logs (e.g. apache access log, application error log) and run analytics to generate metrics. The metrics are saved as csv files, which can be imported into InfluxDB and visualized in grafana.

Scripts

PS Job Server

This is based on Powershell Team's basic server. This version build on top that introduce asynchronous execution, status tracking, and executing really long running powershell scripts on a server, triggered from a browser. It allows for additional arguments to be passed to those scripts from http payload (POST requests). The access can be controlled for a specific AD groups, and the commands to be executed needs to be stored in same folder as this script, and the script name becomes the command variable in the URL. It logs the commands to an audit log, keeps the history of the jobs in a separate file for tracking purposes. A simple bootstrapped html frontend gives an easy UI.

Access Log Analysis

Purpose of this PowerShell script is to process Access logs for usual web/app server platforms. This script takes an access log, splits them into fields, then calculates aggregates on those fields. Usually access logs capture information for each hit, and that is too much information for a long term analysis. The idea is to process the access log, and group the entries at a specific time slots (e.g. 5 min) and aggregate the metrics for long term analysis. It can generate additional metrics based on the aggregated data points (e.g. User concurrency) based on already captured data points.

Usage

AccessLogParser.ps1 -HostName jboss_host123 -source ".\access_log.2016-05-08.log" -target "./access_log_analysis.csv" -fields "3:Timestamp;7:Url;10:HttpCode;11:PageSize;12:TimeTaken" -group "Url;HttpCode" -aggregate "HttpCode:Count;PageSize:Sum;TimeTaken:Percentile_90;TimeTaken:Average" -calculate "Concurrency=(([HttpCode:Count]/300)*([TimeTaken:Average]/1000))" -time "[dd/MMM/yyyy:HH:mm:ss" -Interval 5 -split " "

You can’t perform that action at this time.