alp is Access Log Profiler
You can pick your download here, and install it as follows:
sudo install <downloaded file> /usr/local/bin/alpInstall alp with Homebrew
brew install alp
Install alp with asdf and asdf-alp
asdf plugin-add alp https://github.com/asdf-community/asdf-alp.git
asdf install alp <version>
asdf global alp <version>See: The difference between v0.4.0 and v1.0.0
$ alp --help
usage: alp [<flags>] <command> [<args> ...]
alp is the access log profiler for LTSV, JSON, and others.
Flags:
--help Show context-sensitive help (also try --help-long and --help-man).
-c, --config=CONFIG The configuration file
--file=FILE The access log file
-d, --dump=DUMP Dump profiled data as YAML
-l, --load=LOAD Load the profiled YAML data
--sort=count Output the results in sorted order
-r, --reverse Sort results in reverse order
-q, --query-string Include the URI query string.
--format=table The output format (table, markdown, tsv and csv)
--noheaders Output no header line at all (only --format=tsv, csv)
--show-footers Output footer line at all (only --format=table, markdown)
--limit=5000 The maximum number of results to display.
--location=Local Location name for the timezone
-o, --output=all Specifies the results to display, separated by commas
-m, --matching-groups=PATTERN,...
Specifies URI matching groups separated by commas
-f, --filters=FILTERS Only the logs are profiled that match the conditions
--pos=POSITION_FILE The position file
--nosave-pos Do not save position file
--version Show application version.
Commands:
help [<command>...]
Show help.
ltsv [<flags>]
Profile the logs for LTSV
json [<flags>]
Profile the logs for JSON
regexp [<flags>]
Profile the logs that match a regular expression- Parses a log in LTSV format
- By default, the following labels are parsed:
time- datetime
method- HTTP Method
uri- URI
status- HTTP Status Code
apptime- Response time from the upstream server
reqtime- Request Processing Time (Response time after receiving a request)
- The
--xxx-labeloption can you change the name to any label
$ cat example/logs/ltsv_access.log
time:2015-09-06T05:58:05+09:00 method:POST uri:/foo/bar?token=xxx&uuid=1234 status:200 size:12 apptime:0.057
time:2015-09-06T05:58:41+09:00 method:POST uri:/foo/bar?token=yyy status:200 size:34 apptime:0.100
time:2015-09-06T06:00:42+09:00 method:GET uri:/foo/bar?token=zzz status:200 size:56 apptime:0.123
time:2015-09-06T06:00:43+09:00 method:GET uri:/foo/bar status:400 size:15 apptime:-
time:2015-09-06T05:58:44+09:00 method:POST uri:/foo/bar?token=yyy status:200 size:34 apptime:0.234
time:2015-09-06T05:58:44+09:00 method:POST uri:/hoge/piyo?id=yyy status:200 size:34 apptime:0.234
time:2015-09-06T05:58:05+09:00 method:POST uri:/foo/bar?token=xxx&uuid=1234 status:200 size:12 apptime:0.057
time:2015-09-06T05:58:41+09:00 method:POST uri:/foo/bar?token=yyy status:200 size:34 apptime:0.100
time:2015-09-06T06:00:42+09:00 method:GET uri:/foo/bar?token=zzz status:200 size:56 apptime:0.123
time:2015-09-06T06:00:43+09:00 method:GET uri:/foo/bar status:400 size:15 apptime:-
time:2015-09-06T06:00:43+09:00 method:GET uri:/diary/entry/1234 status:200 size:15 apptime:0.135
time:2015-09-06T06:00:43+09:00 method:GET uri:/diary/entry/5678 status:200 size:30 apptime:0.432
time:2015-09-06T06:00:43+09:00 method:GET uri:/foo/bar/5xx status:504 size:15 apptime:60.000
$ cat example/logs/ltsv_access.log | alp ltsv
+-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+
| COUNT | 1XX | 2XX | 3XX | 4XX | 5XX | METHOD | URI | MIN | MAX | SUM | AVG | P1 | P50 | P99 | STDDEV | MIN(BODY) | MAX(BODY) | SUM(BODY) | AVG(BODY) |
+-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+
| 2 | 0 | 2 | 0 | 0 | 0 | GET | /foo/bar | 0.123 | 0.123 | 0.246 | 0.123 | 0.123 | 0.123 | 0.123 | 0.000 | 56.000 | 56.000 | 112.000 | 56.000 |
| 1 | 0 | 1 | 0 | 0 | 0 | GET | /diary/entry/1234 | 0.135 | 0.135 | 0.135 | 0.135 | 0.135 | 0.135 | 0.135 | 0.000 | 15.000 | 15.000 | 15.000 | 15.000 |
| 5 | 0 | 5 | 0 | 0 | 0 | POST | /foo/bar | 0.057 | 0.234 | 0.548 | 0.110 | 0.057 | 0.100 | 0.057 | 0.065 | 12.000 | 34.000 | 126.000 | 25.200 |
| 1 | 0 | 1 | 0 | 0 | 0 | POST | /hoge/piyo | 0.234 | 0.234 | 0.234 | 0.234 | 0.234 | 0.234 | 0.234 | 0.000 | 34.000 | 34.000 | 34.000 | 34.000 |
| 1 | 0 | 1 | 0 | 0 | 0 | GET | /diary/entry/5678 | 0.432 | 0.432 | 0.432 | 0.432 | 0.432 | 0.432 | 0.432 | 0.000 | 30.000 | 30.000 | 30.000 | 30.000 |
| 1 | 0 | 0 | 0 | 0 | 1 | GET | /foo/bar/5xx | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 | 0.000 | 15.000 | 15.000 | 15.000 | 15.000 |
+-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+
| 11 | 0 | 10 | 0 | 0 | 1 |
+-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+- Parse a log with one JSON per line
- By default, the following keys are parsed:
time- datetime
method- HTTP Method
uri- URI
status- HTTP Status Code
response_time- Response time from the upstream server
request_time- Request Processing Time (Response time after receiving a request)
- The
--xxx-keyoption can you change the name to any key
$ cat example/logs/json_access.log
{"time":"2015-09-06T05:58:05+09:00","method":"POST","uri":"/foo/bar?token=xxx&uuid=1234","status":200,"body_bytes":12,"response_time":0.057}
{"time":"2015-09-06T05:58:41+09:00","method":"POST","uri":"/foo/bar?token=yyy","status":200,"body_bytes":34,"response_time":0.100}
{"time":"2015-09-06T06:00:42+09:00","method":"GET","uri":"/foo/bar?token=zzz","status":200,"body_bytes":56,"response_time":0.123}
{"time":"2015-09-06T06:00:43+09:00","method":"GET","uri":"/foo/bar","status":400,"body_bytes":15,"response_time":"-"}
{"time":"2015-09-06T05:58:44+09:00","method":"POST","uri":"/foo/bar?token=yyy","status":200,"body_bytes":34,"response_time":0.234}
{"time":"2015-09-06T05:58:44+09:00","method":"POST","uri":"/hoge/piyo?id=yyy","status":200,"body_bytes":34,"response_time":0.234}
{"time":"2015-09-06T05:58:05+09:00","method":"POST","uri":"/foo/bar?token=xxx&uuid=1234","status":200,"body_bytes":12,"response_time":0.057}
{"time":"2015-09-06T05:58:41+09:00","method":"POST","uri":"/foo/bar?token=yyy","status":200,"body_bytes":34,"response_time":0.100}
{"time":"2015-09-06T06:00:42+09:00","method":"GET","uri":"/foo/bar?token=zzz","status":200,"body_bytes":56,"response_time":0.123}
{"time":"2015-09-06T06:00:43+09:00","method":"GET","uri":"/foo/bar","status":400,"body_bytes":15,"response_time":"-"}
{"time":"2015-09-06T06:00:43+09:00","method":"GET","uri":"/diary/entry/1234","status":200,"body_bytes":15,"response_time":0.135}
{"time":"2015-09-06T06:00:43+09:00","method":"GET","uri":"/diary/entry/5678","status":200,"body_bytes":30,"response_time":0.432}
{"time":"2015-09-06T06:00:43+09:00","method":"GET","uri":"/foo/bar/5xx","status":504,"body_bytes":15,"response_time":60.000}
$ cat example/logs/json_access.log | alp json
+-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+
| COUNT | 1XX | 2XX | 3XX | 4XX | 5XX | METHOD | URI | MIN | MAX | SUM | AVG | P1 | P50 | P99 | STDDEV | MIN(BODY) | MAX(BODY) | SUM(BODY) | AVG(BODY) |
+-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+
| 2 | 0 | 2 | 0 | 0 | 0 | GET | /foo/bar | 0.123 | 0.123 | 0.246 | 0.123 | 0.123 | 0.123 | 0.123 | 0.000 | 56.000 | 56.000 | 112.000 | 56.000 |
| 1 | 0 | 1 | 0 | 0 | 0 | GET | /diary/entry/1234 | 0.135 | 0.135 | 0.135 | 0.135 | 0.135 | 0.135 | 0.135 | 0.000 | 15.000 | 15.000 | 15.000 | 15.000 |
| 5 | 0 | 5 | 0 | 0 | 0 | POST | /foo/bar | 0.057 | 0.234 | 0.548 | 0.110 | 0.057 | 0.100 | 0.057 | 0.065 | 12.000 | 34.000 | 126.000 | 25.200 |
| 1 | 0 | 1 | 0 | 0 | 0 | POST | /hoge/piyo | 0.234 | 0.234 | 0.234 | 0.234 | 0.234 | 0.234 | 0.234 | 0.000 | 34.000 | 34.000 | 34.000 | 34.000 |
| 1 | 0 | 1 | 0 | 0 | 0 | GET | /diary/entry/5678 | 0.432 | 0.432 | 0.432 | 0.432 | 0.432 | 0.432 | 0.432 | 0.000 | 30.000 | 30.000 | 30.000 | 30.000 |
| 1 | 0 | 0 | 0 | 0 | 1 | GET | /foo/bar/5xx | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 | 0.000 | 15.000 | 15.000 | 15.000 | 15.000 |
+-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+
| 11 | 0 | 10 | 0 | 0 | 1 |
+-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+- Parses the log to match the regular expression
- By default, the following named capture groups are parsed:
time- datetime
method- HTTP Method
uri- URI
status- HTTP Status Code
response_time- Response time from the upstream server
request_time- Request Processing Time (Response time after receiving a request)
- The
--xxx-subexpoption can you change the name to any named capture groups
$ cat example/logs/combined_access.log
127.0.0.1 - - [06/Sep/2015:05:58:05 +0900] "POST /foo/bar?token=xxx&uuid=1234 HTTP/1.1" 200 12 "-" "curl/7.54.0" "-" 0.057
127.0.0.1 - - [06/Sep/2015:05:58:41 +0900] "POST /foo/bar?token=yyy HTTP/1.1" 200 34 "-" "curl/7.54.0" "-" 0.100
127.0.0.1 - - [06/Sep/2015:06:00:42 +0900] "GET /foo/bar?token=zzz HTTP/1.1" 200 56 "-" "curl/7.54.0" "-" 0.123
127.0.0.1 - - [06/Sep/2015:06:00:43 +0900] "GET /foo/bar HTTP/1.1" 400 15 "-" "curl/7.54.0" "-" -
127.0.0.1 - - [06/Sep/2015:05:58:44 +0900] "POST /foo/bar?token=yyy HTTP/1.1" 200 34 "-" "curl/7.54.0" "-" 0.234
127.0.0.1 - - [06/Sep/2015:05:58:44 +0900] "POST /hoge/piyo?id=yyy HTTP/1.1" 200 34 "-" "curl/7.54.0" "-" 0.234
127.0.0.1 - - [06/Sep/2015:05:58:05 +0900] "POST foo/bar?token=xxx&uuid=1234 HTTP/1.1" 200 12 "-" "curl/7.54.0" "-" 0.057
127.0.0.1 - - [06/Sep/2015:05:58:41 +0900] "POST /foo/bar?token=yyy HTTP/1.1" 200 34 "-" "curl/7.54.0" "-" 0.100
127.0.0.1 - - [06/Sep/2015:06:00:42 +0900] "GET /foo/bar?token=zzz HTTP/1.1" 200 56 "-" "curl/7.54.0" "-" 0.123
127.0.0.1 - - [06/Sep/2015:06:00:43 +0900] "GET /foo/bar HTTP/1.1" 400 15 "-" "curl/7.54.0" "-" -
127.0.0.1 - - [06/Sep/2015:06:00:43 +0900] "GET /diary/entry/1234 HTTP/1.1" 200 15 "-" "curl/7.54.0" "-" 0.135
127.0.0.1 - - [06/Sep/2015:06:00:43 +0900] "GET /diary/entry/5678 HTTP/1.1" 200 30 "-" "curl/7.54.0" "-" 0.432
127.0.0.1 - - [06/Sep/2015:06:00:43 +0900] "GET /foo/bar/5xx HTTP/1.1" 504 15 "-" "curl/7.54.0" "-" 60.000
$ cat example/logs/combined_access.log | alp regexp
+-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+
| COUNT | 1XX | 2XX | 3XX | 4XX | 5XX | METHOD | URI | MIN | MAX | SUM | AVG | P1 | P50 | P99 | STDDEV | MIN(BODY) | MAX(BODY) | SUM(BODY) | AVG(BODY) |
+-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+
| 1 | 0 | 1 | 0 | 0 | 0 | POST | foo/bar | 0.057 | 0.057 | 0.057 | 0.057 | 0.057 | 0.057 | 0.057 | 0.000 | 12.000 | 12.000 | 12.000 | 12.000 |
| 2 | 0 | 2 | 0 | 0 | 0 | GET | /foo/bar | 0.123 | 0.123 | 0.246 | 0.123 | 0.123 | 0.123 | 0.123 | 0.000 | 56.000 | 56.000 | 112.000 | 56.000 |
| 1 | 0 | 1 | 0 | 0 | 0 | GET | /diary/entry/1234 | 0.135 | 0.135 | 0.135 | 0.135 | 0.135 | 0.135 | 0.135 | 0.000 | 15.000 | 15.000 | 15.000 | 15.000 |
| 4 | 0 | 4 | 0 | 0 | 0 | POST | /foo/bar | 0.057 | 0.234 | 0.491 | 0.123 | 0.057 | 0.100 | 0.234 | 0.067 | 12.000 | 34.000 | 114.000 | 28.500 |
| 1 | 0 | 1 | 0 | 0 | 0 | POST | /hoge/piyo | 0.234 | 0.234 | 0.234 | 0.234 | 0.234 | 0.234 | 0.234 | 0.000 | 34.000 | 34.000 | 34.000 | 34.000 |
| 1 | 0 | 1 | 0 | 0 | 0 | GET | /diary/entry/5678 | 0.432 | 0.432 | 0.432 | 0.432 | 0.432 | 0.432 | 0.432 | 0.000 | 30.000 | 30.000 | 30.000 | 30.000 |
| 1 | 0 | 0 | 0 | 0 | 1 | GET | /foo/bar/5xx | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 | 0.000 | 15.000 | 15.000 | 15.000 | 15.000 |
+-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+
| 11 | 0 | 10 | 0 | 0 | 1 |
+-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+See: Usage samples
-c, --config- The configuration file
- YAML
--file=FILE- The access log file
-d, --dump=DUMP- File path for creating the profile results to a file
-l, --load=LOAD- File path to read the results of the profile created with the
-d, --dumpoption - Can expect it to work fast if you change the
--sortand--reverseoptions for the same profile results
- File path to read the results of the profile created with the
--sort=count- Output the results in sorted order
- Sort in ascending order
max,min,sum,avgmax-body,min-body,sum-body,avg-bodyp1,p50,p99,stddevurimethodcount- The default is
count
-r, --reverse- Sort in desecending order
-q, --query-string- URIs up to and including query strings are included in the profile
--format=table- Print the profile results in a table, Markdown, TSV and CSV format
- The default is table format
--noheaders- Print no header when TSV and CSV format
--show-footers- Print the total number of each 1xx ~ 5xx in the footer of the table or Markdown format
--limit=5000- Maximum number of profile results to be printed
- This setting is to avoid using too much memory
- The default is 5000 lines
--location="Local"- The timezone of the time specified in the filter condition.
- Default is localhost timezone
-o, --output="all"- Specify the profile results to be print, separated by commas
count,1xx,2xx,3xx,4xx,5xx,method,uri,min,max,sum,avg,p1,p50,p99,stddev,min_body,max_body,sum_body,avg_body- The default is
all
-m, --matching-groups=PATTERN,...- Treat URIs that match regular expressions as the same URI
- See URI matching groups
-f, --filters=FILTERS- Filters the targets for profile
- See Filter
--pos=POSITION_FILE- Stores the number of bytes to which the file has been read.
- If the number of bytes is stored in the POSITION_FILE, the data after that number of bytes will be profiled
- You can profile without truncating the file
- Also, it is expected to work fast because it seeks and skips files
--nosave-pos- Data after the number of bytes specified by
--posis profiled, but the number of bytes reads is not stored
- Data after the number of bytes specified by
Consider the following cases like /diary/entry/1234 and /diary/entry/5678.
If you simply profile URIs with different parameters on the same route, they will be profiled by parameter, but you may want to profile them by the route.
$ cat example/logs/ltsv_access.log | alp ltsv --filters "Uri matches '^/diary/entry'"
+-------+-----+-----+-----+-----+-----+--------+-------------------+-------+-------+-------+-------+-------+-------+-------+--------+-----------+-----------+-----------+-----------+
| COUNT | 1XX | 2XX | 3XX | 4XX | 5XX | METHOD | URI | MIN | MAX | SUM | AVG | P1 | P50 | P99 | STDDEV | MIN(BODY) | MAX(BODY) | SUM(BODY) | AVG(BODY) |
+-------+-----+-----+-----+-----+-----+--------+-------------------+-------+-------+-------+-------+-------+-------+-------+--------+-----------+-----------+-----------+-----------+
| 1 | 0 | 1 | 0 | 0 | 0 | GET | /diary/entry/1234 | 0.135 | 0.135 | 0.135 | 0.135 | 0.135 | 0.135 | 0.135 | 0.000 | 15.000 | 15.000 | 15.000 | 15.000 |
| 1 | 0 | 1 | 0 | 0 | 0 | GET | /diary/entry/5678 | 0.432 | 0.432 | 0.432 | 0.432 | 0.432 | 0.432 | 0.432 | 0.000 | 30.000 | 30.000 | 30.000 | 30.000 |
+-------+-----+-----+-----+-----+-----+--------+-------------------+-------+-------+-------+-------+-------+-------+-------+--------+-----------+-----------+-----------+-----------+
| 2 | 0 | 2 | 0 | 0 | 0 |
+-------+-----+-----+-----+-----+-----+--------+-------------------+-------+-------+-------+-------+-------+-------+-------+--------+-----------+-----------+-----------+-----------+$ cat example/logs/ltsv_access.log | alp ltsv --filters "Uri matches '^/diary/entry'" -m "/diary/entry/.+"
+-------+-----+-----+-----+-----+-----+--------+-----------------+-------+-------+-------+-------+-------+-------+-------+--------+-----------+-----------+-----------+-----------+
| COUNT | 1XX | 2XX | 3XX | 4XX | 5XX | METHOD | URI | MIN | MAX | SUM | AVG | P1 | P50 | P99 | STDDEV | MIN(BODY) | MAX(BODY) | SUM(BODY) | AVG(BODY) |
+-------+-----+-----+-----+-----+-----+--------+-----------------+-------+-------+-------+-------+-------+-------+-------+--------+-----------+-----------+-----------+-----------+
| 2 | 0 | 2 | 0 | 0 | 0 | GET | /diary/entry/.+ | 0.135 | 0.432 | 0.567 | 0.283 | 0.135 | 0.135 | 0.135 | 0.148 | 15.000 | 30.000 | 45.000 | 22.500 |
+-------+-----+-----+-----+-----+-----+--------+-----------------+-------+-------+-------+-------+-------+-------+-------+--------+-----------+-----------+-----------+-----------+
| 2 | 0 | 2 | 0 | 0 | 0 |
+-------+-----+-----+-----+-----+-----+--------+-----------------+-------+-------+-------+-------+-------+-------+-------+--------+-----------+-----------+-----------+-----------+In such a case, there is an option -m, --matching-groups=PATTERN,....
You can also specify multiple items separated by commas.
It is a function to include or exclude targets according to the conditions.
Filter on the following variables:.
Uri- URI
Method- HTTP Method
Time- Datetime string
- See: https://github.com/tkuchiki/parsetime
ResponseTime- Response time
BodyBytes- Bytes of HTTP Body
Status- HTTP Status Code
The following operators are available:.
+,-,*,/,%,**(pow)==,!=,<,>,<=,>=not,!and,&&or,||matches- e.g.
Uri matches "PATTERN"not(Uri matches "PATTERN")
- e.g.
contains- e.g.
Uri contains "STRING"not(Uri contains "STRING")
- e.g.
startsWith- e.g.
Uri startsWith "PREFIX"not(Uri startsWith "PREFIX")
- e.g.
endsWith- e.g.
Uri endsWith "SUFFIX"not(Uri endsWith "SUFFIX")
- e.g.
in- e.g.
Method in ["GET", "POST"]Method not in ["GET", "POST"]
- e.g.
See: https://github.com/antonmedv/expr/blob/master/docs/Language-Definition.md
TimeAgo(duration)- now -
duration ns,us or µs,ms,s,m,h- e.g.
Time >= TimeAgo("5m")
- now -
BetweenTime(val, start, end)- Like SQL's
BETWEEN, returnsstart <= val && val <= end - e.g.
BetweenTime(Time, "2019-08-06T00:00:00", "2019-08-06T00:05:00")
- Like SQL's
See: Usage samples
Donations are welcome as always!
My Kyash account is tkuchiki.
