diff --git a/README.md b/README.md index cf3a5e3..ee35a26 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # lograte A small tool ([used to be](https://github.com/vearutop/lograte/blob/v0.1.0/main.go) ~25 lines of code) to calculate the -rate of lines in STDOUT and group by them by count with alphanumeric filter. +rate of lines in STDOUT and group them by count with alphanumeric filter. ## Installation @@ -44,7 +44,11 @@ ssh -C log-collector.acme.com 'tail -f /var/log/app-error.log | grep "my feature 6862 lines since Mar 3 23:46:32, 605.47 per second ``` -Or show top filtered messages by count or total size. +Or show top filtered messages by count or total size. + +> **_NOTE:_** Filtered messages have all case insensitive sequences of `[a-z]-_%` with at least one digit or all digits replaced with `X`. +> This is usually enough to remove dynamic data from message and decrease cardinality. +> Filtered messages are collected in a limited number of buckets, once the limit is met all other messages are collected into the `Other` bucket. ``` tail -f /var/log/quick.log | lograte -top 5 -t 1s -by-size @@ -54,8 +58,31 @@ tail -f /var/log/quick.log | lograte -top 5 -t 1s -by-size foo-bar-18 i2 2022/09/03 08:22:17.206522 271382 lines since Sep 3 08:22:14, 90447.4 per second, 68.3 MB/s, 791 B/avg ------ Top 5 ------------------------- -29810 lines, 9935.2 lps (11.0%), 14.7 MB/s (21.5%), 1546 B/avg: X X X/X/X X:X:X.X filtered entry -26559 lines, 8851.7 lps (9.8%), 9.1 MB/s (13.4%), 1083 B/avg: X X X/X/X X:X:X.X another filtered entry +29810 lines, 9935.2 lps (11.0%), 14.7 MB/s (21.5%), 1546 B/avg: X X X/X/X X:X:X.X filtered entry X.X +26559 lines, 8851.7 lps (9.8%), 9.1 MB/s (13.4%), 1083 B/avg: X X X/X/X X:X:X.X another filtered X entry ... --------------------------------------- ``` + +### Flags + +``` +lograte -help +``` +``` +Usage of lograte: + -buckets int + max number of buckets to track filtered messages (default 500) + -by-size + order messages by size instead of count + -len int + limit message length (default 120) + -line-buf int + line token buffer size (default 10000000) + -t duration + reporting interval (default 1s) + -top int + show top filtered messages ordered by rate + -version + print version and exit +```