Yet another tail ElasticSearch/ELK logs program.
The purpose of this project is to provide a tail-like experience to parse ELK logs. Projects such as elktail seem to no longer support newer version of ElastiSearch.
There are several examples of using ElasticSearch 'scroll' feature, but as stated in the ES documentation that such an approach is heavy-weight and not intended for real-time requests.
ScrollES, OTOH, uses the 'search_after' feature of ES and is therefore lower impact.
NOTE: Tailing ElasticSearch is tricky in real time since all of the latest documents may not have settled by the time you query. The net-effect is that it is possible to have missing lines in the output. I have attempted to mitigate that as much as possible by returning data that is at least 20 seconds old.
$ scrolles.py --url <elasticsearch url> --index <elasticsearch index>
Help can be shown with the '-h' param:
$ scrolles.py -h
usage: scrolles.py [-h] [-u URL] [-i INDEX] [-k KEY] [-s SEARCH] [-n NUMLINES]
optional arguments:
-h, --help show this help message and exit
-u URL, --url URL ElasticSearch URL
-i INDEX, --index INDEX
ElasticSearch Index
-k KEY, --key KEY Keys to display
-s SEARCH, --search SEARCH
Search string
-n NUMLINES, --numlines NUMLINES
Initial number of lines to show from the logs
You can set default configuration by specifying the arguments in JSON. For example:
{
"url": "http://192.168.1.1:9200",
"index": "logstash-*"
}
Configuration is searched first in ~/.scrolles.json, then at /etc/scrolles.json
You can override configuration by specifying parameters at the command line.