-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add more metrics to Filebeat harvesters #13395
Add more metrics to Filebeat harvesters #13395
Conversation
0765b6a
to
5e3a410
Compare
@ycombinator @cachedout Would be nice if we could display these information in stack monitoring UI. The @kvch why do have some entries no size and read_offset? If we know the file needs to be processed soonish, we should set How about adding some status telling us if the file is currently open or not. In the future we could add more info like |
Thanks for doing this, @kvch and thanks tagging us, @urso From the @elastic/stack-monitoring side, there's an important caveat here in that right now we display the same set of metrics regardless of which type of Beat is being monitored. So, we have a little bit of foundational work to do in order to support the display of this kind of data. I have broadly outlined the work here: |
@urso In this PR only opened files reported. It would be indeed interesting to show other file states. But that should be done on the input level, not in the harvester. |
fe3c0a1
to
60f97c0
Compare
Agreed. I think they are important though. Otherwise lag stats are wrong and confusing if we compare the sums of |
@urso Should I add that functionality to this PR? |
If its not to complicated yes, please. |
60f97c0
to
7a376b0
Compare
Current format of reporting the file states besides running: {"filebeat":
{"log":
{"463bc38-6737-4938-9371-9afef4efb3c0":
{"harvesters":
{"/home/n/go/src/github":
{"com/elastic/beats/filebeat/test":
{"log":"file was removed"}}}}}} I need to revisit the format, because it is not human-readable as filenames are split along dots... |
fa2da63
to
3f2e91f
Compare
As you can see by my comment above, I have started working on reporting file states. However, I stopped because it has started to grow a lot in complexity. I would rather add this feature in a follow-up PR. This PR already provides better visibility into the harvesters' souls. |
I opened a follow-up issue: #14860 |
d450d49
to
5f33d27
Compare
Updated and rebased the PR. |
Jenkins, test this. |
jenkins test this |
Failing tests are unrelated. |
I have added a few new metrics to harvesters of Filebeat: - `"last_event_timestamp"`: `@timestamp` of the last published event - `"last_event_publised_time"`: the time when the last event was published and the offset was updated - `"size"`: file size in bytes - `"read_offset"`: offset of the file - `"start_time"`: harvester start time By reporting these metrics it is possible to create more complex checks in Kibana to help to diagnose issues in harvesters. ``` "harvester": { "files": { "0be8e828-e39f-42ba-8468-029a08451a37": { "last_event_timestamp": "2019-08-29T13:17:22.961Z", "last_event_published_time": "2019-08-29T13:17:22.961Z", "name": "/var/log/dpkg.log", "read_offset": 42417, "size": 42417, "start_time": "2019-08-29T13:17:19.908Z" }, "6950906f-ef19-4d99-aff9-ca81b49e024f": { "last_event_timestamp": "", "last_event_published_time": "", "name": "/var/log/pm-powersave.log", "start_time": "2019-08-29T13:17:22.907Z" }, "752bb2c3-2a61-4055-b3c6-5b8b87204f2b": { "last_event_timestamp": "2019-08-29T13:17:22.905Z", "last_event_published_time": "2019-08-29T13:17:22.905Z", "name": "/var/log/vbox-setup.log", "read_offset": 140, "size": 140, "start_time": "2019-08-29T13:17:19.908Z" }, "c339d330-cfa6-41ae-8a77-83a019ca99ab": { "last_event_timestamp": "2019-08-29T13:17:22.924Z", "last_event_published_time": "2019-08-29T13:17:22.925Z", "name": "/var/log/fontconfig.log", "read_offset": 2269, "size": 2269, "start_time": "2019-08-29T13:17:22.915Z" }, "edb8b270-904a-40dc-bb3b-e6ef278585a2": { "last_event_timestamp": "2019-08-29T13:17:22.915Z", "last_event_published_time": "2019-08-29T13:17:22.915Z", "name": "/var/log/alternatives.log", "read_offset": 5752, "size": 5752, "start_time": "2019-08-29T13:17:22.913Z" }, "f63a4922-164b-47bb-9edd-1b446685090c": { "last_event_@timestamp": "", "last_event_published_time": "", "name": "/var/log/pm-suspend.log", "start_time": "2019-08-29T13:17:22.906Z" } }, "open_files": 6, "running": 6, "started": 6 } ``` Closes elastic#7743 (cherry picked from commit 33a638b)
I have added a few new metrics to harvesters of Filebeat: - `"last_event_timestamp"`: `@timestamp` of the last published event - `"last_event_publised_time"`: the time when the last event was published and the offset was updated - `"size"`: file size in bytes - `"read_offset"`: offset of the file - `"start_time"`: harvester start time By reporting these metrics it is possible to create more complex checks in Kibana to help to diagnose issues in harvesters. ``` "harvester": { "files": { "0be8e828-e39f-42ba-8468-029a08451a37": { "last_event_timestamp": "2019-08-29T13:17:22.961Z", "last_event_published_time": "2019-08-29T13:17:22.961Z", "name": "/var/log/dpkg.log", "read_offset": 42417, "size": 42417, "start_time": "2019-08-29T13:17:19.908Z" }, "6950906f-ef19-4d99-aff9-ca81b49e024f": { "last_event_timestamp": "", "last_event_published_time": "", "name": "/var/log/pm-powersave.log", "start_time": "2019-08-29T13:17:22.907Z" }, "752bb2c3-2a61-4055-b3c6-5b8b87204f2b": { "last_event_timestamp": "2019-08-29T13:17:22.905Z", "last_event_published_time": "2019-08-29T13:17:22.905Z", "name": "/var/log/vbox-setup.log", "read_offset": 140, "size": 140, "start_time": "2019-08-29T13:17:19.908Z" }, "c339d330-cfa6-41ae-8a77-83a019ca99ab": { "last_event_timestamp": "2019-08-29T13:17:22.924Z", "last_event_published_time": "2019-08-29T13:17:22.925Z", "name": "/var/log/fontconfig.log", "read_offset": 2269, "size": 2269, "start_time": "2019-08-29T13:17:22.915Z" }, "edb8b270-904a-40dc-bb3b-e6ef278585a2": { "last_event_timestamp": "2019-08-29T13:17:22.915Z", "last_event_published_time": "2019-08-29T13:17:22.915Z", "name": "/var/log/alternatives.log", "read_offset": 5752, "size": 5752, "start_time": "2019-08-29T13:17:22.913Z" }, "f63a4922-164b-47bb-9edd-1b446685090c": { "last_event_@timestamp": "", "last_event_published_time": "", "name": "/var/log/pm-suspend.log", "start_time": "2019-08-29T13:17:22.906Z" } }, "open_files": 6, "running": 6, "started": 6 } ``` Closes #7743 (cherry picked from commit 33a638b)
This is great! Thank you for doing it, we could finally build ingestion lag monitoring for our logging pipeline! |
I have added a few new metrics to harvesters of Filebeat:
"last_event_timestamp"
:@timestamp
of the last published event"last_event_publised_time"
: the time when the last event was published and the offset was updated"size"
: file size in bytes"read_offset"
: offset of the file"start_time"
: harvester start timeBy reporting these metrics it is possible to create more complex checks in Kibana to help to diagnose issues in harvesters.
Closes #7743