-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
added path_key to include read path into the event under this key #281
Conversation
I think, using https://github.com/sonots/fluent-plugin-record-reformer is fine enough. You can add <source>
type tail
path /var/log/syslog
tag syslog
</source>
<match syslog>
type record_reformer
path /var/log/syslog
output_tag reformed.${tag}
</match> |
In case of many files using record_reformer is hardly possible, especially, if tail_ex will be merged into the in_tail (#269, #277) For example, how to add path of the read file into the event in case of configuration like the following one? <source>
type tail
path /var/log/**.log,/var/log/by-date/%Y/messages.%m/%Y%m%d
tag input_tag
</source> |
Ah, understand. |
@szhem It seems that, after merging tail_ex, it becomes possible to append file path into tag like <source>
type tail
path /var/log/**.log,/var/log/by-date/%Y/messages.%m/%Y%m%d
tag input_tag.*
</source> The Then, you should be able to put the file path into a record field using record-reformer. Is it fine enough for your purpose? |
Well, and yes, and no. def _receive_lines(lines)
tag = @path.tr('/', '.').gsub(/\.+/, '.').gsub(/^\./, '')
@parent_receive_lines.call(lines, tag)
end In case of path ./some/../../complex/../path the tag value will be input_tag.some.complex.path that cannot be translated into the original path, for example, to analyze original log files. Honestly speaking, the field that can be copy-pasted directly into the shell is pretty useful (splunk, logstash and other log-aggregating software use the full path of the source) |
Okay, understand, but we (fluentd committers) still do not know how many people need this functionality. Could it possible to create another extension plugin for this purpose? |
@szhem I created extension plugin based on your patch at https://github.com/sonots/fluent-plugin-tail_path. If you like to be a maintainer of this, I can transfer this repository to you. Please let me know. NOTICE: This plugin requires |
@sonots, thanks a lot for the extension plugin. For now, I have already modified tail_ex plugin for that purpose. |
Okay, let me close. I will release gem soon to see user's demands. |
@szhem Released fluent-plugin-tail_path http://rubygems.org/gems/fluent-plugin-tail_path since fluentd v0.10.45 was released. |
Thanks so much! |
When sending the events to the elasticsearch it''s quite useful to be able to understand what file is the origin of the received event.
The patch adds additional optional 'path_key' tag to the configuration, to hold the filename
All the changes are backward compatible as the path is inserted into the event only if path_key is specified.