-
Notifications
You must be signed in to change notification settings - Fork 215
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tags in Object Key #31
Comments
Currently no. |
Yes please! How can I achieve this? I just could implement this locally but I just started playing with fluentd and plugins. |
How to implement? We can't assume single tag in s3 plugin because event has own tag.
In this case, events in s3 plugin may have foo.bar, foo.baz... tags. |
Yep I just want to add ${tag} in object key like this:
If event has tag foo.bar for example, I'm expecting to find it in the object key. Thanks |
Hm. In your approach, S3 plugin stores multiple objects into S3 at the same time, right? |
Yes. I need the %{tag} because I'm storing access logs grouped by domain and I'm passing the domain name in the tag.. |
Okay. Could you send the pull request? |
I don't know how to implement this, that's why I asked here :( |
I see. We need some time if implement. |
The is another plugin that adds this feature to the s3 plugin. Please check it out here: |
We just began using fluentd in production. Right now we're using the plugin that dave7373 mentioned to achieve storing logs for each event in a different folder. It's working, although I would like to explore if there is a more efficient way. The fluent-output-router starts a new fluent-plugin-s3 for every event. This creates a lot of threads if you have a lot of events. Is it due to fluentd having a single buffer queue structure that new outputs must be instantiated if you want to have separate chunks for each event?
In the approach you discussed above, did you mean that in the write method you would split the chunk into separate pieces based on tag and then write each sub-chunk to a different S3 file? The only problem I see here is that you may get very small S3 files if an event only occurs a few times within a chunk. Where as if you had a individual chunk for each event this would occur less often. Maybe that is not a problem, and can be mitigated by making the chunk size larger? Perhaps it is also more efficient than creating a new output for each event. Are there downsides to making the chunk size larger? According to the documentation the default chunk size is 8m. I should also mention that I would love to work on implementation if we come to some agreement on what the best solution is. Thanks! |
Coool 👍
Hmm... My concern is error handling. On the other hand, if we supports tag separation in
Maintain duplicated retry feature seems high cost and not so many advantages I think. |
Oops sorry, I did mean new tag.
I see, do you believe that this optimization would be better suited to become part of fluentd itself? Perhaps there would be a configuration option that limits the number of threads somehow. Scribe for example, has a configuration option to prevent creating a new thread for each category / tag.
The cost does seem to be becoming larger than I initially thought. There are many advantages though. There are a number of use cases that require a high number of tags. Particularly when handling multiple applications. The number of tags in our case could easily exceed 1000 in the near future and could grow larger. We are already at several hundred. The main benefit I see in storing that many tags in separate folders is if you want to perform analytics on a small subset of events you do not have to open every file to search for the events and potentially speed up queries by quite a bit. |
Can we follow the same approach as in this plugin? https://github.com/fluent/fluent-plugin-mongo/blob/master/lib/fluent/plugin/out_mongo.rb#L93 |
Sorry for the late reply.
This is interesting. I will check Scribe source code later. Currently no because S3 plugin already use same approach to separate record with event time. |
@repeatedly I am not seeing s3 plugin is using |
@ryanc4 S3 plugin itself doesn't extend |
@jsermeno I checked Scribe's |
Is there any update for using tags in object key? |
You can use fluent-plugin-forest to realize this goal: https://github.com/tagomoris/fluent-plugin-forest |
please provide example how we can integarte forest plugin with s3 plugin to give dynaminc tag support in path |
is it possible to have tags in the object key?
The text was updated successfully, but these errors were encountered: