Output to S3 every half hour (sharp)

[Cisneiros]

Hello.

I'm trying to export log files to S3 every half an hour "sharp", inside a year/month/day directory structure. This means I want something like:

2017/03/02/1100-xalala
2017/03/02/1130-xalala
2017/03/02/1200-xalala
2017/03/02/1230-xalala

(where xalala is the rest of the filename, like hostname, index and extension)

I tried using the following format for my time_slice_format: %Y/%m/%d/%H%M.

This does what I want: creates a folder structure per year/month/day. But, since I have a %M there, it will create a new buffer for each minute (and consequently a new file every minute). I've tried experimenting with time key and flush_interval variations, but so far I haven't been able to achieve what I want.

Is it possible using purely fluentd and the S3 Plugin? If not, what can be a good direction I can take to implement a solution myself?

[repeatedly]

With v0.12, you can't realize it.
But v0.14 supports more flexibile time separation. See following article:

http://docs.fluentd.org/v0.14/articles/buffer-section#time

You can realize it by following configuration:

<match pattern>
  @type s3
  # other parameters
  <buffer time>
    timekey 1800
    timekey_wait 5m # or your best wait time
    # other parameters
  </buffer>
</match>

[Cisneiros]

Thank you! It worked like a charm :)