Find a way to upload dynamic file name to Azure Blob Storage

[YilanBoy]

I want to use fluent-bit to forward the server log to Azure blob storage.

When uploading files to Azure blob storage, fluent-bit uses the input tag as the file name. But I want to use the date as the file name.

After some research, I found a way to achieve my goal.

This is my configuration.

[INPUT]
    name   udp
    listen 0.0.0.0
    port   12201
    format json
    tag    gelf_tag

[FILTER]
    Name   lua
    Match  *
    script /scripts/append_datetime.lua
    call   append_datetime

[FILTER]
    Name          rewrite_tag
    Match         gelf_tag
    Rule          $datetime .* $datetime false
    Emitter_Name  re_emitted

[OUTPUT]
    name                  azure_blob
    match                 *
    account_name          account_name
    shared_key            secret_key
    path                  gelf
    container_name        logs
    auto_create_container on
    tls                   on

Lua script

function append_datetime(tag, timestamp, record)
    new_record = record
    new_record["datetime"] = os.date("%Y-%m-%d-%H-%M-%S")
    return 1, timestamp, new_record
end

I used two filters to achieve my goal.

First, I use the lua filter to get the datetime and append it in the log, then I use the rewrite_tag filter to get the datetime out of the log and use it to overwrite the original tag.

Just wanted to share this briefly, but if there is a better way to do it, feel free to let me know! 😆