Ability to specify an index

[andrewmyhre]

In a kubernetes cluster it would be useful if fluent-bit were to specify an index with log requests. In a fluentd configuration we would have the following:

<match **>
    type elasticsearch_dynamic
    log_level info
    include_tag_key true
    host elasticsearch-logging
    port 9200
    logstash_format true
    logstash_prefix logstash-${record['kubernetes']['namespace_name']}
    reload_connections false
    # Set the chunk limit the same as for fluentd-gcp.
    buffer_chunk_limit 2M

I'm not sure but the the appropriate configuration setting might actually be target_index rather than logstash_prefix.
The desired end result is that logs are shipped with _index including the name of the kubernetes namespace.
It would be great if fluent-bit allowed configuring this.

[mvladev]

I think it's the same enhancement as in #242, but for host field.

[brycefisher]

I would also find this feature extremely useful!

[max-lobur]

Can fluent-bit even do logstash_prefix ${record['kubernetes']['namespace_name']}?
I can't find.
I would be very happy If I can set logstash_prefix to k8s namespace at least, I can then set index pattern to <namespace>-* and live with prefix as usually

[kfox1111]

Not quite the same thing, but might work...
Set fluent-bit to point at a fluentd cluster in k8s. Then it looks like the fluentd es plugin might be able to specify an index based on tag:
https://docs.fluentd.org/v1.0/articles/out_elasticsearch

This also separates your log shipping from your log processing so you can scale each out separately.

(If you try this soon, please let me know if it works for you. I haven't tried it yet and am curious if it works)

[zeph]

👍

[dhananjaysathe]

Ok this feature looks like it landed in master when i look at the code https://github.com/fluent/fluent-bit/blob/master/plugins/out_es/es.c#L265
Here is the commit / MR : 036c839#diff-197d21cf7c333bb3fc476eeb1e5a7f83

[anand-nair]

Hi guys,
Checking if this is possible in fluent-bit. Looking for creating indexes in Elasticsearch for each namespace in Kubernetes similar to elasticsearch_dynamic.
Tx

[edsiper]

Elasticsearch output plugin has been updated for logstash_prefix_key option. Changes:

    out_es: add support for record accessor on 'logstash_prefix_key' (#1670)
    
    The following patch adds 'record accessor' support for the
    'logstash_prefix_key' configuration property. This enable to have
    custom indices based on nested fields if required, e.g;
    
     {"key1": 1234, "kubernetes": {"labels: {"app": "prod"}}}
    
    You can configure a logstash prefix as follows:
    
     [OUTPUT]
         name                 es
         match                *
         logstash_format      on
         logstash_prefix_key  $kubernetes['labels']['app']
    
    The final index for the record above and the proposed config will generate:
    
     prod-2021.01.24

[offero]

@edsiper What version?

[edsiper]

It will be part of Fluent Bit v1.7 release (1 or 2 weeks from now)

[64bit]

It seems to be not working on fluentbit v1.7.7. Is this feature actually relased?

The documentation of v1.7 also suggest its not.

How do we make it work as originally posted in description and based on last comment something like logstash_prefix logstash-$kubernetes['namespace_name']} ?

thank you.

[qmonitoring]

It seems to be not working on fluentbit v1.7.7. Is this feature actually relased?

The documentation of v1.7 also suggest its not.

How do we make it work as originally posted in description and based on last comment something like logstash_prefix logstash-$kubernetes['namespace_name']} ?

thank you.

Please see Make Logstash_Prefix_Key more dynamic #3624

I hope that this functionality will be implemented in the near future.

[ckgmg]

with fluent-bit 1.8

Logstash_Prefix_Key $kubernetes['namespace_name']

Using the above configuration, in the es can be normal index according to the namespace

[XI3423-aarora]

shall we have any way to separate logs b/w indexes without doing any change in fluentbit:-configmap(input)??