Set Ignore_Older to 24h by default to prevent older logs from being ingested on install.

[frankreno]

Description

Fixes #596.

Set Ignore_Older to 24h by default to prevent older logs from being ingested when collection is first installed. Relative times are supported. Tested using shorter time range (1m) and validated that the older logs are not ingested.

fluent/fluent-bit#221
https://docs.fluentbit.io/manual/pipeline/inputs/tail#config

Note, this is not supported on systemd plugin so only updating tail plugin.

Fill in your description here.

Testing performed

• ci/build.sh
• Redeploy fluentd and fluentd-events pods
• Confirm events, logs, and metrics are coming in

[perk-sumo]

LGTM 👍

[avdhoot]

@frankreno I doubt it will work(not working for me). To Ignore_Older work input plugin need parser. more fluent/fluent-bit#716

[frankreno]

@avdhoot - we use a parser on our input plugin for container logs. Local testing showed this working but definitely worth another check. Can you share your configuration? Are you seeing older logs ingested for container logs or systemd/kubelet?

[avdhoot]

Non working config:

    [INPUT]
        Name             tail
        Path             /var/log/containers/*.log
        Multiline        On
        Parser_Firstline multi_line
        Tag              containers.*
        Refresh_Interval 1
        Rotate_Wait      60
        Mem_Buf_Limit    5MB
        Skip_Long_Lines  On
        DB               /tail-db/tail-containers-state-sumo.db
        DB.Sync          Normal
        Ignore_Older 1m

Working config. To it work parser should extract time.

[INPUT]
    Name             tail
    Path             /var/log/containers/*.log
    Multiline        Off
    Parser_Firstline multi_line
    Tag              containers.*
    Refresh_Interval 1
    Rotate_Wait      60
    Mem_Buf_Limit    5MB
    Skip_Long_Lines  On
    DB               /tail-db/tail-containers-state-sumo.db
    DB.Sync          Normal
    Ignore_Older 1m
    Parser  Docker

[avdhoot]

@frankreno in case you missed it. Result of above config. There is still some leak but it is manageable.

[frankreno]

Thank you @avdhoot - we will investigate and report back. In that search are you looking at all logs sent to the collector? You should exclude any logs from systemd because there is a known issue that we ingest old logs there. The systemd does not support a way to ignore older log files, so I just want to make sure the leak you see above is not from those logs.

[avdhoot]

To make things more clear. I have used config mentioned in #664 (comment). Currently we are not sending systemd logs. @frankreno

[Adhira-Deogade]

@avdhoot With your configs here, I am still collecting historical logs. Am I doing something wrong? Thanks.