[Filebeat] Duplicated data when using filestream input

[belimawr]

When there is more than one filestream input with the same ID (or without an ID) data duplication will happen when Filebeat is restarted or a new input with duplicated ID is started/created.

The root cause is related to the clean up of old entries in the registry.

Possible ways this bug manifests

• Kubernetes integration (under Elastic-Agent/Fleet), when new pods are added old files are read from the beginning.
• Kubernetes autodiscover, when new pods are added old files are read from the beginning. (confirmed by #24208 (comment))
• Filestream inputs without IDs on a standalone Filebeat, when Filebeat restarts it reads the files from the beginning.
• Filestream inputs with duplicated IDs on a standalone Filebeat, when Filebeat restarts it reads the files from the beginning.

How to detect the issue

The latest (v7.17.2 and v8.1.2) versions of Filebeat will issue a log error when this situation is detected:

filestream input with ID 'xxxxxx' already exists, this will lead to data duplication, please use a different ID

For multiple filestream inputs without IDs the message looks like:

filestream input ID without ID might lead to data  duplication, please add an ID and restart Filebeat

Workaround

Set unique IDs to every filestream input for standalone Filebeat and standalone Elastic-Agent

Related issues

• [Fleet] Duplicate stream id with kubernetes provider kibana#129851
• filebeat Duplicated logs at beginning with kubernetes autodiscovery #24208 confirmed by #24208 (comment))
• IDs are required for filestream inputs on Filebeat monitoring elastic-agent#323
• Fix input ID uniqueness #31512

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[bczifra]

@belimawr based on my reading of elastic/kibana#129851 (comment) it appears that

Set clean_removed: false to every filestream input that does not have an ID or share the same ID

is not a recommended workaround. Is my understanding correct?

[belimawr]

@belimawr based on my reading of elastic/kibana#129851 (comment) it appears that

Set clean_removed: false to every filestream input that does not have an ID or share the same ID

is not a recommended workaround. Is my understanding correct?

Indeed. Thanks for catching this. I'll update the issue.

[jlind23]

@belimawr closing this one in favour of #31512

[belimawr]

@belimawr closing this one in favour of #31512

I'd have kept it open as a "meta issue", but it's ok. Having multiple issues open about the same bug can be confusing.

[irislanderos]

Hi Team,

Cloud AR Operations here. We are working to give credits back this customer ( Org-2878897275, Ubrich ) for the increase in usage due to this technical issue. I was provided with a 1:4 to 1:5 ratio of impact.

1. Is this the closest estimation we have as to the impact? Do these impact the capacity costs as well as DTS? Or just one or the other?

Also, I am aware that the cluster is still bloated -
2. Do we know by about how much so I can calculate the impact of that?

So far I have calculations based on the 1:4 & 1:5 ratio estimates, as well an estimate of credits based on their July 2022 usage.
The results of the ratio calculations and the July 2022 usage vary greatly so I'm stuck on how to proceed with the credit.
Any direction to get to the right place would be appreciated!

Working Calc Doc:
https://docs.google.com/spreadsheets/d/1CFL3lOjUv5AtXalZq6dQ_92AB9Y7T0x_y3s0C5dGyV4/edit#gid=1575888295

Related case: 00929762

@belimawr @bczifra @jlind23

[jlind23]

@irislanderos it is almost impossible to calculate the right ratio. As soon as you have an input without an ID specified, data will be duplicated. If you have X inputs, data may be duplciated X time. Thus having an accurate ratio is clearly impossible. @belimawr thoughts?

[belimawr]

@irislanderos it is almost impossible to calculate the right ratio. As soon as you have an input without an ID specified, data will be duplicated. If you have X inputs, data may be duplciated X time. Thus having an accurate ratio is clearly impossible. @belimawr thoughts?

I agree with @jlind23. It's not possible to have an exact estimation of the data duplication ratio. One thing that plays a role is the number of times Filebeat is restarted. Most of the steps that lead to this bug happen synchronously and the time when they happen will impact the amount of data duplicated.

[irislanderos]

Thank you both for the insight. We'll find another way to resolve this. @jlind23 @belimawr

[spalan479]

Hi All,

Duplicate data is sent from Filebeat.
When I start the elastic agent, I can see two filebeat processes running. If I Kill the first one it auto restarts and the second one disappears when I kill it twice. Once the second process is killed, there are no duplicate data coming in.
Am I missing any configuration here?

Filebeat.yml

============================== Filebeat inputs ===============================

filebeat.inputs:

• type: filestream
  enabled: true
  id: "ecs-app-logging"
  paths:
  • C:\MVD2\Logs\ECS*.log
    json.keys_under_root: true
    json.overwrite_keys: true
    json.add_error_key: true
    json.expand_keys: true

[cmacknz]

The agent will use a second Filebeat process to ingest logs when log monitoring is enabled. Having two Filebeat processes running can be normal and does not necessarily cause data duplication. The processes started by the agent are controlled through the agent policy, and not the Filebeat configuration file.

Please start a thread on https://discuss.elastic.co/tag/elastic-agent and someone will help you determine if this is a configuration issue or a bug in the agent. If it is a bug we will open a new issue to track it here.