[Website]: Improve docs for using Logstash output with Fleet (_id situation)

[lucabelluccini]

Before you submit

• This issue is about a documentation page, flow, or piece of content.

Type of issue

Other

What documentation page or section is affected

https://www.elastic.co/docs/reference/fleet/logstash-output

What happened?

When using the Logstash output for Elastic Agent, in the output we recommend:

output {

  if [@metadata][_id] {
    elasticsearch {
      ...
      document_id => "%{[@metadata][_id]}"
...
    }
  } else {
    elasticsearch {
...
    }
  }

It is necessary because SOME but not all the integrations use _id to avoid duplication.

The drawback is this config duplicates the ES output and can lead to sub-optimal ingestion (due to bulk sizes might vary a lot in size).

A possible optimization for Logstash 8.12+ / or ES Output plugin 11.22.2 would be to leverage the undocumented [@metadata][_ingest_document][id]

The recommended config can become:

filter {
    if [@metadata][_id] {
      mutate { add_field => { "[@metadata][_ingest_document][id]" => "[@metadata][_id]" } }
    } 
}
output {
    elasticsearch {
      ...
    }
}

Additional info

I would like @robbavey to review this.

The field [@metadata][_ingest_document][id] is not an actual public contract, but it might be useful to simplify the ES output.

[github-actions]

Elastic Docs AI Menu

Check one box to run an AI workflow for this issue.

• Triage (docs-triage). Status: not started.
• Size the work (docs-size). Status: not started.
• Scope the docs work (docs-issue-scope). Status: not started.

Powered by GitHub Agentic Workflows and docs-actions. For more information, reach out to the docs team.