[Flaky Test]: `TestKubernetesJournaldInputOtel` – `Condition never satisfied`

[ebeahan]

Failing test case

TestKubernetesJournaldInputOtel

Error message

common.go:42: got an error querying ES, retrying. Error: error in HTTP query: non-200 return code: 404, response: '[404 Not Found]'

Build

https://buildkite.com/elastic/elastic-agent-extended-testing/builds/11183#019a933d-f87e-4939-8038-aa49523e6054

OS

Linux

Stacktrace and notes

FAIL
FAIL	github.com/elastic/elastic-agent/testing/integration/k8s	394.299s
=== Failed
=== FAIL: testing/integration/k8s TestKubernetesJournaldInput (226.53s)
    common.go:42: got an error querying ES, retrying. Error: error in HTTP query: non-200 return code: 404, response: '[404 Not Found] {
          "error" : {
            "root_cause" : [
              {
                "type" : "index_not_found_exception",
                "reason" : "no such index [logs-generic-bc9lynlzgcbhh0if0cuw1w9rpzmeen9d59naaq1w8]",
                "resource.type" : "index_or_alias",
                "resource.id" : "logs-generic-bc9lynlzgcbhh0if0cuw1w9rpzmeen9d59naaq1w8",
                "index_uuid" : "_na_",
                "index" : "logs-generic-bc9lynlzgcbhh0if0cuw1w9rpzmeen9d59naaq1w8"
              }
            ],
            "type" : "index_not_found_exception",
            "reason" : "no such index [logs-generic-bc9lynlzgcbhh0if0cuw1w9rpzmeen9d59naaq1w8]",
            "resource.type" : "index_or_alias",
            "resource.id" : "logs-generic-bc9lynlzgcbhh0if0cuw1w9rpzmeen9d59naaq1w8",
            "index_uuid" : "_na_",
            "index" : "logs-generic-bc9lynlzgcbhh0if0cuw1w9rpzmeen9d59naaq1w8"
          },
          "status" : 404
        }

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[ebeahan]

Recently seen on the k8s integration tests for the following versions:

• v.1.27.16
• v1.28.15
• v1.29.14
• v1.30.0
• v1.31.0
• v1.32.0
• v1.33.0
• v1.34.0

Seems to consistently succeed on the retry.

[ebeahan]

Earliest instances I came across are from 2025-11-12: https://buildkite.com/elastic/elastic-agent-extended-testing/builds/11110

[pierrehilbert]

Assigning to @belimawr as he wrote this test.

[belimawr]

I had a very quick look at the logs, for some reason (likely external to the test) Elasticsearch is rejecting the events.
Here are some logs I found:
Normal Elastic Agent

Failed to index 356 events in last 10s: events were dropped! Look at the event log to view the event and cause.

OTel:

2025-11-17T20:41:38.264Z	error	elasticsearchexporter@v0.137.0/bulkindexer.go:540	failed to index document	{"resource": {"service.instance.id": "83cc2b76-dcbd-4fa8-98d4-d3ce5d4ae8ef", "service.name": "elastic-agent", "service.version": "9.2.2"}, "otelcol.component.id": "elasticsearch", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "index": "logs-generic.otel-dz5cdyu6uqp1bjgc9kk7umsc6tuzpdyjeyrvbsyou", "error.type": "validation_exception", "error.reason": ""}

[swiatekm]

We could start by enabling event logging in this test, and see what it turns up.

[belimawr]

We could start by enabling event logging in this test, and see what it turns up.

Yeah, I ran this a couple of times locally and I cannot reproduce the failure 😢 .

I suspect the problem is the same or similar to: #7222 (comment), with the difference that ES is rejecting the events instead of putting them into the failure store.

[belimawr]

I cannot reproduce the mapping conflicts :/, So I did my best to avoid them based on what I saw in other tests and made the OTel logs JSON at debug level so we get the Beats published events logs and can easily upload the logs to Elasticsearch.

PR: #11256

[ebeahan]

Should we keep this open? sounds like we tried to make improvements but not entirely resolved?

I'm seeing another occurrence with #11256 merged: https://buildkite.com/elastic/elastic-agent-extended-testing/builds/11232#019aa1cd-c1ac-4190-88ea-1af047197dab