[otel_collector_internal_telemetry]: ESQL errors without otelcol_exporter_send_failed_spans

[gregkalapos]

Integration Name

OpenTelemetry Collector Internal Telemetry [packages/otel_collector_internal_telemetry]

Dataset Name

No response

Integration Version

doesn't matter

Agent Version

doesn't matter

Agent Output Type

elasticsearch

Elasticsearch Version

doesn't matter

OS Version and Architecture

doesn't matter

Software/API Version

No response

Error Message

Some of the dashboards show an error - specifically when otelcol_exporter_send_failed_spans isn't yet present.

SET unmapped_fields="nullify"; would solve these in the queries.

Report and suggestion by @ty-elastic

Event Original

No response

What did you do?

Add integration and not have all the internal telemetry ingested yet - meaning some of the metrics are not (yet) present.

What did you see?

Errors on some of the dashboards.

What did you expect to see?

No error - empty charts are fine (as metrics may not be present), but not an error.

Anything else?

No response

[github-actions]

tl;dr: this looks like a real query-hardening bug, not just a docs issue — otel_collector_internal_telemetry contains multiple ES|QL queries that directly reference otelcol_exporter_send_failed_spans, and these can fail when that metric is not mapped/emitted.

Recommendation

Harden the affected ES|QL queries to tolerate missing otelcol_exporter_send_failed_spans (especially in otel_collector_internal_telemetry-58ba1893-9b1b-46e7-8a32-0765a5dd120a.json) and document that span-failure visualizations/alerts require span exporter telemetry. Prefer one canonical field style across this package (metrics.<field> vs <field>), then apply the same pattern everywhere that uses send_failed_spans.

Findings

1. Direct references exist in dashboards and alert templates with no guard for absent field:

   • packages/otel_collector_internal_telemetry/kibana/dashboard/otel_collector_internal_telemetry-58ba1893-9b1b-46e7-8a32-0765a5dd120a.json:1817
   • packages/otel_collector_internal_telemetry/kibana/dashboard/otel_collector_internal_telemetry-58ba1893-9b1b-46e7-8a32-0765a5dd120a.json:1971
   • packages/otel_collector_internal_telemetry/kibana/dashboard/otel_collector_internal_telemetry-58ba1893-9b1b-46e7-8a32-0765a5dd120a.json:1981
   • packages/otel_collector_internal_telemetry/kibana/dashboard/otel_collector_internal_telemetry-58ba1893-9b1b-46e7-8a32-0765a5dd120a.json:2002
   • packages/otel_collector_internal_telemetry/kibana/alerting_rule_template/otel-collector-internal-telemetry-otel-exporter-send-failed-spans.json:21

2. The same dashboard mixes field styles (metrics.otelcol_exporter_send_failed_spans and otelcol_exporter_send_failed_spans) in separate panels/queries, which increases fragility:

   • ...58ba1893...json:1817 uses metrics.otelcol_exporter_send_failed_spans
   • ...58ba1893...json:1971 uses unqualified otelcol_exporter_send_failed_spans

3. Recent related history:

   • PR [otel_collector_internal_telemetry] Add alerting rules #18048 introduced send-failed-spans alerting; current file history points to commit 3b49282585 for otel-collector-internal-telemetry-otel-exporter-send-failed-spans.json.
   • PR [otel_collector_internal_telemetry] Document data_stream.dataset and fix queries for TBS dashboards #19049 and [otel_collector_internal_telemetry] Clarify that prometheus format is not supported #19254 addressed other telemetry query/docs mismatches, indicating this package has had query-assumption issues recently.

4. I could not read the full body of issue [otel_collector_internal_telemetry]: ESQL errors without otelcol_exporter_send_failed_spans #19404 via MCP because it was integrity-filtered in this runtime, so findings above are based on repository evidence and issue title.

Verification

I validated references and history from the workspace:

$ git --no-pager log --oneline -- packages/otel_collector_internal_telemetry/kibana/alerting_rule_template/otel-collector-internal-telemetry-otel-exporter-send-failed-spans.json | head -n 5
3b49282585 [otel_collector_internal_telemetry] Add alerting rules (#18048)

$ grep -RIn "otelcol_exporter_send_failed_spans" packages/otel_collector_internal_telemetry | head -n 40
packages/otel_collector_internal_telemetry/kibana/alerting_rule_template/otel-collector-internal-telemetry-otel-exporter-send-failed-spans.json:21: ... SUM(RATE(metrics.otelcol_exporter_send_failed_spans)) ...
packages/otel_collector_internal_telemetry/kibana/dashboard/otel_collector_internal_telemetry-58ba1893-9b1b-46e7-8a32-0765a5dd120a.json:1817: ... metrics.otelcol_exporter_send_failed_spans ...
packages/otel_collector_internal_telemetry/kibana/dashboard/otel_collector_internal_telemetry-58ba1893-9b1b-46e7-8a32-0765a5dd120a.json:1971: ... otelcol_exporter_send_failed_spans ...
packages/otel_collector_internal_telemetry/kibana/dashboard/otel_collector_internal_telemetry-58ba1893-9b1b-46e7-8a32-0765a5dd120a.json:1981: ... otelcol_exporter_send_failed_spans ...
packages/otel_collector_internal_telemetry/kibana/dashboard/otel_collector_internal_telemetry-58ba1893-9b1b-46e7-8a32-0765a5dd120a.json:2002: ... otelcol_exporter_send_failed_spans ...

Detailed Action Plan

1. In packages/otel_collector_internal_telemetry/kibana/dashboard/otel_collector_internal_telemetry-58ba1893-9b1b-46e7-8a32-0765a5dd120a.json:

   1. Update the send-failed-spans panels at lines 1817, 1827, 1908, 1971, 1981, 2002 to use one canonical field expression.
   2. Add a missing-field-safe pattern before RATE()/INCREASE() so queries do not fail when spans metric is absent.
   3. Keep current grouping/filtering (service.instance.id, exporter filters) unchanged.

2. In packages/otel_collector_internal_telemetry/kibana/alerting_rule_template/otel-collector-internal-telemetry-otel-exporter-send-failed-spans.json:21:

   1. Apply the same resilient field handling.
   2. Ensure the alert returns empty/no-op instead of query failure when the field is missing.

3. In packages/otel_collector_internal_telemetry/docs/README.md (near lines 13-25):

   1. Add explicit note that span-failure charts/alerts depend on otelcol_exporter_send_failed_spans being emitted.
   2. Point users to disable/ignore that panel/alert when trace exporter telemetry is not present.

4. Validate by loading the updated dashboards against a dataset that omits span-export metrics and confirming no ES|QL compile/runtime error.

Related Items

Type	Link	Relevance
Issue	#19404	Report being triaged
PR	#18048	Introduced send-failed alert templates
PR	#19049	Prior query fix in same package
PR	#19254	Prior docs clarification in same package
File	packages/otel_collector_internal_telemetry/kibana/dashboard/otel_collector_internal_telemetry-58ba1893-9b1b-46e7-8a32-0765a5dd120a.json	Main dashboard with mixed direct field references
File	packages/otel_collector_internal_telemetry/kibana/alerting_rule_template/otel-collector-internal-telemetry-otel-exporter-send-failed-spans.json	Alert query with direct field reference
File	packages/otel_collector_internal_telemetry/docs/README.md	User-facing prerequisites and assumptions

Note

🔒 Integrity filter blocked 2 items

The following items were blocked because they don't meet the GitHub integrity level.

• #19404 issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #19404 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

---

What is this? | From workflow: Issue Triage

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.