[airflow_otel] add new content pack by jakubgalecki0 · Pull Request #17786 · elastic/integrations

jakubgalecki0 · 2026-03-12T16:24:39Z

Proposed commit message

Generated Kibana content for Apache Airflow monitoring via the OTel Airflow receiver: 4 dashboards (overview, tasks, DAG processing, scheduler capacity), 11 alert rules (heartbeat, executor saturation, task failures, scheduling delays, DAG import errors, etc.) and 3 SLO templates (task success rate, scheduler loop duration, DAG import error rate).

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

[ ]

How to test this PR locally

Related issues

Screenshots

github-actions · 2026-03-12T16:25:55Z

Vale Linting Results

Summary: 2 warnings found

⚠️ Warnings (2)

File	Line	Rule	Message
packages/airflow_otel/docs/README.md	3	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'using' instead of 'via'.
packages/airflow_otel/docs/README.md	21	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'using' instead of 'via'.

The Vale linter checks documentation changes against the Elastic Docs style guide.

To use Vale locally or report issues, refer to Elastic style guide for Vale.

strawgate · 2026-03-17T14:17:56Z

I ran kb-dashboard 0.4.0 upgrade validation on this PR in an isolated worktree.

Results

packages/airflow_otel/_dev/shared
- upgrade --fail-on-change exit: 1
- upgrade --write exit: 0
- post-upgrade compile --format elastic-integrations exit: 0

Upgrade and post-upgrade compile succeeded with 0.4.0.
Please apply these commands in your branch and commit both YAML + regenerated JSON artifacts.

Suggested commands:

uvx --from kb-dashboard-cli==0.4.0 kb-dashboard upgrade --input-dir packages/airflow_otel/_dev/shared --write
uvx --from kb-dashboard-cli==0.4.0 kb-dashboard compile --input-dir packages/airflow_otel/_dev/shared --output-dir packages/airflow_otel/kibana/dashboard --format elastic-integrations

mykola-elastic · 2026-04-06T11:54:50Z

packages/airflow_otel/img/airflow_dag_processing.png

Is it possible to load (or even stress test) the airflow a bit with some DAGs so that dashboard has some data, not just "0" and "N/A"? Maybe with the help of Claude/Cursor/...

mykola-elastic · 2026-04-06T11:56:17Z

packages/airflow_otel/changelog.yml

+  changes:
+    - description: initial release
+      type: enhancement
+      link: ""


Suggested change

link: ""

link: "https://github.com/elastic/integrations/pull/17786"

please address this one.

mykola-elastic · 2026-04-06T11:59:07Z

packages/airflow_otel/docs/README.md

+```yaml
+receivers:
+  statsd:
+    endpoint: 0.0.0.0:8125
+    transport: udp
+
+exporters:
+  elasticsearch/otel:
+    endpoints: [<ES_ENDPOINT>]
+    api_key: ${env:ELASTICSEARCH_API_KEY}
+    mapping:
+      mode: otel
+
+service:
+  pipelines:
+    metrics:
+      receivers: [statsd]
+      exporters: [elasticsearch/otel]
+```


not sure if that's an improvement here, but it is possible to change the dataset data stream name like in this example: https://github.com/elastic/integrations/pull/17757/changes#diff-cef028242d40a61d0afab7399295d79d745fc777b027ca8f8227ae94ff2cf930R51-R56

So that we can have something like airflow.otel instead of statsdreceiver.otel which is generic

We should have the service specific data stream name. Please add it via a processor in the config of OTel and also update it to the README.

Updated dataset. Thanks!

muthu-mps · 2026-04-07T07:05:01Z

The data_stream.dataset filter is applied only on the Overview dashboard but appears to be missing from the remaining dashboards. Could you verify?
Tasks & Errors dashboards. Task Duration**(Example DAG)**. Does this require Example? When the panel axis display precise values which goes up to hours. Is it okay to remove the mentions of (ms). WDYT?

packages/airflow_otel/kibana/alerting_rule_template/airflow_otel-dag-processor-timeouts.json

elasticmachine · 2026-04-08T17:39:20Z

💚 Build Succeeded

Buildkite Build
Commit: 9e4421c

History

💚 Build #41071 succeeded 13f843d
💚 Build #41033 succeeded bdc7fb8
💚 Build #40836 succeeded fca086d
💚 Build #39944 succeeded 7f6cfe3
💚 Build #39941 succeeded d100ae3
💚 Build #39937 succeeded e8f54e5

elastic-vault-github-plugin-prod · 2026-04-08T18:59:10Z

Package airflow_otel - 0.1.0 containing this change is available at https://epr.elastic.co/package/airflow_otel/0.1.0/

jakubgalecki0 added 3 commits March 12, 2026 17:12

[airflow_otel] add new content pack

b25905b

update readme

78245bd

add CODEOWNERS

3e91b52

jakubgalecki0 requested a review from a team as a code owner March 12, 2026 16:24

jakubgalecki0 changed the title ~~Airflow otel cp~~ [airflow_otel] add new content pack Mar 12, 2026

andrewkroh added New Integration Issue or pull request for creating a new integration package. dashboard Relates to a Kibana dashboard bug, enhancement, or modification. documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. labels Mar 15, 2026

Merge branch 'main' into airflow_otel_cp

a42c39a

jakubgalecki0 added 4 commits March 19, 2026 10:30

Merge branch 'main' into airflow_otel_cp

e8f54e5

upgrade compiler version

d100ae3

add slo note

7f6cfe3

Merge branch 'main' into airflow_otel_cp

fca086d

mykola-elastic reviewed Apr 6, 2026

View reviewed changes

jakubgalecki0 added 2 commits April 8, 2026 10:17

Merge branch 'main' into airflow_otel_cp

bdc7fb8

update dataset name

4d9256b

macroscopeapp bot reviewed Apr 8, 2026

View reviewed changes

packages/airflow_otel/kibana/alerting_rule_template/airflow_otel-dag-processor-timeouts.json Outdated Show resolved Hide resolved

jakubgalecki0 added 3 commits April 8, 2026 16:08

update screenshots

62af2fa

recompile dashboards

181bf5f

Merge branch 'main' into airflow_otel_cp

13f843d

muthu-mps approved these changes Apr 8, 2026

View reviewed changes

ishleenk17 approved these changes Apr 8, 2026

View reviewed changes

Merge branch 'main' into airflow_otel_cp

9e4421c

jakubgalecki0 merged commit ccd2400 into elastic:main Apr 8, 2026
11 checks passed

andrewkroh added the Integration:airflow_otel Airflow OpenTelemetry label Apr 8, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[airflow_otel] add new content pack#17786

[airflow_otel] add new content pack#17786
jakubgalecki0 merged 14 commits intoelastic:mainfrom
jakubgalecki0:airflow_otel_cp

jakubgalecki0 commented Mar 12, 2026

Uh oh!

github-actions bot commented Mar 12, 2026

Uh oh!

strawgate commented Mar 17, 2026

Uh oh!

mykola-elastic Apr 6, 2026 •

edited

Loading

Uh oh!

mykola-elastic Apr 6, 2026

Uh oh!

muthu-mps Apr 7, 2026

Uh oh!

mykola-elastic Apr 6, 2026

Uh oh!

ishleenk17 Apr 8, 2026

Uh oh!

jakubgalecki0 Apr 8, 2026

Uh oh!

muthu-mps commented Apr 7, 2026

Uh oh!

Uh oh!

elasticmachine commented Apr 8, 2026

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented Apr 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

	link: ""
	link: "https://github.com/elastic/integrations/pull/17786"

Conversation

jakubgalecki0 commented Mar 12, 2026

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

github-actions bot commented Mar 12, 2026

Vale Linting Results

Uh oh!

strawgate commented Mar 17, 2026

Results

Uh oh!

mykola-elastic Apr 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mykola-elastic Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

muthu-mps Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

mykola-elastic Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

ishleenk17 Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

jakubgalecki0 Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

muthu-mps commented Apr 7, 2026

Uh oh!

Uh oh!

elasticmachine commented Apr 8, 2026

💚 Build Succeeded

History

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented Apr 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

mykola-elastic Apr 6, 2026 •

edited

Loading