[DJM-812] Airflow task logs child pipeline

[aboitreaud]

What does this PR do?

This PR updates the Airflow integration logs pipeline to extract dag_id, run_id, and task_id from the dirname tag. Extraction is performed in a children subpipeline, which query checks for the presence of dag_id in dirname tag so that it's guaranteed that pipeline will find the attributes to be extracted.

Motivation

Airflow DJM traces to logs correlation is made on dag_id, run_id, and task_id attributes so they need to be present on the logs. See correlation logic here.

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• Add the qa/skip-qa label if the PR doesn't need to be tested during QA.
• If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

Review from baturalp-dd is dismissed. Related teams and files:

• logs-backend
  • airflow/assets/logs/airflow.yaml
• logs-integrations-reviewers
  • airflow/assets/logs/airflow.yaml

[aboitreaud]

👋 Thanks for the review !
Here are some screenshots of a log that went through the newly added subpipeline and of the pipeline itself.

I just flipped the Preserve source attribute and Override on conflict bools to true to make the attribute has the correct value and we also keep the origin tag just in case.

[aboitreaud]

Hi @baturalp-dd, would you mind taking a look again please? I'm sorry I dismissed your review with the changes I made after checking the deployed pipeline on staging.

[aboitreaud]

Hi @baturalp-dd , gentle bump on this 🙇

Review from baturalp-dd is dismissed. Related teams and files:

• logs-backend
  • airflow/assets/logs/airflow.yaml
  • airflow/assets/logs/airflow_tests.yaml
• logs-integrations-reviewers
  • airflow/assets/logs/airflow.yaml
  • airflow/assets/logs/airflow_tests.yaml

[aboitreaud]

/merge

[dd-devflow]

View all feedbacks in Devflow UI.

2025-06-16 17:18:08 UTC ℹ️ Start processing command /merge

---

2025-06-16 17:18:14 UTC ℹ️ MergeQueue: waiting for PR to be ready

This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

---

2025-06-16 17:28:53 UTC ⚠️ MergeQueue: This merge request was unqueued

adrien.boitreaud@datadoghq.com unqueued this merge request

[aboitreaud]

/merge -c

[dd-devflow]

View all feedbacks in Devflow UI.

2025-06-16 17:28:44 UTC ℹ️ Start processing command /merge -c