Added includePipelineObservability flag

[NadezhdaNovotortseva]

Describe your changes:

Added includePipelineObservability configuration flag to allow users to disable pipeline observability extraction during metadata ingestion.

Pipeline observability has a significant performance impact on Airflow ingestion — it queries DagRun history, processes lineage artifacts, and caches observability data for each pipeline. Disabling it when not needed can substantially reduce ingestion time for large Airflow deployments with many DAGs and runs.

Type of change:

• Bug fix
• Improvement
• New feature
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation

Checklist:

• I have read the CONTRIBUTING document.
• My PR title is Fixes <issue-number>: <short explanation>
• My PR is linked to a GitHub issue via Fixes #<issue-number> above.
• I have commented on my code, particularly in hard-to-understand areas.
• For JSON Schema changes: I updated the migration scripts or explained why it is not needed.
• For UI changes: I attached a screen recording and/or screenshots above.
• I have added tests (unit / integration / Playwright as applicable) and listed them above.

Greptile Summary

Adds an includePipelineObservability boolean flag (default true) to PipelineServiceMetadataPipeline that lets users skip pipeline observability extraction during metadata ingestion, addressing performance concerns for large Airflow and dbt Cloud deployments.

• Airflow & dbt Cloud connectors: observability context population and cache building are now gated on source_config.includePipelineObservability; get_table_pipeline_observability returns immediately (no yields) when the flag is false.
• JSON schema + generated TypeScript + UI locale: schema adds the new field with "default": true, ensuring backward compatibility; the TypeScript generated type and locale documentation are updated accordingly.
• Tests: new unit tests for both connectors verify that the method yields nothing when disabled and proceeds normally when enabled.

Confidence Score: 5/5

Safe to merge — the new flag defaults to true so existing deployments are unaffected, and both connectors correctly skip all observability work when set to false.

The change is a straightforward opt-out flag added symmetrically to both the Airflow and dbt Cloud connectors. The JSON schema default of true preserves backward compatibility, the Python generator early-return pattern is correct, and the new tests directly verify both the disabled and enabled code paths.

No files require special attention.

Important Files Changed

Filename	Overview
ingestion/src/metadata/ingestion/source/pipeline/airflow/metadata.py	Observability context initialization and cache population moved inside if includePipelineObservability: guard; get_table_pipeline_observability early-returns correctly when disabled.
ingestion/src/metadata/ingestion/source/pipeline/dbtcloud/metadata.py	Same guard pattern applied; pipeline_fqn context assignment correctly left outside the guard; cache_key = None in the else branch is defensive but harmless since it is only consumed inside the same flag guard.
openmetadata-spec/src/main/resources/json/schema/metadataIngestion/pipelineServiceMetadataPipeline.json	New includePipelineObservability field added with "default": true and a clear positive-framing description; backward-compatible.
openmetadata-ui/src/main/resources/ui/src/generated/metadataIngestion/pipelineServiceMetadataPipeline.ts	Generated TypeScript type updated with the new optional boolean field; JSDoc wording follows the same short-form pattern used by sibling fields such as includeLineage.
ingestion/tests/unit/airflow/test_airflow_metadata.py	New TestGetTablePipelineObservability class with two tests: one confirming zero yields when disabled, one confirming a single empty map is yielded when enabled with no context data.
ingestion/tests/unit/topology/pipeline/test_dbtcloud.py	New test_include_pipeline_observability_disabled test correctly asserts zero results and restores the original flag value via try/finally.
ingestion/tests/unit/test_workflow_parse.py	Existing workflow-parse test updated to explicitly include includePipelineObservability: false in the pipeline metadata config fixture.
openmetadata-ui/src/main/resources/ui/public/locales/en-US/Pipeline/workflows/metadata.md	New $$section block added for includePipelineObservability in the correct position between Include Lineage and Enable Debug Logs.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[yield_pipeline_details] --> B[_yield_pipeline_lineage]
    B --> C{includePipelineObservability?}
    C -- true --> D[Initialize context\ncurrent_pipeline_entity\ncurrent_table_fqns\ncurrent_dag_runs]
    C -- true --> E[Populate observability_cache\nper dag_run]
    C -- false --> F[Skip context init\nSkip cache population]
    B --> G[get_table_pipeline_observability]
    G --> H{includePipelineObservability?}
    H -- false --> I[return early\nyield nothing]
    H -- true --> J[Read context + cache]
    J --> K[Build PipelineObservability\nper table_fqn]
    K --> L[yield table_pipeline_map]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[yield_pipeline_details] --> B[_yield_pipeline_lineage]
    B --> C{includePipelineObservability?}
    C -- true --> D[Initialize context\ncurrent_pipeline_entity\ncurrent_table_fqns\ncurrent_dag_runs]
    C -- true --> E[Populate observability_cache\nper dag_run]
    C -- false --> F[Skip context init\nSkip cache population]
    B --> G[get_table_pipeline_observability]
    G --> H{includePipelineObservability?}
    H -- false --> I[return early\nyield nothing]
    H -- true --> J[Read context + cache]
    J --> K[Build PipelineObservability\nper table_fqn]
    K --> L[yield table_pipeline_map]

Loading

_{Reviews (4): Last reviewed commit: "Merge branch 'feature-include-pipeline-o..." | Re-trigger Greptile}

[github-actions]

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

• No GitHub issue is linked. Link an issue in the Development section of the PR (or add Fixes #12345 to the description). For a same-org cross-repo issue, add Fixes open-metadata/<repo>#123 to the description.

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

[github-actions]

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

[github-actions]

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

[gitar-bot]

Code Review ✅ Approved 1 resolved / 1 findings

Introduces an includePipelineObservability configuration flag to optimize metadata ingestion performance by skipping expensive observability tasks. The PR also addresses noisy debug logging and includes all necessary schema and UI updates.

✅ 1 resolved

✅ Quality: Per-pipeline info log when observability disabled is noisy

📄 ingestion/src/metadata/ingestion/source/pipeline/airflow/metadata.py:951-953 📄 ingestion/src/metadata/ingestion/source/pipeline/dbtcloud/metadata.py:417-419
In both get_table_pipeline_observability guard clauses (airflow/metadata.py:951-953 and dbtcloud/metadata.py:417-419), the disabled path logs at logger.info level. Since this method is invoked once per pipeline/DAG, on the large Airflow/dbt deployments this feature is designed for (many DAGs), this will emit the same 'Pipeline observability extraction is disabled via configuration' message for every pipeline, cluttering ingestion logs. Consider logging at logger.debug level, or logging once at initialization when the flag is False.

Options

Display: compact → Showing less information.

Comment with these commands to change:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Gitar}

[github-actions]

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

[github-actions]

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

[gitar-bot]

Code Review ✅ Approved 1 resolved / 1 findings

Introduces an includePipelineObservability configuration flag to optimize metadata ingestion performance by skipping expensive observability tasks. The PR also addresses noisy debug logging and includes all necessary schema and UI updates.

✅ 1 resolved

✅ Quality: Per-pipeline info log when observability disabled is noisy

📄 ingestion/src/metadata/ingestion/source/pipeline/airflow/metadata.py:951-953 📄 ingestion/src/metadata/ingestion/source/pipeline/dbtcloud/metadata.py:417-419
In both get_table_pipeline_observability guard clauses (airflow/metadata.py:951-953 and dbtcloud/metadata.py:417-419), the disabled path logs at logger.info level. Since this method is invoked once per pipeline/DAG, on the large Airflow/dbt deployments this feature is designed for (many DAGs), this will emit the same 'Pipeline observability extraction is disabled via configuration' message for every pipeline, cluttering ingestion logs. Consider logging at logger.debug level, or logging once at initialization when the flag is False.

Options

Display: compact → Showing less information.

Comment with these commands to change:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Gitar}