Fixes #21953, #23338, #27380: upgrade collate-sqllineage to >=2.1.1 with regression tests

[mohittilala]

Describe your changes:

Fixes #21953
Fixes #23338
Fixes #27380

Upgrades collate-sqllineage minimum version from >=2.0.2 to >=2.1.1 (release) and validates the release with an expanded unit test suite.

Parser fixes unlocked by 2.1.1:

• CTE column lineage now works correctly — test_populate_column_lineage_map_ctes was previously skipped because SqlGlot failed to propagate column lineage through CTEs. Skip removed.
• SqlFluff no longer returns empty source tables for ClickHouse CTAS with CTEs — parser skip removed from test_clickhouse_create_table_with_ctes.

New regression tests (18 → 28):

• test_ctas_union_all_inside_cte_column_lineage — CTAS where a CTE wraps a UNION ALL; column lineage flows from both union branches to the write target
• test_clickhouse_ctas_engine_union_all_not_in — ClickHouse CTAS with ENGINE =, CTEs, UNION ALL, and NOT IN subquery (dbt + Clickhouse incorrect process column lineage for the final model #21953)
• test_bigquery_clone_table_with_digit_starting_name — BigQuery CLONE where the source identifier starts with a digit (GCP BQ Data Lineage doesn't work if tables cloned between different dataset. #23338)
• test_snowflake_copy_into_table_from_stage — COPY INTO table from @stage
• test_snowflake_copy_into_stage_from_table — COPY INTO @stage from table
• test_snowflake_copy_into_stage_from_select — COPY INTO @stage from SELECT subquery
• test_snowflake_copy_into_fully_qualified_stage — COPY INTO with fully-qualified stage path
• test_snowflake_copy_into_table_with_column_list_from_stage_subquery — COPY INTO with explicit column list and $1:field positional syntax (Snowflake Stage → Table lineage not generated for certain COPY INTO patterns #27380)
• test_snowflake_copy_into_stage_subpath_with_external_file_format — stage subpath with external named FILE_FORMAT (Snowflake Stage → Table lineage not generated for certain COPY INTO patterns #27380)
• test_snowflake_copy_into_stage_subpath_date_partitioned — date-partitioned stage subpath (/YYYY/MM/DD/file.csv) stripped to stage root (Snowflake Stage → Table lineage not generated for certain COPY INTO patterns #27380)

Skip cleanup:

• test_complex_postgres_view — replaced broad @pytest.mark.skip with targeted test_sqlfluff=False; SqlGlot and SqlParse extract correct column lineage, only SqlFluff is intermittently flaky on deeply nested UNION ALL (~5% of runs)
• test_postgres_copy_with_jsonb_to_target — removed test_sqlglot=False and test_sqlparse=False; all 3 parsers now handle COPY FROM correctly
• Updated skip comments throughout to describe the actual root cause of each failure

Type of change:

• Bug fix
• Improvement
• New feature
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation

Checklist:

• I have read the CONTRIBUTING document.
• My PR title is Fixes <issue-number>: <short explanation>
• I have commented on my code, particularly in hard-to-understand areas.
• For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

• I have added tests around the new logic.
• For connector/ingestion changes: I updated the documentation.

[Copilot]

Pull request overview

Upgrades the ingestion SQL lineage dependency (collate-sqllineage) to a newer minimum version and expands the unit test suite to validate newly-fixed lineage parsing behaviors across dialects/parsers.

Changes:

• Bump collate-sqllineage minimum version from >=2.0.2 to >=2.1.1.
• Unskip/adjust existing lineage tests where 2.1.1 fixes parser behavior (notably CTE column lineage and ClickHouse CTAS+CTEs).
• Add new regression tests covering ClickHouse CTAS patterns, BigQuery CLONE with digit-starting identifiers, and additional Snowflake COPY INTO stage/table patterns.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
ingestion/setup.py	Updates base dependency to require collate-sqllineage>=2.1.1.
ingestion/tests/unit/lineage/test_sql_lineage.py	Removes the skip on the CTE column-lineage test now that the parser behavior is fixed.
ingestion/tests/unit/lineage/queries/test_specific_dialect_queries.py	Cleans up skip annotations and adds multiple new cross-parser regression tests for dialect-specific lineage.

[github-actions]

🟡 Playwright Results — all passed (17 flaky)

✅ 3664 passed · ❌ 0 failed · 🟡 17 flaky · ⏭️ 89 skipped

Shard	Passed	Failed	Flaky	Skipped
🟡 Shard 1	476	0	4	4
🟡 Shard 2	649	0	2	7
🟡 Shard 3	651	0	5	1
🟡 Shard 4	631	0	3	27
✅ Shard 5	611	0	0	42
🟡 Shard 6	646	0	3	8

🟡 17 flaky test(s) (passed on retry)

• Features/DataAssetRulesDisabled.spec.ts › Verify the Database entity item action after rules disabled (shard 1, 1 retry)
• Flow/Tour.spec.ts › Tour should work from welcome screen (shard 1, 1 retry)
• Flow/Tour.spec.ts › Tour should work from URL directly (shard 1, 1 retry)
• Pages/UserCreationWithPersona.spec.ts › Create user with persona and verify on profile (shard 1, 1 retry)
• Features/BulkEditEntity.spec.ts › Glossary (shard 2, 1 retry)
• Features/ChangeSummaryBadge.spec.ts › Automated badge should appear on entity description with Automated source (shard 2, 1 retry)
• Features/IncidentManager.spec.ts › Complete Incident lifecycle with table owner (shard 3, 1 retry)
• Features/RestoreEntityInheritedFields.spec.ts › Validate restore with Inherited domain and data products assigned (shard 3, 1 retry)
• Features/RestoreEntityInheritedFields.spec.ts › Validate restore with Inherited domain and data products assigned (shard 3, 2 retries)
• Features/RestoreEntityInheritedFields.spec.ts › Validate restore with Inherited domain and data products assigned (shard 3, 1 retry)
• Features/RTL.spec.ts › Verify Following widget functionality (shard 3, 1 retry)
• Pages/Customproperties-part2.spec.ts › entityReferenceList shows item count, scrollable list, no expand toggle (shard 4, 1 retry)
• Pages/Domains.spec.ts › Rename domain with tags and glossary terms preserves associations (shard 4, 1 retry)
• Pages/Domains.spec.ts › Rename domain with assets (tables, topics, dashboards) preserves associations (shard 4, 1 retry)
• Pages/Lineage/LineageFilters.spec.ts › Verify lineage schema filter selection (shard 6, 1 retry)
• Pages/Lineage/LineageRightPanel.spec.ts › Verify custom properties tab IS visible for supported type: searchIndex (shard 6, 1 retry)
• Pages/Users.spec.ts › Permissions for table details page for Data Consumer (shard 6, 1 retry)

📦 Download artifacts

How to debug locally

# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

[gitar-bot]

Code Review ✅ Approved

Upgrades collate-sqllineage to version 2.1.1 to resolve multiple reported issues and includes new regression tests. No issues found.

Options

Display: compact → Showing less information.

Comment with these commands to change:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Gitar}

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ingestion'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[github-actions]

Failed to cherry-pick changes to the 1.12.6 branch.
Please cherry-pick the changes manually.
You can find more details here.