[SPARK-51385][SQL] Normalize out projection added in DeduplicateRelations for union child output deduplication

[vladimirg-db]

What changes were proposed in this pull request?

Strip away extra projection added by DeduplicateRelations when comparing logical plans.

DeduplicateRelations puts one extra Project on the right branch of Union when the outputs of children are conflicting. This is a hack for streaming relations. Unfortunately this logic is generalized an the extra projection is used for simple cases like views:

CREATE VIEW IF NOT EXISTS v1 AS SELECT * FROM VALUES (1, 2);

SELECT * FROM (
  SELECT col1, col2 FROM v1
  UNION ALL
  SELECT col1, col2 FROM v1
);

Single-pass Analyzer should not produce this projection, because it assigns expression IDs in single-pass, so we strip it in NormalizePlan to correctly compare the plans.

Why are the changes needed?

This is to make sure that single-pass and fixed-point Analyzed plans are the same.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing tests.

Was this patch authored or co-authored using generative AI tooling?

No.

[cloud-fan]

thanks, merging to master!