[SPARK-54292][SQL] Support aggregate functions and GROUP BY in |> SELECT pipe operators

[dtenedor]

What changes were proposed in this pull request?

This PR allows aggregate functions and GROUP BY to be used in |> SELECT pipe operators. Previously, these were only allowed in |> AGGREGATE pipe operators.

Example queries now supported:

-- Aggregate in SELECT
table employees |> select sum(salary) as total_salary;

-- Aggregate with GROUP BY
table orders |> select customer_id, count(*) as order_count group by customer_id;

-- Chained operations
table data |> where status = 'active' |> select sum(value) as total;

Why are the changes needed?

By lifting this restriction (with an opt-out mechanism), we make the SQL pipe operator syntax more intuitive while maintaining backwards compatibility.

Does this PR introduce any user-facing change?

Yes, but it is backwards compatible:

• Previously failing queries now succeed: Queries using aggregate functions in |> SELECT will now work instead of throwing PIPE_OPERATOR_CONTAINS_AGGREGATE_FUNCTION errors
• All previously succeeding queries continue to work: No regression; queries using |> AGGREGATE or non-aggregate pipe operators are unaffected

Backwards Compatibility Guarantee:

• ✅ No queries that worked before will break
• ✅ Only queries that previously failed will now succeed

How was this patch tested?

1. Unit Tests: Added comprehensive test coverage in pipe-operators.sql:

   • Positive tests: aggregates in SELECT, with WHERE, with chaining, with GROUP BY
   • Negative tests: aggregates in WHERE (still fails as expected)
   • Regression tests: verified |> AGGREGATE still works correctly

2. Golden Files: Regenerated and verified pipe-operators.sql.out and analyzer results

3. Test Execution: All tests pass successfully:

Was this patch authored or co-authored using generative AI tooling?

Yes, claude-4.5-sonnet with manual editing and approval.

[dtenedor]

cc @srielau @cloud-fan

[dongjoon-hyun]

Thank you for adding this new feature, @dtenedor .

[dongjoon-hyun]

Could you update this PR, @dtenedor ?

[dongjoon-hyun]

Got it.

[dtenedor]

This one is passing CI and seems self-contained and safe, do you approve?

[dongjoon-hyun]

Sure, +1, LGTM (for Apache Spark 4.2.0). Thank you, @dtenedor .

[dtenedor]

OK, great. I will merge it to the master branch only, for Apache Spark 4.2.0 and later. Thank you for review!

[cloud-fan]

This makes me wondering why we need the config. If people don't use agg function in pipe SELECT, then they just don't use it. If they use it, then they will see this error message and enable this config immediately.

cc @srielau

[dtenedor]

I had the chance to speak with Wenchen in person about this -- it is a good point, the config doesn't really help in any scenarios here. I will remove it.

[dtenedor]

Thanks all for reviews -- merging to master.