[DataFusion] Implement optimizer rule to remove redundant projections

[alamb]

Note: migrated from original JIRA: https://issues.apache.org/jira/browse/ARROW-6892

Currently we have code in the SQL query planner that wraps aggregate queries in a projection (if needed) to preserve the order of the final results. This is needed because the aggregate query execution always returns a result with grouping expressions first and then aggregate expressions.

It would be better (simpler, more readable code) to always wrap aggregates in projections and have an optimizer rule to remove redundant projections. There are likely other use cases where redundant projections might exist too.

[alamb]

Comment from Neville Dipale(nevi_me) @ 2020-09-25T17:29:08.891+0000:

[~andygrove]  [~jorgecarleitao] [~alamb]  do you know if this is resolved? There's been a lot of improvements to the optimizer, so checking if they perhaps included this.

Comment from Jorge Leitão(jorgecarleitao) @ 2020-09-25T17:35:05.272+0000:

IMO this is still open. It is a cool task, though. A simple test is something along the lines of:
{code:java}
df = ctx.table("t")?;
df = df.select(vec![col("a")])?;
df = df.select(vec![col("a"), col("b") + 1])?;

let plan = df.to_logical_plan()
// plan must contain a single projection
{code}

[alamb]

wrong repo