Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[SPARK-32616][SQL] Window operators should be added determinedly
### What changes were proposed in this pull request? Use the `LinkedHashMap` instead of `immutable.Map` to hold the `Window` expressions in `ExtractWindowExpressions.addWindow`. ### Why are the changes needed? This is a bug fix for apache#29270. In that PR, the generated plan(especially for the queries q47, q49, q57) on Jenkins always can not match the golden plan generated on my laptop. It happens because `ExtractWindowExpressions.addWindow` now uses `immutable.Map` to hold the `Window` expressions by the key `(spec.partitionSpec, spec.orderSpec, WindowFunctionType.functionType(expr))` and converts the map to `Seq` at the end. Then, the `Seq` is used to add Window operators on top of the child plan. However, for the same query, the order of Windows expression inside the `Seq` could be undetermined when the expression id changes(which can affect the key). As a result, the same query could have different plans because of the undetermined order of Window operators. Therefore, we use `LinkedHashMap`, which records the insertion order of entries, to make the adding order determined. ### Does this PR introduce _any_ user-facing change? Maybe yes, users now always see the same plan for the same queries with multiple Window operators. ### How was this patch tested? It's really hard to make a reproduce demo. I just tested manually with apache#29270 and it looks good. Closes apache#29432 from Ngone51/fix-addWindow. Authored-by: yi.wu <yi.wu@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
- Loading branch information