[SPARK-35853][SQL] Remark the shuffle origin to ENSURE_REQUIREMENTS as far as possible

[ulysses-you]

What changes were proposed in this pull request?

Add a rule RemarkShuffleOrigin in AQE queryStagePreparationRules after EnsureRequirements.

Why are the changes needed?

In some queries, we might repartition by some columns with a large partition number manually to make parallelism big enough. However if its output partitioning satisfies some other node (e.g. join/aggregate), this shuffle can not be optimized by AQE due to the shuffle origin.

So, this new rule aims to remark the shuffle origin to ENSURE_REQUIREMENTS as far as possible if it's safe.

Does this PR introduce any user-facing change?

yes, the plan may be changed

How was this patch tested?

Add test.

[SparkQA]

Test build #140126 has finished for PR 33015 at commit f1beaf0.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44653/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44653/

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44654/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44654/

[ulysses-you]

cc @maropu @cloud-fan @yaooqinn @maryannxue

[cloud-fan]

In some queries, we might repartition by some columns with a large partition number manually to make parallelism big enough.

I don't think it's safe to "guess" the user intention and allow AQE to break partitioning. Let's create a new hint for this optimize-write repartition, to make the user intention clearer.

[ulysses-you]

How about that PR #32932 ?

[SparkQA]

Test build #140128 has finished for PR 33015 at commit 7ce90ad.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.