[SPARK-41530][CORE] Rename MedianHeap to PercentileMap and support percentile

[cloud-fan]

What changes were proposed in this pull request?

MedianHeap was added to track the median of task durations, for task speculation. However, median may not be the best option and a configurable percentile could be better.

This PR extends the existing MedianHeap to support percentile.

Why are the changes needed?

Prepare for tracking more statistics of task durations.

Does this PR introduce any user-facing change?

no

How was this patch tested?

new tests

[cloud-fan]

cc @Ngone51

[mridulm]

Prepare for tracking more statistics of task durations.

Can you give more details on this please ?

[dongjoon-hyun]

Technically, MedianHeap is removed by being superceded by PercentileHeap after this PR.

• Since new class is important than the removed one, could you revise the PR title to mention PercentileHeap explicitly?
• Also, in the test suite, I believe it would be better not to mention mediaHeap.

[cloud-fan]

Can you give more details on this please ?

Using the median task duration to trigger task speculation may not be the best option. It's better to allow users to configure the percentile if they want task speculation to happen more or less likely.

[mridulm]

Using the median task duration to trigger task speculation may not be the best option. It's better to allow users to configure the percentile if they want task speculation to happen more or less likely.

Sounds good.
Note that given the complexity of how we determine whether a task is speculatable or not, it is not just a matter of turning the dial on the percentile :-)

[dongjoon-hyun]

+1, LGTM. Thank you, @cloud-fan .

[dongjoon-hyun]

Merged to master for Apache Spark 3.4. Thank you, @cloud-fan and @mridulm .