[SEDONA-405] Replace Metric with LongAccumulator to reduce memory overhead for spatial join

[jiayuasu]

Did you read the Contributor Guide?

• Yes, I have read Contributor Rules and Contributor Development Guide

Is this PR related to a JIRA ticket?

• Yes, the URL of the associated JIRA ticket is https://issues.apache.org/jira/browse/SEDONA-405. The PR name follows the format [SEDONA-405] my subject.

What changes were proposed in this PR?

Replace the old Metric class with the Spark built-in LongAccumulator to reduce the memory overhead of spatial join

This should significantly reduce the Sedona driver program memory overhead for both RDD-based join and SQL-based join

How was this patch tested?

Passed existing tests.

Did this PR include necessary documentation updates?

• Yes, I have updated the documentation update.

[umartin]

Just my two cents. I would prefer removing the Sedona Metric class and use the Spark build in LongAccumulator instead. The LongAccumulator has no overhead. There would be no need for a debug option. Less code, less knobs and metrics for everyone. Keeping the Sedona Metric class means there is no way to have metrics for larger jobs, where you would really want them for tuning.

[jiayuasu]

@umartin The Metric is designed to capture the <paritionId, buildSideCount> and <paritionId, streamSideCount> for each partition in a task. If we use the LongAccumulator without the partitionId, then there is no partition-wise information and there is no point of even having this LongAccumulator.

Or maybe I misunderstood your suggestion. Please advise.

[umartin]

This is from the Spark UI where I replaced the Sedona Metric with a LongAccumulator. Spark already tracks the accumulators by task in "Tasks" as well as presenting the summary in "Accumulators"

[jiayuasu]

@umartin Oh, now it is clear to me. Thanks for the suggestion. Will fix it according to your suggestion

[umartin]

Thank you! I'm really looking forward to the 1.5 release. There are tons of great changes. You've been really busy!