[SPARK-31976][ML][PYSPARK] LinearSVC use MemoryUsage to control the size of block

[zhengruifeng]

What changes were proposed in this pull request?

in LinearSVC, use maxBlockMemoryInMB instead of blockSize

Why are the changes needed?

According to the performance test in https://issues.apache.org/jira/browse/SPARK-31783, the performance gain is mainly related to the nnz of block.

Does this PR introduce any user-facing change?

yes, blockSize is changed to maxBlockMemoryInMB

How was this patch tested?

added testsuites

[SparkQA]

Test build #124865 has finished for PR 28974 at commit 5d7432d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• trait HasMaxBlockMemoryInMB extends Params
• class HasMaxBlockMemoryInMB(Params):

[zhengruifeng]

ping @WeichenXu123 @mengxr

[HyukjinKwon]

cc @huaxingao @srowen too

[HyukjinKwon]

Oops, mistake.

[huaxingao]

nit: add a scala doc?

[huaxingao]

@zhengruifeng Did you have a chance to do a benchmark test to verify the performance gain?

[zhengruifeng]

@huaxingao since this changed is suggested by @mengxr and @WeichenXu123 , I perfer to append the performace tests after they think current design is OK.

In current commit, if the maxBlockMemoryInMB==0 (by default), the old cold path trainOnRows is chosen. If we want to discard this code path in the future, there will be a small behavior change, that the default maxBlockMemoryInMB need to be set with another default value (i.e. 4MB).

Existing performace tests were done against numRows, I think the results can be converted to maxBlockMemoryInMB. (But I agree with anohter test directly on maxBlockMemoryInMB).

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!