[SPARK-4387][PySpark] Refactoring python profiling code to make it extensible

[davies]

This PR is based on #3255 , fix conflicts and code style.

Closes #3255.

[SparkQA]

Test build #25057 has started for PR 3901 at commit 31bf6b6.

• This patch merges cleanly.

[JoshRosen]

Thanks for picking this up!

[SparkQA]

Test build #25057 has finished for PR 3901 at commit 31bf6b6.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class ProfilerCollector(object):
  • class BasicProfiler(object):
  • >>> class MyCustomProfiler(BasicProfiler):

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25057/
Test FAILed.

[SparkQA]

Test build #25067 has started for PR 3901 at commit 6a5d4df.

• This patch merges cleanly.

[SparkQA]

Test build #25068 has started for PR 3901 at commit 349e341.

• This patch merges cleanly.

[SparkQA]

Test build #25070 has started for PR 3901 at commit 2700e47.

• This patch merges cleanly.

[SparkQA]

Test build #25073 has started for PR 3901 at commit 4b79ce8.

• This patch merges cleanly.

[davies]

@JoshRosen I did some refactor on this, please take another pass of review, thanks.

[SparkQA]

Test build #25068 has finished for PR 3901 at commit 349e341.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class ProfilerCollector(object):
  • class Profiler(object):
  • >>> class MyCustomProfiler(BasicProfiler):
  • class PStatsParam(AccumulatorParam):
  • class BasicProfiler(Profiler):

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25068/
Test FAILed.

[SparkQA]

Test build #25070 has finished for PR 3901 at commit 2700e47.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class ProfilerCollector(object):
  • class Profiler(object):
  • >>> class MyCustomProfiler(BasicProfiler):
  • class PStatsParam(AccumulatorParam):
  • class BasicProfiler(Profiler):

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25070/
Test FAILed.

[SparkQA]

Test build #25067 has finished for PR 3901 at commit 6a5d4df.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class ProfilerCollector(object):
  • class Profiler(object):
  • >>> class MyCustomProfiler(BasicProfiler):
  • class PStatsParam(AccumulatorParam):
  • class BasicProfiler(Profiler):

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25067/
Test PASSed.

[SparkQA]

Test build #25073 has finished for PR 3901 at commit 4b79ce8.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class ProfilerCollector(object):
  • class Profiler(object):
  • >>> class MyCustomProfiler(BasicProfiler):
  • class PStatsParam(AccumulatorParam):
  • class BasicProfiler(Profiler):

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25073/
Test FAILed.

[SparkQA]

Test build #25077 has started for PR 3901 at commit b4a9306.

• This patch merges cleanly.

[SparkQA]

Test build #25077 has finished for PR 3901 at commit b4a9306.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • SparkSubmit.printErrorAndExit(s"Cannot load main class from JAR $primaryResource")
  • class BinaryClassificationMetrics(
  • class ProfilerCollector(object):
  • class Profiler(object):
  • >>> class MyCustomProfiler(BasicProfiler):
  • class PStatsParam(AccumulatorParam):
  • class BasicProfiler(Profiler):

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25077/
Test PASSed.

[udnay]

Thanks @davies

[davies]

@JoshRosen @mateiz This PR will introduce new API for pyspark, please help to review it.

[davies]

@JoshRosen Is it ready to go?

[davies]

@JoshRosen Should we include this in 1.3?

[JoshRosen]

@davies Sure, let's pull this in for 1.3. Sorry for letting this hang in the review queue for so long. This looks good to me, so I'm going to merge it to master.

[orenmazor]

💃

[solmazsh]

🚀

[yagnik]

💃