[SPARK-2714] DAGScheduler logs jobid when runJob finishes

[YanTangZhai]

DAGScheduler logs jobid when runJob finishes

[AmplabJenkins]

Can one of the admins verify this patch?

[markhamstra]

What is the need to expose the jobId after the job is finished?

[YanTangZhai]

Hi @markhamstra When DAGScheduler concurrently runs multiple jobs, SparkContext only logs "Job finished" and logs in the same file which doesn't tell who is who. It's difficult to found which job has finished or how much time it has taken from multiple "Job finished: ..., took ... s" logs.

[andrewor14]

ok to test

[andrewor14]

Can you do string formatting instead to make this more readable:

logInfo("Job %d finished: %s, took %ds".format(
  (waiter.jobId, callSite.shortForm, (System.nanoTime - start) / 1e9))

[SparkQA]

QA tests have started for PR 1617 at commit fb42f0f.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 1617 at commit fb42f0f.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

QA tests have started for PR 1617 at commit 7aec2a9.

• This patch merges cleanly.

[YanTangZhai]

@andrewor14 Thanks. Please review again.

[SparkQA]

QA tests have started for PR 1617 at commit fbb1150.

• This patch merges cleanly.

[SparkQA]

Tests timed out after a configured wait of 120m.

[SparkQA]

Tests timed out after a configured wait of 120m.

[andrewor14]

Oops, the last one has to be %f, here and in L431

[SparkQA]

QA tests have started for PR 1617 at commit 0a0243f.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 1617 at commit 0a0243f.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

Thanks, merging this into master.