[SPARK-15857]Add caller context in Spark: invoke YARN/HDFS API to set…

[weiqingy]

What changes were proposed in this pull request?

1. Pass 'jobId' to Task.

2. Add a new function 'setCallerContext' in Utils. 'setCallerContext' function will call APIs of 'org.apache.hadoop.ipc.CallerContext' to set up spark caller contexts, which will be written into HDFS hdfs-audit.log or Yarn resource manager log.

3. 'setCallerContext' function will be called in Yarn client, ApplicationMaster, and Task class.

   The Spark caller context written into HDFS log will be "JobID_stageID_stageAttemptId_taskID_attemptNumbe on Spark", and the Spark caller context written into Yarn log will be"{spark.app.name} running on Spark".

How was this patch tested?

Manual Tests against some Spark applications in Yarn client mode and cluster mode. Need to check if spark caller contexts were written into HDFS hdfs-audit.log and Yarn resource manager log successfully.

For example, run SparkKmeans on Spark:
In Yarn resource manager log, there will be a record with the spark caller context.
...
2016-07-21 13:36:26,318 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=wyang IP=127.0.0.1 OPERATION=Submit Application Request TARGET=ClientRMService RESULT=SUCCESS APPID=application_1469125587135_0004 CALLERCONTEXT=SparkKMeans running on Spark
...

In HDFS hdfs-audit.log, there will be records with spark caller contexts.
...
2016-07-21 13:38:30,799 INFO FSNamesystem.audit: allowed=true ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/lr_big.txt/_spark_metadata dst=null perm=null proto=rpc callerContext=SparkKMeans running on Spark
...
2016-07-21 13:39:35,584 INFO FSNamesystem.audit: allowed=true ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/lr_big.txt dst=null perm=null proto=rpc callerContext=JobId_0_StageID_0_stageAttemptId_0_taskID_1_attemptNumber_0 on Spark
...

If the hadoop version on which Spark runs does not have CallerContext APIs, there will be no information of Spark caller context in those logs.

… up caller context

[AmplabJenkins]

Can one of the admins verify this patch?

[jerryshao]

nit: catch should follow the above }.

[jerryshao]

Spark caller context written into Yarn log will be"{spark.app.name} running on Spark".

This may not be so useful, I think we could get app name form yarn through many different ways, simply printing one line log to RM is not so useful.

[weiqingy]

Thanks the feedback, Jerry. I am going to update the patch.