[SPARK-21859][CORE] Fix SparkFiles.get failed on driver in yarn-cluster and yarn-client mode

[lgrcyanny]

What changes were proposed in this pull request?

when use SparkFiles.get a file on driver in yarn-client or yarn-cluster, it will report file not found exception.
This exception only happens on driver, SparkFiles.get on executor works fine.
we can reproduce the bug as follows:

val conf = new SparkConf().setAppName("SparkFilesTest")
val sc = new SparkContext(conf)
def testOnDriver(fileName: String) = {
    val file = new File(SparkFiles.get(fileName))
    if (!file.exists()) {
        println(s"$file not exist")
    } else {
        // print file content on driver
        val content = Source.fromFile(file).getLines().mkString("\n")
        println(s"File content: ${content}")
    }
}
// the output will be file not exist

conf = SparkConf().setAppName("test files")
sc = SparkContext(appName="spark files test")
def test_on_driver(filename):
    file = SparkFiles.get(filename)
    print("file path: {}".format(file))
    if os.path.exists(file):
        with open(file) as f:
        lines = f.readlines()
        print(lines)
    else:
        print("file doesn't exist")
        run_command("ls .")

the output will be file not exist

How was this patch tested?

tested in integration tests and manual tests
submit the demo case in yarn-cluster and yarn-client mode, and verify the test result

./bin/spark-submit --master yarn-cluster --files README.md --class "testing.SparkFilesTest" testing.jar
./bin/spark-submit --master yarn-client --files README.md --class "testing.SparkFilesTest" testing.jar
./bin/spark-submit --master yarn-cluster --files README.md test_get_files.py
./bin/spark-submit --master yarn-client --files README.md test_get_files.py

[AmplabJenkins]

Can one of the admins verify this patch?

[vanzin]

It seems like exposing these files to the driver was added as a feature in SPARK-16787, whether intentionally or not. I'm fine with fixing things this way (aside from the YARN issue), but if doing this it's probably a good idea to fix the SparkSubmit help text (and maybe other documentation that may also be wrong). It currently says:

        |  --files FILES               Comma-separated list of files to be placed in the working
        |                              directory of each executor. File paths of these files
        |                              in executors can be accessed via SparkFiles.get(fileName).

Files are not necessarily placed in the working directory, and they're also now available to the driver, so this text should be updated accordingly.

[vanzin]

I think @jerryshao mentioned this in your other PR, but this is not correct. YARN distributes these files through other means, so doing this might cause other issues.

What you want to do here to make the files show up in the driver is to add some code in SparkContext. There's this code at the end of def addFile, which was added in SPARK-16787:

    val timestamp = System.currentTimeMillis
    if (addedFiles.putIfAbsent(key, timestamp).isEmpty) {
      logInfo(s"Added file $path at $key with timestamp $timestamp")
      // Fetch the file locally so that closures which are run on the driver can still use the
      // SparkFiles API to access files.
      Utils.fetchFile(uri.toString, new File(SparkFiles.getRootDirectory()), conf,
        env.securityManager, hadoopConfiguration, timestamp, useCache = false)
      postEnvironmentUpdate()
    }

You basically want to do that for all files in spark.yarn.dist.files when in YARN client mode.

[lgrcyanny]

For yarn-client mode, --files are are already added to "spark.yarn.dist.files", I agree with you that, just addFile in SparkContext for yarn-client mode for "spark.yarn.dist.files". BTW, I will fix the doc for Spark-Submit either.
Thanks @vanzin