Review DistributedCache usage (pretty sure it's very broken)

[ctubbsii]

The best explanation (if it even works) of how to use Hadoop's distributed cache is at https://stackoverflow.com/a/26421057/196405

But, I have not tested this.

Our code does not do anything like this.... we tell the distributed cache to add a cache file, but then we just loop through the list of cached URIs, pointlessly, and then reach directly out to the DFS to read it.

All of the Hadoop APIs to read the local cache files are (rightfully) deprecated, because there's no way to map them to the files that were added to the cache. But the APIs to add them to the cache are not deprecated... but they also aren't documented.

We should test the above solution, and if it works, use it, otherwise, completely remove any use of the distributed cache in our MapReduce code.

[jzgithub1]

Is this related to the VFS issue #933?

[ctubbsii]

@jzgithub1 No. It's unrelated.

[jzgithub1]

Working on this now.

[jzgithub1]

public static void setSplitFile(Job job, String file) in our KeyRangePartitioner uses Job.addCacheFile and is not used in any of our code but could be used in somebody's client code so we might break other peoples' client code if we remove it.

public static void setSplitFile(Job job, String file) in the RangePartitioner is called in the KeyRangePartitioner in the KeyRangePartitioner.setSplitFile function but nowhere else. Removing that function could break someone's client code also.

setClientProperties in ConfiguratorBase calls DistributedCacheHelper.addCacheFile(new URI(clientPropsPath), conf) which calls Job.addCacheFile too.

If our interface to the Hadoop API is working for people who are using the our client API we might want to keep this code in place and test what we have.

[ctubbsii]

If we're not using setSplitFile in our own code, I'm not surprised, but since it's API, it should at least have a test (UT or IT). We definitely shouldn't change it. ConfiguratorBase is not public API, so we're free to change that if necessary, but I don't think even that would need to change for this issue... just it's implementation.

[jzgithub1]

Solution for Hadoop DistributedCache an this web page works: https://stackoverflow.com/a/26421057/196405

I modified the TokenFileIT test to make the these calls on the hadoop.Job object in the run() method:

  public int run(String[] args) throws Exception {
    ...
    job.addCacheFile(new URI("hdfs://localhost:8020/some_file.json#some"));
    job.addCacheFile(new URI("hdfs://localhost:8020/other_file.json#other"));
    ...
  }

I also added an override method for setup(context) in the integrated tests internal hadoop.mapper class like so:

      protected void setup(Context context) throws IOException, InterruptedException {
        if (context.getCacheFiles() != null && context.getCacheFiles().length > 0) {

          File some_file = new File("./some");
          File other_file = new File("./other");

          BufferedReader br = new BufferedReader(new FileReader("./some"));
          try {
            StringBuilder sb = new StringBuilder();
            String line = br.readLine();

            while (line != null) {
              sb.append(line);
              sb.append(System.lineSeparator());
              line = br.readLine();
            }
            String everything = sb.toString();
          } finally {
            br.close();
          }

          // Do things to these two files, like read them
          // or parse as JSON or whatever.
        }
        super.setup(context);
      }

On my command line I added two small json files to HDFS via the commands:
jzeiberg@dhcp-10-50-230-160  ~/.cache  hdfs dfs -put other_file.json /
jzeiberg@dhcp-10-50-230-160  ~/.cache  hdfs dfs -put some_file.json /

When I ran the code in the debugger I was able to read the contents of the somefile.json in the setup function above using the "some" mnemonic specified by adding #some after the fully qualified hdfs URI in the job.addCacheFile function call.

Since this haddoop distributed cache works the way the stackoverflow webpage says it does I will now proceed to use this distributed cache method where we attempt to use distributed cache elsewhere in our code.

[ctubbsii]

If the framework places it in the current working directory when the file is cached locally, it could be a problem for our integration tests, since the current working directory in those cases is <git-checkout>/test and not <git-checkout>/test/target. If that happens, we'll need to come up with a workaround for our test, so the test doesn't pollute our source tree during a build.