adding the ability to define a local_output_dir so one can use hadoop cl...

[jassinm]

simple change : ability to output sc hadoop config files to local_output_dir.
user can define local_output_dir in the plugin
export HADOOP_CONF_DIR=local_output_dir
and run hadoop jobs from local machine as long as he runs the same client as sc.

[jtriley]

This looks good. I'm wondering if we should just find the jars on the master node and copy them to the user's local machine for them automatically? We can do this using:

master.ssh.get('/path/to/jar", '/local/path/')

This way users don't have to scramble to find the proper hadoop version, etc. What do you think?

Also can you provide an example of running some hadoop jobs using the local HADOOP_CONF_DIR? Even if it seems obvious it may not be straightforward to all users...