Add extra PATH to find tensorboard

[wangyum]

What changes were proposed in this pull request?

Generally speaking we set PYSPARK_PYTHON to spark-env.sh. It means that PYSPARK_PYTHON may be not in os.environ. This PR add extra PATH to find tensorboard. more see #61.

PYTHONPATH may looks like this:

'PYTHONPATH': '/opt/cloudera/parcels/2.1.0-2.6.0.d20170113-15.36.26/lib/spark/jars/spark-core_2.11-2.2.0-SNAPSHOT.jar:/opt/cloudera/parcels/Anaconda/bin/python::/opt/cloudera/parcels/Anaconda/bin/python:/data/M05/yarn/nm/usercache/tandem/appcache/application_1491988027269_1406/container_e55_1491988027269_1406_01_000004/__pyfiles__:/opt/cloudera/parcels/2.1.0-2.6.0.d20170113-15.36.26/lib/spark/python/lib/py4j-0.10.4-src.zip:/opt/cloudera/parcels/2.1.0-2.6.0.d20170113-15.36.26/lib/spark/python/lib/pyspark.zip:/data/M05/yarn/nm/usercache/tandem/appcache/application_1491988027269_1406/container_e55_1491988027269_1406_01_000004/tfspark.zip'

How was this patch tested?

manual tests

[wangyum]

CC @leewyang

[leewyang]

@wangyum Thanks for the PR. (Sorry for the delay, since I've been out).

Quick question: is there a reason why you can't do something like this:

os.environ['PATH'] = os.environ['PATH'] + os.pathsep + os.environ['PYTHONPATH']

i.e. why do you need to get the dirname() of each sub-component of the PYTHONPATH?

I can't really test this in my environment, so if this is required, that's fine... (just wanted to make sure).

[wangyum]

PYSPARK_PYTHON is something like /opt/cloudera/parcels/Anaconda/bin/python, It's a file. I have test os.environ['PATH'] = os.environ['PATH'] + os.pathsep + os.environ['PYTHONPATH'] and it does not work.

[leewyang]

K, thanks for checking.
+1