-
Spark set up:
-
Other configuration:
python 2.6+, OpenCV 2.4.11, NumPy, Pyplot, SciPy.
- Clone to local disk
- Set "Main_Path" and "SPARK_HOME" variable to your environment setting.
- Run start.sh
- Download three-day image-stream to local disk. ImagePullerManager.py will pull image from RethinkDB.
- Convert image-stream to Hadoop SequenceFile seperately using "tar-to-seq.jar".
- Submit application to Spark and Run Spark
- The test application deploy spark in Standalone Mode
- This is just a test application that runs in local machine.
- Set up 4 cores to simulate 4 parallel threads running application.
- Does not save RESULT of SKY_REGION_MASK and SUN_TRACK_COEFFICIENT to disk.