Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Install and Configure ML
Build the Spark application
Build spot-ml Spark application.
[soluser@edge-server]$ cd spot-ml [soluser@edge-server]$ sbt assembly
Copy the entire spot-ml folder to the primary ML node, the node that will launch Spark application. For simplicity, this is often the lowest-numbered node in this role. Log in to the primary ML node.
[soluser@edge-server]$ scp –r spot-ml node-04:/home/solution_user/. [soluser@edge-server]$ ssh node-04 [soluser@node-04]$cd /home/solution_user/spot-ml
Edit configuration variables in /etc/spot.conf
If it hasn't been done, copy /etc/spot.conf to ML node.
Once /etc/spot.conf is in ML node, make sure the following configurations are set:
SPK_EXEC - Number of Spark executors SPK_EXEC_MEM - Total memory per executor SPK_DRIVER_MEM - Total driver memory SPK_DRIVER_MAX_RESULTS - Total memory for driver max results SPK_EXEC_CORES - Number of cores per executor SPK_DRIVER_MEM_OVERHEAD - Driver memory overhead SPAK_EXEC_MEM_OVERHEAD - Executor memory overhead TOL - Results threshold
More details about how to set up Spark properties con be found here.