Install and Configure ML

Ricardo Barona edited this page Dec 21, 2016 · 4 revisions

Build the Spark application

Build spot-ml Spark application.

[soluser@edge-server]$ cd spot-ml
[soluser@edge-server]$ sbt assembly

Copy the entire spot-ml folder to the primary ML node, the node that will launch Spark application. For simplicity, this is often the lowest-numbered node in this role. Log in to the primary ML node.

[soluser@edge-server]$ scp –r spot-ml node-04:/home/solution_user/.
[soluser@edge-server]$ ssh node-04
[soluser@node-04]$cd /home/solution_user/spot-ml

Edit configuration variables in /etc/spot.conf

If it hasn't been done, copy /etc/spot.conf to ML node.

Once /etc/spot.conf is in ML node, make sure the following configurations are set:

 SPK_EXEC - Number of Spark executors
 SPK_EXEC_MEM - Total memory per executor
 SPK_DRIVER_MEM - Total driver memory
 SPK_DRIVER_MAX_RESULTS - Total memory for driver max results
 SPK_EXEC_CORES - Number of cores per executor
 SPK_DRIVER_MEM_OVERHEAD - Driver memory overhead
 SPAK_EXEC_MEM_OVERHEAD - Executor memory overhead
 TOL - Results threshold

More details about how to set up Spark properties con be found here.

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.