Eclipse plugin tailored for swift big data access using scalding. This is an eclipse plugin for creating a Scalding/WebScalding projects. The plugin has the capability of deploying the fatjar of the project along with a script file (to execute the fatjar)
Prerequisites
- Mac/Windows environment (Implemented in Mac, tried in Windows. see no problems in working on a Linux machine)
- SBT downloaded and extracted, ( Works on 0.13.7)
- Eclipse with Scala-ide (Works on Eclipse Luna and scala-ide 4.0.0)
- Plugin m2eclipse-scala if you are plannig to work with Maven instead of SBT
Installation
- Download the latest release from here
- Install the plugin into your Eclipse IDE from the archive file using the instructions mentioned here
How to use
Configuring the cluster and SBT installation
- Selected Preferences -> Tailorswift
- Provide the SBT "sbt-launch.jar" location
- Provide the connection string and password for the Cluster connection.
Creating a WebScalding Project
- Select "File" -> "New" -> "Project" -> "Webscalding" -> "New WebScalding Project (SBT/Maven)" -> "Next"
- Enter a name for the project
- "Finish"
A SCALA project should be created and user can run it in a cluster.
Runing a WebScalding Project
- Select "Run" -> "Run Configurations" -> "New Configuration" and provide the details
- OR select a SCALA class overriding the com.twitter.scalding.Job class and select option "as WebScalding Job"
An example setup video is given here
Have fun!
Errors and fixes
-
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
Update -mem argument in the sbtopt file in the sbt installation directory. e.g., "-mem 23000"