spring-cloud-dataflow-server-yarn is no longer actively maintained by VMware, Inc.

Spring Cloud Data Flow Server for Apache Yarn

The Spring Cloud Data Flow team have been happy stewards of this project, but we are focusing our efforts towards Local, Kubernetes and Cloud Foundry implementations going forward, so the Spring Cloud Data Flow team will no longer maintain it.

However, we are looking to donate the project to the community. If you are interested in forking the project to continue to develop and maintaining it, we would be happy to cross-link and promote your forked project. Please ping us at Gitter to discuss it.

Building

Build and skip all tests:

./mvnw clean package -DskipTests

Or build project and run tests(tests take long time to run):

./mvnw clean package

Running the Server

All needed artifacts are packaged in a distribution file spring-cloud-dataflow-server-yarn-dist/target/spring-cloud-dataflow-server-yarn-dist-1.0.0.BUILD-SNAPSHOT.zip.

1. Unzip the dist

# unzip spring-cloud-dataflow-server-yarn-dist-1.0.0.BUILD-SNAPSHOT.zip
# find .
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin/dataflow-server-yarn
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin/dataflow-shell
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin/dataflow-server-yarn-h2
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin/dataflow-server-yarn-cli
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/lib
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/lib/spring-cloud-deployer-yarn-tasklauncherappmaster-1.0.0.BUILD-SNAPSHOT.jar
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/lib/spring-cloud-dataflow-server-yarn-client-1.0.0.BUILD-SNAPSHOT.jar
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/lib/spring-cloud-deployer-yarn-appdeployerappmaster-1.0.0.BUILD-SNAPSHOT.jar
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/lib/spring-cloud-dataflow-server-yarn-h2-1.0.0.BUILD-SNAPSHOT.jar
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/lib/spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT.jar
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/lib/spring-cloud-dataflow-shell-1.0.0.BUILD-SNAPSHOT.jar
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/config
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/config/servers.yml

1. Configure servers.yml for hadoop and redis settings.

   Start server:

   spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin/dataflow-server-yarn

2. Start the Data Flow Shell:

   spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin/dataflow-shell

3. Create a stream:

   dataflow:>stream create --name ticktock --definition "time|hdfs --rollover=100" --deploy
   dataflow:>stream destroy --name ticktock

4. Killing a running application from the command line:

   spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin/dataflow-server-yarn-cli shell
   
   
   $ submitted
     APPLICATION ID                  USER          NAME                            QUEUE    TYPE      STARTTIME       FINISHTIME  STATE    FINALSTATUS  ORIGINAL TRACKING URL
     ------------------------------  ------------  ------------------------------  -------  --------  --------------  ----------  -------  -----------  --------------------------
     application_1447944262603_0003  jvalkealahti  spring-cloud-dataflow-yarn-app  default  DATAFLOW  19/11/15 15:49  N/A         RUNNING  UNDEFINED    http://172.16.14.143:45566
   
   $ kill -a application_1447944262603_0003
   Kill request for application_1447944262603_0003 sent