Skip to content
This repository has been archived by the owner on Jul 9, 2022. It is now read-only.

spring-attic/spring-cloud-dataflow-server-yarn

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

spring-cloud-dataflow-server-yarn is no longer actively maintained by VMware, Inc.

Spring Cloud Data Flow Server for Apache Yarn

The Spring Cloud Data Flow team have been happy stewards of this project, but we are focusing our efforts towards Local, Kubernetes and Cloud Foundry implementations going forward, so the Spring Cloud Data Flow team will no longer maintain it.

However, we are looking to donate the project to the community. If you are interested in forking the project to continue to develop and maintaining it, we would be happy to cross-link and promote your forked project. Please ping us at Gitter to discuss it.

Building

Build and skip all tests:

./mvnw clean package -DskipTests

Or build project and run tests(tests take long time to run):

./mvnw clean package

Running the Server

All needed artifacts are packaged in a distribution file spring-cloud-dataflow-server-yarn-dist/target/spring-cloud-dataflow-server-yarn-dist-1.0.0.BUILD-SNAPSHOT.zip.

  1. Unzip the dist

# unzip spring-cloud-dataflow-server-yarn-dist-1.0.0.BUILD-SNAPSHOT.zip
# find .
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin/dataflow-server-yarn
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin/dataflow-shell
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin/dataflow-server-yarn-h2
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin/dataflow-server-yarn-cli
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/lib
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/lib/spring-cloud-deployer-yarn-tasklauncherappmaster-1.0.0.BUILD-SNAPSHOT.jar
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/lib/spring-cloud-dataflow-server-yarn-client-1.0.0.BUILD-SNAPSHOT.jar
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/lib/spring-cloud-deployer-yarn-appdeployerappmaster-1.0.0.BUILD-SNAPSHOT.jar
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/lib/spring-cloud-dataflow-server-yarn-h2-1.0.0.BUILD-SNAPSHOT.jar
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/lib/spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT.jar
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/lib/spring-cloud-dataflow-shell-1.0.0.BUILD-SNAPSHOT.jar
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/config
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/config/servers.yml
  1. Configure servers.yml for hadoop and redis settings.

    Start server:

    spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin/dataflow-server-yarn
  2. Start the Data Flow Shell:

    spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin/dataflow-shell
  3. Create a stream:

    dataflow:>stream create --name ticktock --definition "time|hdfs --rollover=100" --deploy
    dataflow:>stream destroy --name ticktock
  4. Killing a running application from the command line:

    spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin/dataflow-server-yarn-cli shell
    
    
    $ submitted
      APPLICATION ID                  USER          NAME                            QUEUE    TYPE      STARTTIME       FINISHTIME  STATE    FINALSTATUS  ORIGINAL TRACKING URL
      ------------------------------  ------------  ------------------------------  -------  --------  --------------  ----------  -------  -----------  --------------------------
      application_1447944262603_0003  jvalkealahti  spring-cloud-dataflow-yarn-app  default  DATAFLOW  19/11/15 15:49  N/A         RUNNING  UNDEFINED    http://172.16.14.143:45566
    
    $ kill -a application_1447944262603_0003
    Kill request for application_1447944262603_0003 sent