Spring Cloud Data Flow Implementation for Apache YARN
Shell Java XSLT CSS Batchfile Ruby
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
.mvn
spring-cloud-dataflow-server-yarn-autoconfig
spring-cloud-dataflow-server-yarn-client
spring-cloud-dataflow-server-yarn-dist
spring-cloud-dataflow-server-yarn-docs
spring-cloud-dataflow-server-yarn-h2
spring-cloud-dataflow-server-yarn-shell-core
spring-cloud-dataflow-server-yarn-shell
spring-cloud-dataflow-server-yarn
src
.gitignore
.settings.xml
.travis.yml
README.adoc
mvnw
mvnw.cmd
pom.xml

README.adoc

Spring Cloud Data Flow Server for Apache Yarn

Build Status Stories in Ready Stories in Progress

Building

Build and skip all tests:

./mvnw clean package -DskipTests

Or build project and run tests(tests take long time to run):

./mvnw clean package

Running the Server

All needed artifacts are packaged in a distribution file spring-cloud-dataflow-server-yarn-dist/target/spring-cloud-dataflow-server-yarn-dist-1.0.0.BUILD-SNAPSHOT.zip.

  1. Unzip the dist

# unzip spring-cloud-dataflow-server-yarn-dist-1.0.0.BUILD-SNAPSHOT.zip
# find .
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin/dataflow-server-yarn
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin/dataflow-shell
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin/dataflow-server-yarn-h2
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin/dataflow-server-yarn-cli
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/lib
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/lib/spring-cloud-deployer-yarn-tasklauncherappmaster-1.0.0.BUILD-SNAPSHOT.jar
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/lib/spring-cloud-dataflow-server-yarn-client-1.0.0.BUILD-SNAPSHOT.jar
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/lib/spring-cloud-deployer-yarn-appdeployerappmaster-1.0.0.BUILD-SNAPSHOT.jar
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/lib/spring-cloud-dataflow-server-yarn-h2-1.0.0.BUILD-SNAPSHOT.jar
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/lib/spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT.jar
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/lib/spring-cloud-dataflow-shell-1.0.0.BUILD-SNAPSHOT.jar
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/config
./spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/config/servers.yml
  1. Configure servers.yml for hadoop and redis settings.

    Start server:

    spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin/dataflow-server-yarn
  2. Start the Data Flow Shell:

    spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin/dataflow-shell
  3. Create a stream:

    dataflow:>stream create --name ticktock --definition "time|hdfs --rollover=100" --deploy
    dataflow:>stream destroy --name ticktock
  4. Killing a running application from the command line:

    spring-cloud-dataflow-server-yarn-1.0.0.BUILD-SNAPSHOT/bin/dataflow-server-yarn-cli shell
    
    
    $ submitted
      APPLICATION ID                  USER          NAME                            QUEUE    TYPE      STARTTIME       FINISHTIME  STATE    FINALSTATUS  ORIGINAL TRACKING URL
      ------------------------------  ------------  ------------------------------  -------  --------  --------------  ----------  -------  -----------  --------------------------
      application_1447944262603_0003  jvalkealahti  spring-cloud-dataflow-yarn-app  default  DATAFLOW  19/11/15 15:49  N/A         RUNNING  UNDEFINED    http://172.16.14.143:45566
    
    $ kill -a application_1447944262603_0003
    Kill request for application_1447944262603_0003 sent