Permalink
Switch branches/tags
Nothing to show
Find file Copy path
83ebef5 Oct 15, 2018
1 contributor

Users who have contributed to this file

122 lines (90 sloc) 4.88 KB

TaskLauncher Data Flow Sink

This application launches a registered task application using the Data Flow Server REST API.

Input

Launch request args including:

  • the task name (required and registered as a task with the target Data Flow Server)

  • deployment properties (key value pairs, optional).

  • program arguments for the task (a list, optional).

Headers:

  • Content-Type: application/json

Payload:

A JSON document:

{
  "name":"foo",
  "deploymentProps": {"key1":"val1","key2":"val2"},
  "args":["--debug", "--foo", "bar"]
}

minimally,

{"name":"foo"}

Output

N/A (launches task to the local system).

Options

The tasklauncher-dataflow sink supports the following configuration properties:

spring.cloud.dataflow.client.authentication.basic.password

The login password. (String, default: <none>)

spring.cloud.dataflow.client.authentication.basic.username

The login username. (String, default: <none>)

spring.cloud.dataflow.client.enable-dsl

Enable Data Flow DSL access. (Boolean, default: false)

spring.cloud.dataflow.client.server-uri

The Data Flow server URI. (String, default: http://localhost:9393)

spring.cloud.dataflow.client.skip-ssl-validation

Skip Ssl validation. (Boolean, default: true)

trigger.initial-delay

The initial delay in milliseconds. (Integer, default: 1000)

trigger.max-period

The maximum polling period in milliseconds. Will be set to period if period > maxPeriod. (Integer, default: 30000)

trigger.period

The polling period in milliseconds. (Integer, default: 1000)

Using the TaskLauncher

A tasklauncher is a sink that consumes LaunchRequest messages, as described above, and launches a task using the configured Spring Cloud Data Flow server (given by --dataflow.uri). The task launcher periodically polls its input for launch requests but will pause when the SCDF server’s concurrent task execution limit given by spring.cloud .dataflow.task .maximum-concurrent-tasks is reached (see the reference docs for more details).

When the number of running tasks drops below this limit message polling resumes. This is intended to prevent the SCDF deployer’s deployment platform from running out of resources under heavy task load. The poller is scheduled using a DynamicPeriodicTrigger. By default the polling rate is 1 second, but may be configured to any duration. When paused, or if there are no launch requests, the trigger period will increase, applying exponential backoff, up to a configured maximum (30 seconds by default).

Note
When the poller is paused it puts pressure on the message broker so some tuning will be necessary in extreme cases to balance resource utilization.

Build

$ ./mvnw clean install -PgenerateApps
$ cd apps

You can find the corresponding binder based projects here. You can then cd into one one of the folders and build it:

$ ./mvnw clean package

Examples

Register a task app and create a task, the timestamp sample provides a simple demonstration.

dataflow:>app register --name timestamp --type task --uri ...
dataflow:>stream create http | task-launcher-dataflow-sink --deploy

Send a launch request,

$curl http://localhost:<port> -H"Content-Type:application/json" -d '{"name":"timestamp"}'
dataflow:>task execution list
╔═════════╤══╤════════════════════════════╤════════════════════════════╤═════════╗
║Task Name│ID│         Start Time         │          End Time          │Exit Code║
╠═════════╪══╪════════════════════════════╪════════════════════════════╪═════════╣
║timestamp│1 │Fri Aug 10 08:48:05 EDT 2018│Fri Aug 10 08:48:05 EDT 2018│0        ║
╚═════════╧══╧════════════════════════════╧════════════════════════════╧═════════╝