Name	Name	Last commit message	Last commit date
parent directory ..
src	src
README.adoc	README.adoc
pom.xml	pom.xml

Spring Cloud Data Flow Server

The Data Flow Server deploys Streams by delegating to Spring Cloud Skipper and deploys Tasks directly. Both Skipper and the Data Flow server can deploy to the local machine, Cloud Foundry, and Kubernetes.

Note	The local version of the Data Flow server should only be used for Stream development but can be used in a production environment for Task deployment as a replacement for the Spring Cloud Batch Admin Server.

Getting Started

This shows how to start the Data Flow Server and Shell to create the time | log stream.

Start the Data Flow Server and Shell

Start Kafka locally (e.g. kafka-server-start.sh /usr/local/etc/kafka/server.properties)
Build from the spring-cloud-dataflow root directory:
```
./mvnw clean install
```

Start the Data Flow Server application:

java -jar spring-cloud-dataflow-server/target/spring-cloud-dataflow-server-<version>.jar

Start the shell

$ java -jar spring-cloud-dataflow-shell/target/spring-cloud-dataflow-shell-<version>.jar

Creating the `time | log` stream:

Create the 'ticktock' stream:

dataflow:>stream create --name ticktock --definition "time | log"
Created new stream 'ticktock'

This is equivalent to the following HTTP POST request

$ curl -X POST -d "name=ticktock&definition=time | log" http://localhost:9393/streams/definitions?deploy=false

List all streams available in the repository:

dataflow:>stream list
╔═══════════╤═════════════════╤══════════╗
║Stream Name│Stream Definition│  Status  ║
╠═══════════╪═════════════════╪══════════╣
║ticktock   │time | log       │undeployed║
╚═══════════╧═════════════════╧══════════╝

This is equivalent to the following HTTP get request

$ curl http://localhost:9393/streams/definitions

Deploy the 'ticktock' stream:

dataflow:>stream deploy --name ticktock
Deployed stream 'ticktock'

This is equivalent to the following HTTP get request

$ curl -X POST http://localhost:9393/streams/deployments/ticktock

If successful you should see output similar to the following in the Data Flow Server console:

...o.s.c.d.spi.local.LocalAppDeployer    : deploying app ticktock.log instance 0
   Logs will be in /some/path/ticktock.log

If you tail the stdout_0.log file from the directory mentioned, you should see output similar to the following:

2016-04-26 15:10:18.320  INFO 59890 --- [pool-1-thread-1] log.sink    : 04/26/16 15:10:18
2016-04-26 15:10:19.322  INFO 59890 --- [pool-1-thread-1] log.sink    : 04/26/16 15:10:19
2016-04-26 15:10:20.322  INFO 59890 --- [pool-1-thread-1] log.sink    : 04/26/16 15:10:20

Configuration

Default

To configure the Data Flow Server you can follow the configuration setup guidelines specified in the boot documentation found here

Note: The dataflow-server.yml containing the defaults can be found here

Spring Cloud Configuration

The Spring Cloud Data Flow Server offers the user the ability to configure properties via spring-cloud-config. All configurations retrieved from the cloud config will take precedence over Boot’s defaults enumerated above. The Spring Cloud Data Flow Server will look for the server at localhost:8888, however this can be overwritten by setting the spring.cloud.config.uri property to the desired url.

Cloud-Config-Server configuration

To specify a repository in the cloud config server configuration.yml for the Data Flow Server, setup a repo profile with the pattern spring-cloud-dataflow-server. For example:

spring:
  cloud:
     config:
       server:
         git:
           uri: https://github.com/myrepo/configurations
           repos:
            spring-cloud-dataflow-server:
              pattern: spring-cloud-dataflow-server
              uri: https://github.com/myrepo/configurations
              searchPaths: dataFlowServer

Fail Fast

In some cases, it may be desirable to fail startup of a service if it cannot connect to the Config Server. If this is the desired behavior, set the bootstrap configuration property spring.cloud.config.failFast=true and the client will halt with an Exception.

Note

If the Data Flow Server cannot connect to the cloud config server, the following warning message will be logged:

`WARN 42924 --- [main] c.c.c.ConfigServicePropertySourceLocator : Could not locate PropertySource: I/O error on GET request for "http://localhost:8888/spring-cloud-dataflow-server/default":Connection refused; nested exception is java.net.ConnectException: Connection refused`

To disable the cloud config server set the spring.cloud.config.enabled property to false.

Docker-Compose Integration Tests

Note	The docker-compose files have been decomposed so that you have to specify the database and messaging engine separately. Previously mysql and kafka was the default.

To run the docker compose integration, enable the -Pfailsafe maven profile and the -Dgroups="docker-compose" JUnit tag.

You can run just the integration tests like this:

./mvnw clean test-compile failsafe:integration-test -pl spring-cloud-dataflow-server -Pfailsafe -Dgroups="docker-compose"

With the help of the TestProperties properties one can change the docker-compose files used, the exact versions of the SCDF, Skipper servers installed, or the versions Stream and Task apps:

./mvnw clean test-compile failsafe:integration-test -pl spring-cloud-dataflow-server -Pfailsafe -Dgroups="docker-compose" \
   -Dtest.docker.compose.paths="../src/docker-compose/docker-compose.yml,../src/docker-compose/docker-compose-influxdb.yml,../src/docker-compose/docker-compose-postgres.yml,../src/docker-compose/docker-compose-rabbitmq.yml" \
   -Dtest.docker.compose.stream.apps.uri=https://dataflow.spring.io/rabbitmq-maven-5-0-x \
   -Dtest.docker.compose.dataflow.version=2.8.0-SNAPSHOT \
   -Dtest.docker.compose.skipper.version=2.7.0-SNAPSHOT \

The test.docker.compose.paths property accepts comma separated list of docker compose file names and supports file:, classpath:, and http:/https: URI schemas. If the schema prefix is not explicitly set, the file is treated as local one.

./mvnw clean test-compile integration-test -pl spring-cloud-dataflow-server -Pfailsafe -Dgroups="docker-compose" \
   -Dtest.docker.compose.paths=""../src/docker-compose/docker-compose.yml,\
    ../src/docker-compose/docker-compose-mysql.yml, \
    ../src/docker-compose/docker-compose-kafka.yml, \
    ../src/docker-compose/docker-compose-dood.yml, \
    ../src/docker-compose/docker-compose-prometheus.yml" \
   -Dtest.docker.compose.stream.apps.uri=https://dataflow.spring.io/kafka-docker-5-0-x \
   -Dtest.docker.compose.task.apps.uri=https://dataflow.spring.io/task-docker-3-0-x \
   -Dtest.docker.compose.dataflow.version=2.8.0-SNAPSHOT \
   -Dtest.docker.compose.skipper.version=2.7.0-SNAPSHOT \
   -Dtest.docker.compose.apps.port.range=80 \
   -Dtest.docker.compose.pullOnStartup=false

Use the -Dtest.docker.compose.dood=true property to activate the DooD (Docker-Out-Of-Docker) shortcut that implicitly will add docker-compose-dood.yml to your configuration :

./mvnw clean test-compile integration-test -pl spring-cloud-dataflow-server -Pfailsafe -Dgroups="docker-compose" \
   -Dtest.docker.compose.dood=true \
   -Dtest.docker.compose.pullOnStartup=false

Use the -Dit.test=class-name#method-name to filter only selected tests.

Run Integration Tests with HTTPS connections

./mvnw clean test-compile integration-test -pl spring-cloud-dataflow-server -Pfailsafe -Dgroups="docker-compose" \
   -Dtest.docker.compose.paths=""../src/docker-compose/docker-compose.yml,\
    ../src/docker-compose/docker-compose-mysql.yml, \
    ../src/docker-compose/docker-compose-kafka.yml, \
    ../src/docker-compose/docker-compose-ssl.yml" \
   -Dtest.docker.compose.dataflow.uri=https://dataflow-server:9393 \
   -Dtest.docker.compose.skipper.uri=https://skipper-server:7577 \
   -Dtest.docker.compose.waiting.for.service.format=https://$HOST:$EXTERNAL_PORT \
   -Dit.test=DataFlowIT#streamPartitioning \
   -Dspring.cloud.dataflow.client.server-uri=https://localhost:9393 \
   -Dspring.cloud.dataflow.client.skip-ssl-validation=true \
   -Dtest.platform.connection.application-over-https=true

Use the following env. variables to run an IT test from IDE:

TEST_DOCKER_COMPOSE_PATHS=../src/docker-compose/docker-compose.yml,\
    ../src/docker-compose/docker-compose-mysql.yml, \
    ../src/docker-compose/docker-compose-kafka.yml, \
    ../src/docker-compose/docker-compose-ssl.yml
TEST_DOCKER_COMPOSE_SKIPPER_URI=https://skipper-server:7577
TEST_DOCKER_COMPOSE_DATAFLOW_URI=https://dataflow-server:9393
TEST_DOCKER_COMPOSE_WAITING_FOR_SERVICE_FORMAT=https://$HOST:$EXTERNAL_PORT
TEST_PLATFORM_CONNECTION_APPLICATIONOVERHTTPS=true
SPRING_CLOUD_DATAFLOW_CLIENT_SKIPSSLVALIDATION=true
SPRING_CLOUD_DATAFLOW_CLIENT_SERVERURI=https://localhost:9393
COMPOSE_HTTP_TIMEOUT=300

Use the following env. variables to run an IT test from IDE with existing docker-compose cluster:

TEST_DOCKER_COMPOSE_DISABLE_EXTENSION=true
SPRING_CLOUD_DATAFLOW_CLIENT_SERVERURI=https://localhost:9393
SPRING_CLOUD_DATAFLOW_CLIENT_SKIPSSLVALIDATION=true
TEST_PLATFORM_CONNECTION_APPLICATIONOVERHTTPS=true

Create HTTPS docker-compose from shell:

export DATAFLOW_URI=https://dataflow-server:9393
export SKIPPER_URI=https://skipper-server:7577
docker-compose -f ./src/docker-compose/docker-compose.yml \
               -f ./src/docker-compose/docker-compose-mysql.yml, \
               -f ./src/docker-compose/docker-compose-kafka.yml, \
               -f ./src/docker-compose/docker-compose-ssl.yml

Run Integration Tests against k8s

For this you need a pre-installed Data Flow on GKE or minikube. Register the OOTB kafka/docker and task/docker apps and the run:

./mvnw integration-test -pl spring-cloud-dataflow-server -Dtest=foo -DfailIfNoTests=false -Dgroups="docker-compose" \
    -Dtest.docker.compose.disable.extension=true \
    -Dspring.cloud.dataflow.client.server-uri=http://your-k8s-scdf-server \
    -Pfailsafe

replace http://your-k8s-scdf-server with the url of your Data Flow server.
the test.docker.compose.disable.extension=true property disables the docker-compose fixture.
the spring.cloud.dataflow.client.server-uri= property points to the pre-installed DataFlow REST API.

For the analytic tests you need to pre-install Prometheus according to the installation instructions and forward the prometheus server’s 9090 port

kubectl port-forward prometheus-67896dcc8-7wm4h 9090:9090

Replace the prometheus-67896dcc8-7wm4h with your pod name.

Or use the test.platform.connection.prometheusUrl property to set an alternative Prometheus url.

Integration Tests with Minikube

The Integration Tests require that the Kubernetes cluster running Data Flow provides a Load Balancer service.

Most cloud providers offer the Load Balancer service by default. For Minikube, you would need to install one yourself. The MetalLB is a lightweight load balancer implementation for Kubernetes and is one option to provide the LoadBalancer service on Minikube.

Install MetalLB

kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.8.3/manifests/metallb.yaml

Create a ConfigMap with the appropriate settings

cat <<EOF >metallbcm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.99.116/28
EOF

Note	The IP in the addresses section is based on running Minikube via the default VirtualBox driver with default Minikube IP is `192.168.99.100`. For different Minikube configurations, you might need to customize the address.

Apply the ConfigMap:

kubectl apply -f metallbcm.yaml

At this point MetalLB is setup and ready to hand out IP’s to any service of type LoadBalancer, just as things would work in say GKE.

If a change is made to the ConfigMap and re-applied, bounce the pods in the metallb-system namespace to pick-up the new changes, ie:

kubectl delete pod --all -n metallb-system

Testcontainers Integration Tests

Some database and security tests are executed in containers. There are a set of junit tags which can be used to disect which tests to run. Here are some examples how to run those locally.

Run oauth security tests:

./mvnw integration-test -pl spring-cloud-dataflow-server -Dgroups=oauth -Pfailsafe

Run database tests:

./mvnw integration-test -pl spring-cloud-dataflow-server -Dgroups=database -Pfailsafe

Run mariadb, postgres or mssql tests:

./mvnw integration-test -pl spring-cloud-dataflow-server -Dgroups=mariadb -Pfailsafe
./mvnw integration-test -pl spring-cloud-dataflow-server -Dgroups=postgres -Pfailsafe
./mvnw integration-test -pl spring-cloud-dataflow-server -Dgroups=mssql -Pfailsafe

Run mariadb tests with shared database and with latest dataflow:

./mvnw integration-test -pl spring-cloud-dataflow-server -Dgroups="mariadb&database-shared&dataflow_main" -Pfailsafe

Run database and oauth tests:

./mvnw integration-test -pl spring-cloud-dataflow-server -Dgroups="database|oauth" -Pfailsafe

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Files

spring-cloud-dataflow-server

spring-cloud-dataflow-server

README.adoc

Spring Cloud Data Flow Server

Getting Started

Start the Data Flow Server and Shell

Creating the `time | log` stream:

Configuration

Default

Spring Cloud Configuration

Cloud-Config-Server configuration

Fail Fast

Note

Docker-Compose Integration Tests

Run Integration Tests with HTTPS connections

Run Integration Tests against k8s

Integration Tests with Minikube

Testcontainers Integration Tests

Files

spring-cloud-dataflow-server

Directory actions

More options

Directory actions

More options

Latest commit

History

spring-cloud-dataflow-server

Folders and files

parent directory

README.adoc

Spring Cloud Data Flow Server

Getting Started

Start the Data Flow Server and Shell

Creating the time | log stream:

Configuration

Default

Spring Cloud Configuration

Cloud-Config-Server configuration

Fail Fast

Note

Docker-Compose Integration Tests

Run Integration Tests with HTTPS connections

Run Integration Tests against k8s

Integration Tests with Minikube

Testcontainers Integration Tests

Creating the `time | log` stream: