Skip to content

Commit

Permalink
First draft on Streaming Developer Guide
Browse files Browse the repository at this point in the history
 - Add Stream Introduction
   - Spring Cloud Stream & SCDF interactions
   - Binding properties
   - Binder properties

 - Standalone Stream development & testing
  - On RabbitMQ, Apache Kafka

 - Create OOTB stream
 - Creation of custom stream application

 - Continuous Delivery of streaming applications

 - Add stream features
  • Loading branch information
ilayaperumalg committed Apr 19, 2019
1 parent f4d2f2a commit a497b51
Show file tree
Hide file tree
Showing 25 changed files with 2,131 additions and 244 deletions.
4 changes: 2 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,6 @@
.cache/
node_modules/
yarn-error.log

yarn.lock
public/
content/files/ext/
content/files/ext/
882 changes: 882 additions & 0 deletions .idea/workspace.xml

Large diffs are not rendered by default.

6 changes: 5 additions & 1 deletion content/documentation/1-installation/0-index.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,8 @@ summary: true

# Installation

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
This guide helps you to install and get started with Spring Cloud Data Flow on

- Local
- Cloud Foundry
- Kubernetes
2 changes: 1 addition & 1 deletion content/documentation/1-installation/1-local.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
path: 'installation/local/'
title: 'Local'
description: 'Getting Started Data Flow using the local server'
description: 'Installation guide on local development environment'
---

# Installation Local
Expand Down
2 changes: 1 addition & 1 deletion content/documentation/1-installation/2-cloudfoundry.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
path: 'installation/cloudfoundry/'
title: 'Cloud Foundry'
description: 'Lorem markdownum madefacta, circumtulit aliis, restabat'
description: 'Installation guide on Cloud Foundry'
---

# Installation Cloud Foundry
Expand Down
2 changes: 1 addition & 1 deletion content/documentation/1-installation/3-kubernetes.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
path: 'installation/kubernetes/'
title: 'Kubernetes'
description: 'Lorem markdownum madefacta, circumtulit aliis, restabat'
description: 'Installation guide on Kubernetes'
---

# Installation- Kubernetes
Expand Down
41 changes: 38 additions & 3 deletions content/documentation/2-concepts/2-what-are-streams.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,46 @@
---
path: 'concepts/what-are-streams/'
title: 'What are streams'
description: 'Lorem markdownum madefacta, circumtulit aliis, restabat'
description: 'Concepts on Streaming pipelines'
---

# What are streams

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
A streaming pipeline is typically made of distributed applications that connect using a `messaging middleware` or a `streaming platform`.
The streaming pipeline can be `linear` or `non-linear` based on the data flow through the distributed applications of the streaming pipeline.
As a streaming application developer, you can focus on developing your streaming application’s business logic while delegating the plumbing of the application to the messaging middleware/streaming platform using the Spring Cloud Stream framework.

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
## Spring Cloud Stream

The streaming application can produce or consume events to/from the messaging middleware or the streaming platform.
In Spring Cloud Stream,

- the application’s endpoints which produce the events to the messaging middleware or the streaming platform represent the `output elements`
- the application's endpoints which consume the events from the messaging middleware or the streaming platform represent the `input elements`.

Spring Cloud Stream framework provides `@Input` and `@Output` annotations which you can use to qualify these input and output elements.

### Spring Cloud Stream Binding

When the Spring Cloud Stream application gets deployed, its `input` and `output` elements that are configured using the `@Input` and `@Output` annotations are bound to the messaging middleware or the streaming platform using the `binding` properties per destination on the messaging middleware or the streaming platform.

#### Spring Cloud Stream Binding Properties

Binding properties require `spring.cloud.stream.bindings` prefix.

Currently, following properties are supported:

- destination - the destination on the messaging middleware or the streaming platform (example: RabbitMQ exchange or Apache Kafka topic)
- group - the consumer group name to be used for the application. Only for consumer applications. In Spring Cloud Data Flow, this will always be the `stream name`.
- contentType - the content type to be used
- binder - the name of the binder to use for the binding. This property is useful for multi binder use cases.
- producer/consumer configuration properties - TBD

#### Spring Cloud Stream Properties for Messaging Middleware

Depending on the binder used for binding your application to the Messaging Middleware or the Streaming platform, you can provide the configuration properties for each binder.
All these properties would take the prefix as `spring.cloud.stream.<binderName>.binder`.

For instance, all the Apache Kafka binder related configuration properties have the prefix `spring.cloud.stream.kafka.binder`

### Streams in Spring Cloud Data Flow
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,4 @@ summary: true

# Stream Processing

Discuss at a high level what is covered in the following guides.
In this getting started guide, you can see how a simple streaming pipeline can be created using out-of-the-box streaming applications from Spring Cloud Data Flow.

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
---
path: 'stream-developer-guides/getting-started/simple-stream'
title: 'Simple Stream'
description: 'Create and Deploy a simple streaming pipeline '
---

# Introduction

Spring Cloud Data Flow provides a list of streaming applications that you can use out of the box and address some of the common streaming use cases.
You can also extend from these out of the box applications or create your custom applications.
All the out-of-the-box streaming applications are:

- Available as Apache Maven artifacts or Docker images
- Built with RabbitMQ or the Apache Kafka binder
- Built with Prometheus and InfluxDB monitoring systems

TODO: List all the out of the box applications

## Registering the out of the box applications

When registering the out of the box streaming applications, you can choose your artifact type `Maven` or `Docker` depending on the target platform.
The `Local` development and the `CloudFoundry` platform support both `Maven` and `Docker` applications.
The `Kubernetes` platform supports only `Docker` based applications.

You can also choose the messaging middleware or the streaming platform: either `RabbitMQ` or `Apache Kafka`.

### Registering applications using the Dashboard UI

### Registering out of the box streaming applications using the Shell

```
dataflow:> app import --uri http://bit.ly/Einstein-SR2-stream-applications-kafka-maven
```

The above SCDF shell command registers the `Maven` artifacts of the out of the box streaming applications that can work on `RabbitMQ` messaging middleware.

If you would like to register `docker` artifacts

```
dataflow:> app import --uri http://bit.ly/Einstein-SR2-stream-applications-kafka-docker
```

## Create the stream

Spring Cloud Data Flow provides a DSL (Domain Specific Language) for creating a stream pipeline.
The individual applications inside the streaming pipeline are connected via a `|` symbol.
This pipe symbol is the logical representation of the messaging middleware or the streaming platform you would use to connect your applications in the streaming pipeline.

### Stream pipeline configuration

The stream pipeline can have configuration properties at

- application level
- deployer level

The `application` properties are applied as the configuration for each individual application.
The `application` properties can be set during the stream `creation` or the `deployment` time.
When set during the stream `deployment`, these properties need to be prefixed with `app.<application-name>`

The `deployer` properties are specific to the target deployment platform `Local`, `CloudFoundry` or `Kubernetes`.
The `deployer` properties can be set only when deploying the stream.
These properties need to be prefixed with `deployer.<application-name>`

Let's create a stream that ingests incoming HTTP events (Source) into a logging application (Sink).

The stream DSL `http --server.port=9000 | log` can represent the streaming pipeline that has `http` source application ingesting http events into `log` sink application.

The property `server.port` is the `http` application properties set at the stream creation.

## UI

Screen shots here

### Shell

```
dataflow:> stream create http-ingest --definition "http --server.port=9000 | log"
```

## Stream Deployment

```
dataflow:> stream deploy http-ingest
```

Once the stream is deployed and running, you can now post some `HTTP` events and see if they get displayed in the `log` application.

```
dataflow:> http post --data "Happy streaming" --target http://localhost:9000
```

### Local Depoloyment

Once the stream is deployed on `Local` development environment, you can look the runtime applications via Dashboard's runtime page or using the SCDF Shell command `runtime apps`.
The runtime applications show information about where each application is running in the local environment and their log files locations.

**NOTE** If you are running SCDF on docker, to access the log files of the streaming applications:

`docker exec <stream-application-docker-container-id> tail <stream-application-log-file>`

### Cloud Foundry

### Kubernetes
Original file line number Diff line number Diff line change
Expand Up @@ -7,5 +7,7 @@ summary: true

# Stream Processing

Discuss at a high level what is covered in the following guides.
We start off using only Spring Cloud Stream and then introduce Spring Cloud Data Flow.
In this stream processing guide, you will see how you can

- develop, test and deploy an individual streaming application
- create a streaming pipeline using those individual applications in Spring Cloud Data Flow

0 comments on commit a497b51

Please sign in to comment.