-
Notifications
You must be signed in to change notification settings - Fork 87
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
First draft on Streaming Developer Guide
- Add Stream Introduction - Spring Cloud Stream & SCDF interactions - Binding properties - Binder properties - Standalone Stream development & testing - On RabbitMQ, Apache Kafka - Create OOTB stream - Creation of custom stream application - Continuous Delivery of streaming applications - Add stream features Resolves #3096 Resolves #3098 Resolves #3172 Resolves #3173
- Loading branch information
1 parent
f4d2f2a
commit d634b47
Showing
25 changed files
with
2,131 additions
and
244 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -2,6 +2,6 @@ | |
.cache/ | ||
node_modules/ | ||
yarn-error.log | ||
|
||
yarn.lock | ||
public/ | ||
content/files/ext/ | ||
content/files/ext/ |
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,11 +1,46 @@ | ||
--- | ||
path: 'concepts/what-are-streams/' | ||
title: 'What are streams' | ||
description: 'Lorem markdownum madefacta, circumtulit aliis, restabat' | ||
description: 'Concepts on Streaming pipelines' | ||
--- | ||
|
||
# What are streams | ||
|
||
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. | ||
A streaming pipeline is typically made of distributed applications that connect using a `messaging middleware` or a `streaming platform`. | ||
The streaming pipeline can be `linear` or `non-linear` based on the data flow through the distributed applications of the streaming pipeline. | ||
As a streaming application developer, you can focus on developing your streaming application’s business logic while delegating the plumbing of the application to the messaging middleware/streaming platform using the Spring Cloud Stream framework. | ||
|
||
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. | ||
## Spring Cloud Stream | ||
|
||
The streaming application can produce or consume events to/from the messaging middleware or the streaming platform. | ||
In Spring Cloud Stream, | ||
|
||
- the application’s endpoints which produce the events to the messaging middleware or the streaming platform represent the `output elements` | ||
- the application's endpoints which consume the events from the messaging middleware or the streaming platform represent the `input elements`. | ||
|
||
Spring Cloud Stream framework provides `@Input` and `@Output` annotations which you can use to qualify these input and output elements. | ||
|
||
### Spring Cloud Stream Binding | ||
|
||
When the Spring Cloud Stream application gets deployed, its `input` and `output` elements that are configured using the `@Input` and `@Output` annotations are bound to the messaging middleware or the streaming platform using the `binding` properties per destination on the messaging middleware or the streaming platform. | ||
|
||
#### Spring Cloud Stream Binding Properties | ||
|
||
Binding properties require `spring.cloud.stream.bindings` prefix. | ||
|
||
Currently, following properties are supported: | ||
|
||
- destination - the destination on the messaging middleware or the streaming platform (example: RabbitMQ exchange or Apache Kafka topic) | ||
- group - the consumer group name to be used for the application. Only for consumer applications. In Spring Cloud Data Flow, this will always be the `stream name`. | ||
- contentType - the content type to be used | ||
- binder - the name of the binder to use for the binding. This property is useful for multi binder use cases. | ||
- producer/consumer configuration properties - TBD | ||
|
||
#### Spring Cloud Stream Properties for Messaging Middleware | ||
|
||
Depending on the binder used for binding your application to the Messaging Middleware or the Streaming platform, you can provide the configuration properties for each binder. | ||
All these properties would take the prefix as `spring.cloud.stream.<binderName>.binder`. | ||
|
||
For instance, all the Apache Kafka binder related configuration properties have the prefix `spring.cloud.stream.kafka.binder` | ||
|
||
### Streams in Spring Cloud Data Flow |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
34 changes: 0 additions & 34 deletions
34
content/documentation/3-stream-developer-guides/1-getting-started/1-simple-stream
This file was deleted.
Oops, something went wrong.
105 changes: 105 additions & 0 deletions
105
...nt/documentation/3-stream-developer-guides/1-getting-started/1-simple-stream.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,105 @@ | ||
--- | ||
path: 'stream-developer-guides/getting-started/simple-stream' | ||
title: 'Simple Stream' | ||
description: 'Create and Deploy a simple streaming pipeline ' | ||
--- | ||
|
||
# Introduction | ||
|
||
Spring Cloud Data Flow provides a list of streaming applications that you can use out of the box and address some of the common streaming use cases. | ||
You can also extend from these out of the box applications or create your custom applications. | ||
All the out-of-the-box streaming applications are: | ||
|
||
- Available as Apache Maven artifacts or Docker images | ||
- Built with RabbitMQ or the Apache Kafka binder | ||
- Built with Prometheus and InfluxDB monitoring systems | ||
|
||
TODO: List all the out of the box applications | ||
|
||
## Registering the out of the box applications | ||
|
||
When registering the out of the box streaming applications, you can choose your artifact type `Maven` or `Docker` depending on the target platform. | ||
The `Local` development and the `CloudFoundry` platform support both `Maven` and `Docker` applications. | ||
The `Kubernetes` platform supports only `Docker` based applications. | ||
|
||
You can also choose the messaging middleware or the streaming platform: either `RabbitMQ` or `Apache Kafka`. | ||
|
||
### Registering applications using the Dashboard UI | ||
|
||
### Registering out of the box streaming applications using the Shell | ||
|
||
``` | ||
dataflow:> app import --uri http://bit.ly/Einstein-SR2-stream-applications-kafka-maven | ||
``` | ||
|
||
The above SCDF shell command registers the `Maven` artifacts of the out of the box streaming applications that can work on `RabbitMQ` messaging middleware. | ||
|
||
If you would like to register `docker` artifacts | ||
|
||
``` | ||
dataflow:> app import --uri http://bit.ly/Einstein-SR2-stream-applications-kafka-docker | ||
``` | ||
|
||
## Create the stream | ||
|
||
Spring Cloud Data Flow provides a DSL (Domain Specific Language) for creating a stream pipeline. | ||
The individual applications inside the streaming pipeline are connected via a `|` symbol. | ||
This pipe symbol is the logical representation of the messaging middleware or the streaming platform you would use to connect your applications in the streaming pipeline. | ||
|
||
### Stream pipeline configuration | ||
|
||
The stream pipeline can have configuration properties at | ||
|
||
- application level | ||
- deployer level | ||
|
||
The `application` properties are applied as the configuration for each individual application. | ||
The `application` properties can be set during the stream `creation` or the `deployment` time. | ||
When set during the stream `deployment`, these properties need to be prefixed with `app.<application-name>` | ||
|
||
The `deployer` properties are specific to the target deployment platform `Local`, `CloudFoundry` or `Kubernetes`. | ||
The `deployer` properties can be set only when deploying the stream. | ||
These properties need to be prefixed with `deployer.<application-name>` | ||
|
||
Let's create a stream that ingests incoming HTTP events (Source) into a logging application (Sink). | ||
|
||
The stream DSL `http --server.port=9000 | log` can represent the streaming pipeline that has `http` source application ingesting http events into `log` sink application. | ||
|
||
The property `server.port` is the `http` application properties set at the stream creation. | ||
|
||
## UI | ||
|
||
Screen shots here | ||
|
||
### Shell | ||
|
||
``` | ||
dataflow:> stream create http-ingest --definition "http --server.port=9000 | log" | ||
``` | ||
|
||
## Stream Deployment | ||
|
||
``` | ||
dataflow:> stream deploy http-ingest | ||
``` | ||
|
||
Once the stream is deployed and running, you can now post some `HTTP` events and see if they get displayed in the `log` application. | ||
|
||
``` | ||
dataflow:> http post --data "Happy streaming" --target http://localhost:9000 | ||
``` | ||
|
||
### Local Depoloyment | ||
|
||
Once the stream is deployed on `Local` development environment, you can look the runtime applications via Dashboard's runtime page or using the SCDF Shell command `runtime apps`. | ||
The runtime applications show information about where each application is running in the local environment and their log files locations. | ||
|
||
**NOTE** If you are running SCDF on docker, to access the log files of the streaming applications: | ||
|
||
`docker exec <stream-application-docker-container-id> tail <stream-application-log-file>` | ||
|
||
### Cloud Foundry | ||
|
||
### Kubernetes |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.