First draft on Streaming Developer Guide #7

ilayaperumalg · 2019-04-19T17:15:11Z

Add Stream Introduction
- Spring Cloud Stream & SCDF interactions
- Binding properties
- Binder properties
Standalone Stream development & testing
On RabbitMQ, Apache Kafka
Create OOTB stream
Creation of custom stream application
Continuous Delivery of streaming applications
Add stream features

Resolves spring-cloud/spring-cloud-dataflow#3096
Resolves spring-cloud/spring-cloud-dataflow#3098
Resolves spring-cloud/spring-cloud-dataflow#3172
Resolves spring-cloud/spring-cloud-dataflow#3173

.idea/workspace.xml

cppwfs · 2019-04-22T15:26:40Z

Can't pull down the PR because of the .idea/ directory as well as shown here:

error: The following untracked working tree files would be overwritten by merge:
	.idea/workspace.xml
Please move or remove them before you merge.

ilayaperumalg · 2019-04-22T15:30:24Z

@cppwfs Removed the .idea directory. Let me know if this works now.

cppwfs · 2019-04-22T15:49:13Z

@ilayaperumalg WOOHOO it works!

sabbyanandan

Just some general thoughts and suggestions.

content/documentation/1-installation/1-local.md

content/documentation/2-concepts/2-what-are-streams.md

content/documentation/3-stream-developer-guides/2-streams/1-standalone-stream-rabbitmq.md

content/documentation/3-stream-developer-guides/2-streams/3-data-flow-stream.md

content/documentation/3-stream-developer-guides/3-programming-models/3-kafka-streams.md

content/documentation/3-stream-developer-guides/4-continuous-delivery/1-cd-basics.md

content/documentation/2-concepts/2-what-are-streams.md

content/documentation/3-stream-developer-guides/1-getting-started/1-simple-stream.md

content/documentation/3-stream-developer-guides/2-streams/0-index.md

content/documentation/3-stream-developer-guides/2-streams/1-standalone-stream-rabbitmq.md

content/documentation/3-stream-developer-guides/2-streams/2-standalone-stream-kafka.md

content/documentation/3-stream-developer-guides/2-streams/1-standalone-stream-rabbitmq.md

content/documentation/3-stream-developer-guides/2-streams/2-standalone-stream-kafka.md

content/documentation/3-stream-developer-guides/2-streams/1-standalone-stream-rabbitmq.md

cppwfs · 2019-04-22T19:56:15Z

content/documentation/3-stream-developer-guides/2-streams/3-data-flow-stream.md

+### Application Registration
+
+Spring Cloud Data Flow server has an application registry to refer all the individual streaming applications.
+When you register an application, you would provide


Suffix with a colon

cppwfs · 2019-04-22T19:57:30Z

content/documentation/3-stream-developer-guides/2-streams/3-data-flow-stream.md

+
+Let's register the standalone applications we tested previously.
+
+NOTE: If you are running your Spring Cloud Data Flow server on docker environment, make sure the you have all your application artifact URIs are accessible.


server on docker environment should be server on the docker environment

cppwfs · 2019-04-22T19:58:48Z

content/documentation/3-stream-developer-guides/2-streams/3-data-flow-stream.md

+
+Let's register the standalone applications we tested previously.
+
+NOTE: If you are running your Spring Cloud Data Flow server on docker environment, make sure the you have all your application artifact URIs are accessible.


make sure the you have all your should be make sure that your

cppwfs · 2019-04-22T20:20:28Z

content/documentation/3-stream-developer-guides/2-streams/3-data-flow-stream.md

+
+Once the stream is deployed on `Local` development environment, you can look the runtime applications via Dashboard's runtime page or using the SCDF Shell command `runtime apps`.
+The runtime applications show information about where each application is running in the local environment and their log files locations.
+


Not sure if we need to show the log info here.
But if we do we need show them a screenshot of where to obtain the log file location from runtime apps and then an example bash shell of them catting or tailing the log.

cppwfs · 2019-04-22T20:25:09Z

content/documentation/3-stream-developer-guides/4-continuous-delivery/1-cd-basics.md


-https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#spring-cloud-dataflow-stream-lifecycle-update
+The above SCDF shell command registers the `Maven` artifacts of the out of the box streaming applications that can work on `RabbitMQ` messaging middleware.


This says rabbitmq, but the examples show kafka.

cppwfs · 2019-04-22T20:29:19Z

content/documentation/5-feature-guides/1-streams/0-index.md

@@ -6,3 +6,123 @@ summary: true
 ---

 Discuss the set of features that will be covered.


This line is causing the generated page to not render properly. Needs to be moved below the meta-tags.

cppwfs · 2019-04-26T18:02:44Z

@ilayaperumalg we need to remove the .idea files from the PR. The PR is failing to be pulled due to the following error:

error: The following untracked working tree files would be overwritten by merge:
	.idea/codeStyles/codeStyleConfig.xml
	.idea/dataflow.spring.io.iml
	.idea/misc.xml
	.idea/modules.xml
	.idea/vcs.xml
	.idea/workspace.xml
Please move or remove them before you merge.
Aborting

ilayaperumalg · 2019-04-27T02:42:14Z

@cppwfs Sorry, it was my mistake when I pushed the changes without noticing the .idea directory (though I added this into .gitignore). Pushed the changes to remove the .idea directory now.

cppwfs · 2019-04-27T04:08:32Z

No worries at all! Thanks for fixing!

…

Sent from my iPhone

On Apr 26, 2019, at 10:42 PM, Ilayaperumal Gopinathan ***@***.***> wrote: @cppwfs Sorry, it was my mistake when I pushed the changes without noticing the .idea directory (though I added this into .gitignore). Pushed the changes to remove the .idea directory now. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

- Add Stream Introduction - Spring Cloud Stream & SCDF interactions - Binding properties - Binder properties - Standalone Stream development & testing - On RabbitMQ, Apache Kafka - Create OOTB stream - Creation of custom stream application - Continuous Delivery of streaming applications - Add stream features Resolves #3096 Resolves #3098 Resolves #3172 Resolves #3173

cppwfs · 2019-04-29T12:57:15Z

content/documentation/pages/3-stream-developer-guides/2-streams/1-standalone-stream-rabbitmq.md

+
+1. Visit the [Spring Initialzr site](https://start.spring.io/).
+1. Select the latest release of spring boot.
+1. Create a new Maven project with a Group name of `com.example` and an Artifact name of `usage-detail-sender`.


instead of com.example we've been using io.spring

cppwfs · 2019-04-29T13:12:01Z

content/documentation/pages/3-stream-developer-guides/1-getting-started/1-simple-stream

@@ -1,34 +1,110 @@
 ---
 path: 'stream-developer-guides/getting-started/simple-stream'
-title: 'Simple Stream '
-description: 'Simple Stream '
+title: 'Simple Stream'


The file 1-simple-stream file is missing the .md extension and is not getting rendered on the site.

cppwfs · 2019-04-29T13:14:55Z

content/documentation/pages/3-stream-developer-guides/1-getting-started/1-simple-stream

+If you would like to register `docker` artifacts
+
+```
+dataflow:> app import --uri  http://bit.ly/Einstein-SR2-stream-applications-rabbit-docker


Need to discuss if we want UI images instead or in addition to shell commands @markpollack @sabbyanandan .

cppwfs · 2019-04-29T13:19:19Z

content/documentation/pages/3-stream-developer-guides/1-getting-started/1-simple-stream

+### Registering out of the box streaming applications using the Shell
+
+```
+dataflow:> app import --uri  http://bit.ly/Einstein-SR2-stream-applications-kafka-maven


The app import should use RabbitMQ instead of Kafka, so that it matches the text and example below.

cppwfs · 2019-04-29T13:38:43Z

content/documentation/pages/3-stream-developer-guides/2-streams/1-standalone-stream-rabbitmq.md

+
+#### Biz Logic
+
+1.  In your favorite IDE create the `com.example.demo.domain` package


demo should be usagedetailsender. Based on the initializr instructions.

These changes need to be made globally in this sample.

cppwfs · 2019-04-29T17:38:07Z

content/documentation/pages/3-stream-developer-guides/2-streams/1-standalone-stream-rabbitmq.md

+You can run the standalone `UsageCostLogger` sink application as,
+
+```
+java -jar usage-cost-logger-0.0.1-SNAPSHOT.jar --spring.cloud.stream.bindings.input.destination=test-usage-cost --server.port=9092


app started and then stopped, using the instructions above.

the pom has kafka in it as well....not rabbit. There needs to be another directory level, one for kafka, one for rabbit, stream-developer-guides/streams/standalone-stream-kafka and stream-developer-guides/streams/standalone-stream-rabbitmq

Did you add the code from the biz logic step?

The link for UsagecostDetail is also wrong.

yes.

package io.spring.dataflow.sample; import io.spring.dataflow.sample.domain.UsageCostDetail; import io.spring.dataflow.sample.domain.UsageDetail; import org.springframework.cloud.stream.annotation.EnableBinding; import org.springframework.cloud.stream.annotation.StreamListener; import org.springframework.cloud.stream.messaging.Processor; import org.springframework.messaging.handler.annotation.SendTo; @EnableBinding(Processor.class) public class UsageCostProcessor { private double ratePerSecond = 0.1; private double ratePerMB = 0.05; @StreamListener(Processor.INPUT) @SendTo(Processor.OUTPUT) public UsageCostDetail processUsageCost(UsageDetail usageDetail) { UsageCostDetail usageCostDetail = new UsageCostDetail(); usageCostDetail.setUserId(usageDetail.getUserId()); usageCostDetail.setCallCost(usageDetail.getDuration() * this.ratePerSecond); usageCostDetail.setDataCost(usageDetail.getData() * this.ratePerMB); return usageCostDetail; } }

I refactored the code by moving the package in Github but missed to update the link. will fix.

cppwfs · 2019-04-29T17:39:54Z

content/documentation/pages/3-stream-developer-guides/2-streams/2-standalone-stream-kafka.md

+
+Another option instead of using the UI to initialize your project you can do the following:
+
+1. Click the [here](https://start.spring.io/starter.zip?fakeusernameremembered=&fakepasswordremembered=&type=maven-project&language=java&bootVersion=2.1.4.RELEASE&baseDir=usage-detail-sender&groupId=io.spring.dataflow.sample&artifactId=usage-detail-sender&name=usage-detail-sender&description=Sample+project+for+Spring+Cloud+Stream+Source&packageName=io.spring.dataflow.sample&packaging=jar&javaVersion=1.8&inputSearch=&style=amqp&style=cloud-stream) to download the preconfigured usage-detail-sender.zip.


Link needs to be updated using kafka instead of AMQ.

cppwfs · 2019-04-29T17:40:52Z

content/documentation/pages/3-stream-developer-guides/2-streams/2-standalone-stream-kafka.md

+
+#### Testing
+
+We can individually test these custom applications before creating a pipeline using Spring Cloud Data Flow.


Needs to be updated for Kafka

cppwfs · 2019-04-29T17:41:29Z

content/documentation/pages/3-stream-developer-guides/2-streams/2-standalone-stream-kafka.md

+
+1. Visit the [Spring Initialzr site](https://start.spring.io/).
+1. Select the latest release of spring boot.
+1. Create a new Maven project with a Group name of `com.example` and an Artifact name of `usage-cost-processor`.


com.example needs to be io.spring.dataflow.sample

cppwfs · 2019-04-29T17:41:57Z

content/documentation/pages/3-stream-developer-guides/2-streams/2-standalone-stream-kafka.md

@@ -0,0 +1,222 @@
+---
+path: 'stream-developer-guides/streams/standalone-stream-kafka/'


Update instructions to be Kafka instead of RabbitMQ.

markpollack · 2019-04-30T23:49:45Z

merged as 7cfa4f2

ilayaperumalg force-pushed the streaming-developer-guide branch from a497b51 to d634b47 Compare April 19, 2019 18:07

cppwfs reviewed Apr 22, 2019

View reviewed changes

.idea/workspace.xml Outdated Show resolved Hide resolved

sabbyanandan reviewed Apr 22, 2019

View reviewed changes

cppwfs reviewed Apr 22, 2019

View reviewed changes

cppwfs self-assigned this Apr 23, 2019

ilayaperumalg force-pushed the streaming-developer-guide branch from be03cfd to 5eca5d2 Compare April 26, 2019 17:37

ilayaperumalg added 4 commits April 29, 2019 08:29

Address review comments

65d0f1b

Remove .idea directory

eced75a

Polishing stream developer guides

67acab8

ilayaperumalg force-pushed the streaming-developer-guide branch from bcf1019 to 67acab8 Compare April 29, 2019 14:47

cppwfs reviewed Apr 29, 2019

View reviewed changes

Update standalone app configuration/steps

666da15

sabbyanandan assigned markpollack and unassigned cppwfs Apr 30, 2019

markpollack closed this Apr 30, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

First draft on Streaming Developer Guide #7

First draft on Streaming Developer Guide #7

ilayaperumalg commented Apr 19, 2019 •

edited

cppwfs commented Apr 22, 2019

ilayaperumalg commented Apr 22, 2019

cppwfs commented Apr 22, 2019

sabbyanandan left a comment

cppwfs Apr 22, 2019

cppwfs Apr 22, 2019

cppwfs Apr 22, 2019

cppwfs Apr 22, 2019

cppwfs Apr 22, 2019

cppwfs Apr 22, 2019

cppwfs commented Apr 26, 2019

ilayaperumalg commented Apr 27, 2019

cppwfs commented Apr 27, 2019 via email

cppwfs Apr 29, 2019

cppwfs Apr 29, 2019

cppwfs Apr 29, 2019

cppwfs Apr 29, 2019

cppwfs Apr 29, 2019

cppwfs Apr 29, 2019

cppwfs Apr 29, 2019

markpollack Apr 29, 2019

markpollack Apr 29, 2019

markpollack Apr 29, 2019

cppwfs Apr 29, 2019

ilayaperumalg Apr 30, 2019

cppwfs Apr 29, 2019

cppwfs Apr 29, 2019

cppwfs Apr 29, 2019

cppwfs Apr 29, 2019

markpollack commented Apr 30, 2019


		Let's register the standalone applications we tested previously.

		NOTE: If you are running your Spring Cloud Data Flow server on docker environment, make sure the you have all your application artifact URIs are accessible.


		Once the stream is deployed on `Local` development environment, you can look the runtime applications via Dashboard's runtime page or using the SCDF Shell command `runtime apps`.
		The runtime applications show information about where each application is running in the local environment and their log files locations.


		https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#spring-cloud-dataflow-stream-lifecycle-update
		The above SCDF shell command registers the `Maven` artifacts of the out of the box streaming applications that can work on `RabbitMQ` messaging middleware.

		@@ -6,3 +6,123 @@ summary: true
		---

		Discuss the set of features that will be covered.


		#### Biz Logic

		1. In your favorite IDE create the `com.example.demo.domain` package


		Another option instead of using the UI to initialize your project you can do the following:

		1. Click the [here](https://start.spring.io/starter.zip?fakeusernameremembered=&fakepasswordremembered=&type=maven-project&language=java&bootVersion=2.1.4.RELEASE&baseDir=usage-detail-sender&groupId=io.spring.dataflow.sample&artifactId=usage-detail-sender&name=usage-detail-sender&description=Sample+project+for+Spring+Cloud+Stream+Source&packageName=io.spring.dataflow.sample&packaging=jar&javaVersion=1.8&inputSearch=&style=amqp&style=cloud-stream) to download the preconfigured usage-detail-sender.zip.


		#### Testing

		We can individually test these custom applications before creating a pipeline using Spring Cloud Data Flow.

		@@ -0,0 +1,222 @@
		---
		path: 'stream-developer-guides/streams/standalone-stream-kafka/'

First draft on Streaming Developer Guide #7

First draft on Streaming Developer Guide #7

Conversation

ilayaperumalg commented Apr 19, 2019 • edited

cppwfs commented Apr 22, 2019

ilayaperumalg commented Apr 22, 2019

cppwfs commented Apr 22, 2019

sabbyanandan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cppwfs commented Apr 26, 2019

ilayaperumalg commented Apr 27, 2019

cppwfs commented Apr 27, 2019 via email

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

markpollack commented Apr 30, 2019

ilayaperumalg commented Apr 19, 2019 •

edited