Kamir patch 2

[kamir]

I have added the word_count.sh and word_count_kafka.sh example applications in the new module named wayang-applications.

This PR allows me to close this contribution with a partial result:
A working example of an application using Kafka Source and Kafka Sink in on JavaPlan in Wayang.

Moving forward, I will start with a new feature branch in a forked repository, to keep the main repository clean in the future.
The branch can be deleted after the merge.

[github-actions]

license-eye has checked 1554 files.

Valid	Invalid	Ignored	Fixed
1420	1	133	0

Click to see the invalid file list

• wayang-applications/bin/bootstrap.sh

Use this command to fix any missing license headers

```bash

docker run -it --rm -v $(pwd):/github/workspace apache/skywalking-eyes header fix

</details>

[juripetersen]

@kamir do you think that creating a wayang-kafka module inside of wayang-platforms would be a cleaner approach to add these operators? Right now it seems like you edit both wayang-java and wayang-spark, where both of them contain versions of operators from Kafka. I think conceptually, having all implementations for the specific apis in wayang-kafka would be neat. Then you could have wayang-kafka/src/main/java/JavaKafkaOperator wayang-kafka/src/main/java/SparkKafkaOperator etc and the respective platforms themselves don't have to concern about Kafka.

What do you think?

[zkaoudi]

That's an interesting idea @juripetersen. If we go that direction, we should do for all data sources, right? maybe a parent module wayang-datasources?

[kamir]

@kamir do you think that creating a wayang-kafka module inside of wayang-platforms would be a cleaner approach to add these operators? Right now it seems like you edit both wayang-java and wayang-spark, where both of them contain versions of operators from Kafka. I think conceptually, having all implementations for the specific apis in wayang-kafka would be neat. Then you could have wayang-kafka/src/main/java/JavaKafkaOperator wayang-kafka/src/main/java/SparkKafkaOperator etc and the respective platforms themselves don't have to concern about Kafka.

What do you think?

Yes, this is a great idea. I was thinking about the issue of "double implementation" but did not make so far towards a solution.
Yes, lets do this.

To get ready for this task I suggest, that I first finish the demo, and the blog.
We all together nail the release 1.0.
And after we reached the 1.0 level, we make this restructuring and release it as 1.1.

What do you think?

[juripetersen]

I'd say if that is the pragmatic solution, let's go for it. But knowing that the restructure won't happen for a long time, let's try to create an issue addressing it so we will resolve that technical debt eventually.

[kamir]

Issue is created:

#469

[zkaoudi]

But just to make sure we are all on the same page: Is the code for a Kafka source operator and a Java source operator the same? For example, for the TextFileSource it's not the same code.

[kamir]

The SparkPlatform handles the Kafka Source differently, using its own Spark-Kafka-Source.
Our JavaPlatform works directly with Consumer, and with Producer API. Hence we will have different implementations for this particular Problem.

My understanding was, that all Kafka related functionality should be "grouped" in a specific module, and there, the individual implementations for Java / Spark coexist, but no other place in the project Wayang project has a Kafka dependency.

[juripetersen]

My understanding was, that all Kafka related functionality should be "grouped" in a specific module, and there, the individual implementations for Java / Spark coexist, but no other place in the project Wayang project has a Kafka dependency.

That is essentially what I meant, yes. But we will push that change into the future as part of #469.

[juripetersen]

Most of the comments I made should be resolvable pretty quick. Let me know if something is unclear!

[github-actions]

license-eye has checked 1555 files.

Valid	Invalid	Ignored	Fixed
1420	2	133	0

Click to see the invalid file list

• env_template.sh
• wayang-applications/bin/env_template.sh

Use this command to fix any missing license headers

```bash

docker run -it --rm -v $(pwd):/github/workspace apache/skywalking-eyes header fix

</details>

[kamir]

All issues solved.

Test Scripts still work on my system.

[kamir]

Build runs in cloud environment.

[juripetersen]

@kamir there is still one debug print in the wayang-submit. If you resolve that, I will approve the review.

[kamir]

The debugging line has been removed.

[juripetersen]

Thank you, looks good to me!

[github-actions]

license-eye has checked 1555 files.

Valid	Invalid	Ignored	Fixed
1422	1	132	0

Click to see the invalid file list

• wayang-applications/bin/env.demo1.sh

Use this command to fix any missing license headers

```bash

docker run -it --rm -v $(pwd):/github/workspace apache/skywalking-eyes header fix

</details>

[novatechflow]

@juripetersen please review

[juripetersen]

There are still two files with "os specific" configurations. Let's remove them.

[kamir]

The OSX specific paths have been removed. We will provide some specific scripts for WIN, OSX, Linux soon.

[juripetersen]

@2pk03 @kamir I approved the review. I am not sure why this still requires one more review by someone.

[novatechflow]

Awesome, well done @kamir and @juripetersen!