Add Kafka as an event source

[rickfish]

Allow messages to be published to a Kafka topic. That Kafka topic will be used as a source for generating Conductor events to start workflows, fail tasks and complete tasks. The original implementation was taken from PR #672 which was never completed and code was added to handle secure topics, error processing and multiple consumers per topic.

Documentation on how to use it is in docs/docs/kafkaeventsource.md.

[codecov]

Codecov Report

Merging #1911 (aa16061) into dev (4c417be) will decrease coverage by 0.77%.
The diff coverage is 16.82%.

@@             Coverage Diff              @@
##                dev    #1911      +/-   ##
============================================
- Coverage     65.76%   64.98%   -0.78%     
- Complexity     4091     4113      +22     
============================================
  Files           310      316       +6     
  Lines         19270    19574     +304     
  Branches       1758     1792      +34     
============================================
+ Hits          12673    12721      +48     
- Misses         5730     5980     +250     
- Partials        867      873       +6     

Impacted Files	Coverage Δ	Complexity Δ
...va/com/netflix/conductor/contribs/KafkaModule.java	0.00% <0.00%> (ø)	0.00 <0.00> (?)
...lix/conductor/contribs/kafka/KafkaLoginModule.java	0.00% <0.00%> (ø)	0.00 <0.00> (?)
...tor/core/events/kafka/KafkaEventQueueProvider.java	0.00% <0.00%> (ø)	0.00 <0.00> (?)
...m/netflix/conductor/core/config/Configuration.java	44.26% <0.00%> (-24.97%)	22.00 <0.00> (ø)
...nductor/core/events/queue/MessageEventFailure.java	0.00% <0.00%> (ø)	0.00 <0.00> (?)
...x/conductor/core/events/queue/ObservableQueue.java	0.00% <0.00%> (ø)	0.00 <0.00> (ø)
...tor/contribs/queue/kafka/KafkaObservableQueue.java	16.66% <16.66%> (ø)	12.00 <12.00> (?)
...tor/core/events/queue/EventProcessingFailures.java	80.00% <80.00%> (ø)	7.00 <7.00> (?)
...ix/conductor/core/events/SimpleEventProcessor.java	70.21% <100.00%> (+1.09%)	29.00 <3.00> (+2.00)
... and 6 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4c417be...aa16061. Read the comment docs.

[coveralls]

Pull Request Test Coverage Report for Build 5300

• 59 of 309 (19.09%) changed or added relevant lines in 9 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-0.8%) to 69.48%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
core/src/main/java/com/netflix/conductor/core/events/queue/ObservableQueue.java	0	1	0.0%
core/src/main/java/com/netflix/conductor/core/events/queue/EventProcessingFailures.java	8	10	80.0%
contribs/src/main/java/com/netflix/conductor/contribs/KafkaModule.java	0	5	0.0%
contribs/src/main/java/com/netflix/conductor/core/events/kafka/KafkaEventQueueProvider.java	0	7	0.0%
core/src/main/java/com/netflix/conductor/core/events/queue/MessageEventFailure.java	0	10	0.0%
core/src/main/java/com/netflix/conductor/core/config/Configuration.java	0	22	0.0%
contribs/src/main/java/com/netflix/conductor/contribs/kafka/KafkaLoginModule.java	0	40	0.0%
contribs/src/main/java/com/netflix/conductor/contribs/queue/kafka/KafkaObservableQueue.java	41	204	20.1%

Totals
Change from base Build 5298:	-0.8%
Covered Lines:	13600
Relevant Lines:	19574

---

💛 - Coveralls

[rickfish]

Note that one test failed that has nothing to do with my changes: com.netflix.conductor.test.integration.ExternalPayloadStorageSpec. Not sure what to do about that.

[rickfish]

@aravindanr somehow I clicked on something that got rid of your question about the KafkaLoginModule class. Sorry, not sure what happened.

The KafkaLoginModule class can be referenced in a JAAS configuration file to allow adding JAAS security for access to the Kafka topic. Please see example files under src/main/resources in contribs. The 'kafka.default.jaas.config.file' or 'kafka.events.jaas.config.file' property can be specified that will point to the location of the JAAS config file and that JAAS config file can include a reference to the KafkaLoginModule class which will allow for secure access to the topic. I don't know of any other way to provide secure access although i have only used Kafka at my current employer and that is the way they require that security be done for Kafka.

If you look at the init() method of KafkaObservableQueue you will see the reference to config.getKafkaEventsJaasConfigFile(). If found it will set the java.security.auth.login.config system property.

[github-actions]

This PR is stale, because it has been open for 45 days with no activity. Remove the stale label or comment, or this will be closed in 7 days.

[github-actions]

This PR was closed, because it has been stalled for 7 days with no activity.