[FLINK-10986][tests] Implement DB to setup Apache Kafka #7173
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This enables submission of multiple jobs in the Jepsen tests. The job specifications are in an .edn file that must be passed as command line arguments. The checker verifies that all jobs are running at the end of the test. The job cancellation function now cancels all jobs at once. Delete script/run-tests.sh and move code to docker/run-tests.sh.
GJL
commented
Nov 29, 2018
| :job-args "--parallelism 1 --checkpointInterval 5000 --numKeys 1000 --topic kafka-test-topic --sleep 200 --semantic exactly-once --bootstrap.servers localhost:9092 --transaction.timeout.ms 600000 --checkpointDir hdfs:///flink-checkpoints" | ||
| :main-class "com.dataartisans.flink.example.eventpattern.DataGeneratorJob"} | ||
| {:job-jar "/jepsen/bin/flink-state-machine-kafka-1.0-SNAPSHOT.jar" | ||
| :job-args "--parallelism 1 --checkpointInterval 5000 --input-topic kafka-test-topic --bootstrap.servers localhost:9092 --checkpointDir hdfs:///flink-checkpoints" |
Member
Author
There was a problem hiding this comment.
Add --auto.offset.reset earliest
GJL
commented
Nov 29, 2018
| ;; See the License for the specific language governing permissions and | ||
| ;; limitations under the License. | ||
|
|
||
| {:dbs [:hadoop :zookeeper :flink-mesos-session] |
This enables toggling the setup of test dependencies, such as Hadooop, Mesos, and, ZooKeeper through the --test-spec edn file. The type of the Flink cluster can also be specified via: :flink-yarn-job :flink-yarn-session :flink-mesos-session, and :flink-standalone-session. Retryable operations that exhausted all attempts, now propagate the exception by default. This is to fail fast if during the db/setup!, an operation, such as Flink job submission, fails to complete.
tillrohrmann
approved these changes
Dec 5, 2018
Contributor
tillrohrmann
left a comment
tillrohrmann
left a comment
There was a problem hiding this comment.
Afaik it looks good to me. +1 for merging.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
Implement option to setup Kafka for distributed testing
Brief change log
Verifying this change
This change is already covered by existing tests, such as Jepsen tests.
Does this pull request potentially affect one of the following parts:
@Public(Evolving): (yes / no)Documentation