Skip to content

Conversation

@lgajowy
Copy link
Contributor

@lgajowy lgajowy commented Jul 2, 2019

This pr adds basic IOIT that can be run for 1000 records (there's only hashcode for 1000 records currently). To make the code review easier and because I'm not a Kafka expert, I took a baby steps approach here - after this PR is approved consequent PRs will provide:

  • metrics collection
  • hashes for bigger loads
  • jenkins job to run automatically and on-demand & dashboards for them

I ran this test on existing kubernetes kafka cluster that is there in .test-infra folder (with success).

@aromanenko-dev @iemejia could you take a look?


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

Post-Commit Tests Status (on master branch)

Lang SDK Apex Dataflow Flink Gearpump Samza Spark
Go Build Status --- --- Build Status --- --- Build Status
Java Build Status Build Status Build Status Build Status
Build Status
Build Status
Build Status Build Status Build Status
Build Status
Python Build Status
Build Status
--- Build Status
Build Status
Build Status --- --- Build Status

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website
Non-portable Build Status Build Status Build Status Build Status
Portable --- Build Status --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

This is useful for ad hoc configuration amendments etc. and helpful
for everyday development.
@lgajowy lgajowy requested review from aromanenko-dev and iemejia July 2, 2019 13:28
@iemejia iemejia changed the title [BEAM-4420] Kafka io it [BEAM-4420] Add KafkaIO Integration Tests Jul 2, 2019
@iemejia
Copy link
Member

iemejia commented Jul 2, 2019

Checkstyle not happy, you should use the vendored version of guava. Can you please also squash the commits into one with the proper title. Thanks.

Copy link
Contributor

@aromanenko-dev aromanenko-dev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Additionally to what Ismael said, I added several review comments

@lgajowy
Copy link
Contributor Author

lgajowy commented Jul 3, 2019

@iemejia @aromanenko-dev thanks for the review. I provided fixes.

@iemejia I didn't squash the commits on purpose - those are two atomic, independent changes. Having them as separate commits shows better why exactly something changed with proper commit titles and descriptions (see the description for the Kubernetes change). Squash will obfuscate this and will result with multiple unrelated changes in one commit. Other than that it will be easier to revert parts if needed. Suppose I was to revert only the Kubernetes change. Should I be forced to revert the whole test too?

Copy link
Contributor

@aromanenko-dev aromanenko-dev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
Agree to keep two independent commits

@aromanenko-dev aromanenko-dev merged commit 32e2e3e into apache:master Jul 3, 2019
@lgajowy
Copy link
Contributor Author

lgajowy commented Jul 3, 2019

Thanks!

@lgajowy lgajowy deleted the kafka-io-it branch July 3, 2019 13:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants