Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BEAM-3317] Use fixed system time for testing #4723

Merged

Conversation

pawel-kaczmarczyk
Copy link
Contributor

@pawel-kaczmarczyk pawel-kaczmarczyk commented Feb 21, 2018

The problem here was caused by unfortunate timing during watermark retrieval. KinesisReader uses MovingFunction to decide if watermark should be advanced or not, parametrized with numSignificantBuckets=2 and numSignificantSamples=10. The test relied only on numSignificantSamples and did not take into consideration the time buckets. Now it sets the system time to a fixed value before execution so there will be no unfortunate time interatctions. I've also applied the same changes to other test that was likely to fail as it followed the same scheme.


Follow this checklist to help us incorporate your contribution quickly and easily:

  • Make sure there is a JIRA issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes.
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue.
  • Write a pull request description that is detailed enough to understand:
    • What the pull request does
    • Why it does it
    • How it does it
    • Why this approach
  • Each commit in the pull request should have a meaningful subject line and body.
  • Run mvn clean verify to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

@pawel-kaczmarczyk
Copy link
Contributor Author

@kennknowles @aromanenko-dev could you please take a look?

@aromanenko-dev
Copy link
Contributor

aromanenko-dev commented Feb 22, 2018

@pawel-kaczmarczyk
For me, it seems reasonable to set fixed "system" time in tests, so LGTM.
The only one concern on my side - do you have an easy way how to reproduce initial issue to make sure that test will pass all the time?

@pawel-kaczmarczyk
Copy link
Contributor Author

@aromanenko-dev It's enough to slow down the reader.advance() method that is called 10 times in the loop. I tested it by inserting a 1 second sleep inside this method and then the test failed every time.

@aromanenko-dev
Copy link
Contributor

aromanenko-dev commented Feb 22, 2018

@pawel-kaczmarczyk Great, thanks!
@kennknowles Could you do review and merge if it's ok for you? Thank you

assertThat(reader.getWatermark()).isEqualTo(new Instant(timestampMs));
try {
Instant now = Instant.now();
DateTimeUtils.setCurrentMillisFixed(now.getMillis());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is OK with me. I think even better would be to inject a clock into the reader (just pass to constructor) if it depends on such a service. I don't really know much about this connector, though, so it is just a vague suggestion.

@kennknowles
Copy link
Member

Can you please squash the commits together? Then I will merge.

@pawel-kaczmarczyk
Copy link
Contributor Author

@kennknowles commits squashed, thanks for your review!

@iemejia iemejia merged commit 51b793a into apache:master Feb 23, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants