Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BEAM-8645] A test case for TimestampCombiner. #10081

Merged
merged 4 commits into from Dec 2, 2019
Merged

[BEAM-8645] A test case for TimestampCombiner. #10081

merged 4 commits into from Dec 2, 2019

Conversation

HuangLED
Copy link
Contributor

@HuangLED HuangLED commented Nov 13, 2019

Adds test case for TimestampCombiner.

Skip tag is added (until BEAM-8657 and BEAM-8645 are resolved)

Related mailing list dicussions: https://lists.apache.org/thread.html/d3af1f2f84a2e59a747196039eae77812b78a991f0f293c717e5f4e1@%3Cdev.beam.apache.org%3E


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • [ X] Choose reviewer(s) and mention them in a comment (R: @username).
  • [ X] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang SDK Apex Dataflow Flink Gearpump Samza Spark
Go Build Status --- --- Build Status --- --- Build Status
Java Build Status Build Status Build Status Build Status
Build Status
Build Status
Build Status Build Status Build Status
Build Status
Python Build Status
Build Status
Build Status
Build Status
--- Build Status
Build Status
Build Status
Build Status
--- --- Build Status
XLang --- --- --- Build Status --- --- ---

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website
Non-portable Build Status Build Status
Build Status
Build Status Build Status
Portable --- Build Status --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

@HuangLED HuangLED changed the title [BEAM-8645] A test case to show the expected behavior of TimestampCombiner. [BEAM-8645] A test case for TimestampCombiner. Nov 14, 2019
@HuangLED
Copy link
Contributor Author

R: @robertwb

@HuangLED
Copy link
Contributor Author

Run Portable_Python PreCommit

@HuangLED
Copy link
Contributor Author

Hi, Robert,

 I synced with your fixing PR for combiner lifting, now LATEST works (I removed skip tag), but EARLIEST still not correct.  

 Maybe let us merge this test case, and then you can test against your up-coming PR easier. WDYT? 

Copy link
Contributor

@robertwb robertwb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you try rebasing on master and seeing if that fixes the tests?

sdks/python/apache_beam/transforms/combiners_test.py Outdated Show resolved Hide resolved
sdks/python/apache_beam/transforms/combiners_test.py Outdated Show resolved Hide resolved
sdks/python/apache_beam/transforms/combiners_test.py Outdated Show resolved Hide resolved
sdks/python/apache_beam/transforms/combiners_test.py Outdated Show resolved Hide resolved
@HuangLED
Copy link
Contributor Author

Pulled from master, and retried. The EARLIEST case still failed, with error:

"apache_beam.runners.direct.executor: WARNING: A task failed with exception: Failed assert: [(('k', 500), Timestamp(7))] not in [(('k', 500), Timestamp(2))] [while running 'assert per window/Match']"

@HuangLED
Copy link
Contributor Author

would you please point out which PR in the master Might resolve the issue? I can follow and trace that part a bit using this test case.

Copy link
Contributor

@robertwb robertwb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Strange. I'll try to figure out what's up here. Some minor style comments below.

sdks/python/apache_beam/transforms/combiners_test.py Outdated Show resolved Hide resolved
sdks/python/apache_beam/transforms/combiners_test.py Outdated Show resolved Hide resolved
@HuangLED
Copy link
Contributor Author

Updated.

result = (p
| TestStream()
.add_elements([window.TimestampedValue(('k', 100), 2)])
.add_elements([window.TimestampedValue(('k', 400), 7)])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Turns out this happens to work because 7 is the last element.

@HuangLED
Copy link
Contributor Author

Run Portable_Python PreCommit

@HuangLED
Copy link
Contributor Author

Thanks for applying the unskip test.

All test now passed. This PR is ready to be merged.

@HuangLED
Copy link
Contributor Author

HuangLED commented Dec 2, 2019

@robertwb ping for merging this PR

@robertwb robertwb merged commit eb49b69 into apache:master Dec 2, 2019
11moon11 pushed a commit to 11moon11/beam that referenced this pull request Dec 12, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants