Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BEAM-8575] Windows idempotency: Applying the same window fn (or wind… #10071

Merged
merged 3 commits into from Nov 13, 2019
Merged

[BEAM-8575] Windows idempotency: Applying the same window fn (or wind… #10071

merged 3 commits into from Nov 13, 2019

Conversation

liumomo315
Copy link
Contributor

…ow fn + GBK) to the input multiple times will have the same effect as applying it once

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang SDK Apex Dataflow Flink Gearpump Samza Spark
Go Build Status --- --- Build Status --- --- Build Status
Java Build Status Build Status Build Status Build Status
Build Status
Build Status		 Build Status Build Status Build Status
Build Status
Python Build Status
Build Status
Build Status
Build Status		 --- Build Status
Build Status		 Build Status
Build Status		 --- --- Build Status
XLang --- --- --- Build Status --- --- ---

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website
Non-portable Build Status Build Status
Build Status		 Build Status Build Status
Portable --- Build Status --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

@liumomo315
[BEAM-8575] Windows idempotency: Applying the same window fn (or wind…
10cf667 
…ow fn + GBK) to the

input multiple times will have the same effect as applying it once
@liumomo315
Copy link
Contributor Author

beam_PostCommit_Py_VR_Dataflow

@liumomo315
Copy link
Contributor Author

R: @lcwik

Hi Luke, I suppose I will spend longer time on the custom Window fn test, so I separate the other two more trivial tests on window idempotency (from #9957) into this PR to get them in faster. Thanks!

I will later update the original PR of the custom Window Fn test.

lukecwik
lukecwik approved these changes Nov 12, 2019
View reviewed changes
Copy link
Member

@lukecwik lukecwik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor cleanup for test names, then will merge

sdks/python/apache_beam/transforms/window_test.py Outdated Show resolved Hide resolved
sdks/python/apache_beam/transforms/window_test.py Outdated Show resolved Hide resolved
@lukecwik
Copy link
Member

Run Python PreCommit

liumomo315 and others added 2 commits November 12, 2019 13:15
@liumomo315 @lukecwik
Update sdks/python/apache_beam/transforms/window_test.py
64b75b9 
Co-Authored-By: Lukasz Cwik <lcwik@google.com>
@liumomo315 @lukecwik
Update sdks/python/apache_beam/transforms/window_test.py
8dc96dc 
Co-Authored-By: Lukasz Cwik <lcwik@google.com>
@liumomo315
Copy link
Contributor Author

Thanks Luke for the quick review! Updated test names. Good to merge:)

@lukecwik
Copy link
Member

Run Portable_Python PreCommit

@lukecwik lukecwik merged commit c5e3253 into apache:master Nov 13, 2019
@alanmyrvold
Copy link
Member

6 tests started failing consistently in https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ after this merge?

apache_beam.transforms.window_test.WindowTest.test_window_assignment_idempotency
apache_beam.transforms.window_test.WindowTest.test_window_assignment_idempotency
apache_beam.transforms.window_test.WindowTest.test_window_assignment_through_multiple_gbk_idempotency
apache_beam.transforms.window_test.WindowTest.test_window_assignment_through_multiple_gbk_idempotency
apache_beam.transforms.window_test.WindowTest.test_window_assignment_idempotency
apache_beam.transforms.window_test.WindowTest.test_window_assignment_through_multiple_gbk_idempotency

@liumomo315 does this seem related?

@liumomo315
Copy link
Contributor Author

Yeah... having another PR to fix it https://github.com/apache/beam/pull/10111/files @alanmyrvold PTAL. Thanks!

11moon11 pushed a commit to 11moon11/beam that referenced this pull request Dec 12, 2019
@liumomo315 @lukecwik @11moon11
[BEAM-8575] Windows idempotency: Applying the same window fn (or wind… (
3ba2e9e 
apache#10071)

* [BEAM-8575] Windows idempotency: Applying the same window fn (or window fn + GBK) to the
input multiple times will have the same effect as applying it once

* Update sdks/python/apache_beam/transforms/window_test.py

Co-Authored-By: Lukasz Cwik <lcwik@google.com>

* Update sdks/python/apache_beam/transforms/window_test.py

Co-Authored-By: Lukasz Cwik <lcwik@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lukecwik lukecwik lukecwik approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@liumomo315 @lukecwik @alanmyrvold