Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BEAM-8645] Create a py test case for Re-iteration on GBK result. #10135

Merged
merged 2 commits into from
Nov 19, 2019
Merged

[BEAM-8645] Create a py test case for Re-iteration on GBK result. #10135

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
26969f4
Add python test for re-iteration on GBK result.
HuangLED Nov 15, 2019
0098539
Fix comments.
HuangLED Nov 18, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
18 changes: 18 additions & 0 deletions sdks/python/apache_beam/transforms/ptransform_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -471,6 +471,24 @@ def test_group_by_key(self):
assert_that(result, equal_to([(1, [1, 2, 3]), (2, [1, 2]), (3, [1])]))
pipeline.run()

def test_group_by_key_reiteration(self):
class MyDoFn(beam.DoFn):
def process(self, gbk_result):
key, value_list = gbk_result
sum_val = 0
# Iterate the GBK result for multiple times.
for _ in range(0, 17):
sum_val += sum(value_list)
return [(key, sum_val)]

pipeline = TestPipeline()
pcoll = pipeline | 'start' >> beam.Create(
[(1, 1), (1, 2), (1, 3), (1, 4)])
result = (pcoll | 'Group' >> beam.GroupByKey()
| 'Reiteration-Sum' >> beam.ParDo(MyDoFn()))
assert_that(result, equal_to([(1, 170)]))
pipeline.run()

def test_partition_with_partition_fn(self):

class SomePartitionFn(beam.PartitionFn):
Expand Down