Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BEAM-14347] Add generic registration for Combiners #17579

Merged
merged 4 commits into from
May 10, 2022

Conversation

damccorm
Copy link
Contributor

@damccorm damccorm commented May 6, 2022

This is a continuation of the work started in #17429 and #17574 and extends the existing registration logic to allow registration of combiners.

Design doc: https://docs.google.com/document/d/1MLZLUhb-Sn7AJ666yn3TnYgMKE6wiXIBs7cjB_20hMA/edit?usp=sharing

After this, the big remaining chunks of work are:

  1. Perf testing
  2. Updates to the docs/our examples

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests

See CI.md for more information about GitHub Actions CI.

@asf-ci
Copy link

asf-ci commented May 6, 2022

Can one of the admins verify this patch?

3 similar comments
@asf-ci
Copy link

asf-ci commented May 6, 2022

Can one of the admins verify this patch?

@asf-ci
Copy link

asf-ci commented May 6, 2022

Can one of the admins verify this patch?

@asf-ci
Copy link

asf-ci commented May 6, 2022

Can one of the admins verify this patch?

@github-actions github-actions bot added the go label May 6, 2022
@codecov
Copy link

codecov bot commented May 6, 2022

Codecov Report

Merging #17579 (b2419ad) into master (85d9824) will decrease coverage by 0.00%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #17579      +/-   ##
==========================================
- Coverage   73.95%   73.95%   -0.01%     
==========================================
  Files         693      693              
  Lines       91716    91690      -26     
==========================================
- Hits        67832    67812      -20     
+ Misses      22635    22629       -6     
  Partials     1249     1249              
Flag Coverage Δ
go 50.37% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
sdks/python/apache_beam/utils/interactive_utils.py 95.12% <0.00%> (-2.44%) ⬇️
.../python/apache_beam/io/gcp/tests/pubsub_matcher.py 90.47% <0.00%> (-1.11%) ⬇️
sdks/python/apache_beam/internal/metrics/metric.py 93.00% <0.00%> (-1.00%) ⬇️
...ks/python/apache_beam/runners/worker/sdk_worker.py 88.94% <0.00%> (-0.16%) ⬇️
sdks/python/apache_beam/io/gcp/gcsio.py 92.26% <0.00%> (ø)
sdks/python/apache_beam/io/gcp/pubsub.py 90.65% <0.00%> (ø)
sdks/python/apache_beam/io/gcp/bigquery.py 63.87% <0.00%> (ø)
sdks/python/apache_beam/io/gcp/gcsfilesystem.py 88.23% <0.00%> (ø)
.../apache_beam/io/gcp/datastore/v1new/datastoreio.py 86.16% <0.00%> (ø)
...beam/transforms/fully_qualified_named_transform.py 100.00% <0.00%> (ø)
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 85d9824...b2419ad. Read the comment docs.

@damccorm
Copy link
Contributor Author

damccorm commented May 6, 2022

R: @lostluck

Since this one has a bunch of generic + generated code and is a tougher one to review without context, I'm going to send this directly to you

@github-actions
Copy link
Contributor

github-actions bot commented May 6, 2022

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

Copy link
Contributor

@lostluck lostluck left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Naming aside: LGTM.

sdks/go/pkg/beam/registration/registration.tmpl Outdated Show resolved Hide resolved
sdks/go/pkg/beam/registration/registration.tmpl Outdated Show resolved Hide resolved
Copy link
Contributor

@lostluck lostluck left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will merge once the conflict from the emitters/iters is resolved.

@damccorm
Copy link
Contributor Author

damccorm commented May 9, 2022

Will merge once the conflict from the emitters/iters is resolved.

Done

@lostluck lostluck merged commit a5fbc8e into apache:master May 10, 2022
@damccorm damccorm changed the title [BEAM-14347] Add generic registration for accumulators [BEAM-14347] Add generic registration for Combiners May 12, 2022
@damccorm damccorm deleted the users/damccorm/combinerRegistration branch June 1, 2022 17:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants