-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enrichment Transform with BigTable handler #30001
Enrichment Transform with BigTable handler #30001
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mostly looks good, though I didn't get a chance to look at the enrichment_handlers folder yet. Leaving comments for the pieces I've reviewed so far
Thanks for the review! Made the suggested changes. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
BigTable piece mostly LGTM as well, just have some lingering comments
sdks/python/apache_beam/transforms/enrichment_handlers/bigtable.py
Outdated
Show resolved
Hide resolved
Thanks! Made the changes from last review. Throttle test cases are left. Feel free to defer the review until that or up to you! |
sdks/python/apache_beam/transforms/enrichment_handlers/bigtable.py
Outdated
Show resolved
Hide resolved
sdks/python/apache_beam/transforms/enrichment_handlers/bigtable.py
Outdated
Show resolved
Hide resolved
Added the throttle unit test. Bigtable improvements are remaining now. |
sdks/python/apache_beam/transforms/enrichment_handlers/bigtable.py
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM - thanks!
Thank you for your reviews! I'll merge once the tests pass |
Tests passed. Removing the trigger file |
* enrichment v1 * add documentation * add doc comment * rerun * update docs, lint * update docs, lint * add generic type * add generic type * adjust doc path * create test row * use request type * use request type * change module name * more tests * remove non-functional params * lint, doc * change types for general use * callable type * dict type * update signatures * fix unit test * bigtable with column family, ids, rrio-throttler * update tests for row filter * convert handler types from dict to Row * update tests for bigtable * ran pydocs * ran pydocs * mark postcommit * remove _test file, fix import * enable postcommit * add more tests * skip tests when dependencies are not installed * add deleted imports from last commit * add skip test condition * fix import order, add TooManyRequests to try-catch * make throttler, repeater non-optional * add exception level and tests * correct pydoc statement * add throttle tests * add bigtable improvements * default app_profile_id * add documentation, ignore None assignment * add to changes.md * change test structure that throws exception, skip http test for now * drop postcommit trigger file
This PR contains the implementation of:
Cross-lang enablement and caching will be added in subsequent PRs.
Part of #29787
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123
), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>
instead.CHANGES.md
with noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.