[BEAM-3342] CBT `Read` IO Connector Wrapper #11295

mf2199 · 2020-04-02T17:56:23Z

[CBT connector: Expanding bigtableio with read functionality.]

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	SDK	Apex	Dataflow	Gearpump	Samza
Go		---	---	---	---
Java
Python		---		---	---
XLang	---	---	---	---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website
Non-portable
Portable	---		---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

chamikaramj · 2020-05-08T22:31:23Z

Retest this please

chamikaramj · 2020-05-08T22:31:35Z

Run Python PostCommit

chamikaramj · 2020-06-07T00:22:55Z

Run Python PreCommit

chamikaramj · 2020-06-07T00:23:07Z

Retest this please

chamikaramj · 2020-06-07T00:24:06Z

Run Python PostCommit

aaltay · 2020-06-19T01:00:28Z

retest this please

…build engine artefact?)

mf2199 · 2020-06-21T23:33:05Z

@chamikaramj @aaltay The build errors point to 'missing' arguments that were made so by design, e.g.:

15:41:25 E       ValueError: Pipeline has validations errors: 
15:41:25 E       Missing required option: project.
15:41:25 E       Missing required option: region.
15:41:25 E       Missing GCS path option: temp_location.
15:41:25 E       Missing GCS path option: staging_location.

[from "Run Python PreCommit" log]

These are meant to be supplied via command line, as they may contain non-public information and are subject to change.

Any ideas?

chamikaramj · 2020-07-22T16:30:53Z

Retest this please

chamikaramj · 2020-07-22T16:31:23Z

Logs were not available anymore. Re-triggerred.

chamikaramj · 2020-07-22T16:49:14Z

sdks/python/apache_beam/io/gcp/bigtableio.py

-
-"""BigTable connector
-
-This module implements writing to BigTable tables.


Unfortunately we forgot to mark this as experimental. So we'll have to leave the sink here and add the new source to "experimental/bigtableio.py" for backwards compatibility. When we are confident about the performance of the source we can move it her. For example, I think we need to add support for dynamic work rebalancing similar to Java BigTable source (can be in a future PR).

If I understand correctly, we are to restore this file in the original version while keeping the altered one in the experimental folder. If so, then it's done.

chamikaramj · 2020-07-22T17:01:49Z

sdks/python/apache_beam/io/gcp/experimental/bigtableio_read_it_test.py

+
+def parse_commane_line_arguments():
+  parser = argparse.ArgumentParser()
+


You do not need to set these up manually. Please see the way our other integration tests are setup. You can either configure a new Jenkins test suite or add integration tests to existing Python post-commit tests. Latter might be easier.

aaltay · 2020-08-13T19:58:11Z

@mf2199 - What is the next step on this PR?

stale · 2020-12-13T01:58:04Z

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.

stale · 2020-12-25T11:29:47Z

This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

param17 · 2021-03-23T16:26:52Z

Can we re-open this? What's the next step?
cc: @mf2199

chamikaramj · 2021-03-23T19:17:12Z

IIRC we could not get BT integration tests working with this PR. Also I think some folks from Google side was going to improve this or look into this but not sure if what work happened. I would like to get this PR to a good working state (including working ITs) before submitting. I think a getting a connector that is not fully fleshed out submitted will do more harm than good.

Another option will be to add a multi-language wrapper for already established Java BT connector which should work for portable runners and Dataflow Runner v2.

sachinag · 2022-07-08T19:50:53Z

@chamikaramj can you see if BT team can get this over the line please?

aaltay · 2022-07-14T01:11:24Z

/cc @johnjcasey

chamikaramj · 2022-07-14T01:15:32Z

I prefer implementing a x-lang based connector instead of proceeding with this one.

Added ReadFromBigtable and _BigtableReadFn classes.

d6e2f6b

probot-autolabeler bot added gcp io python labels Apr 2, 2020

mf2199 changed the title ~~CBT connector: Expanding bigtableio with read functionality.~~ [testing only] [WIP-draft] CBT connector: Expanding bigtableio with read functionality. Apr 2, 2020

mf2199 changed the title ~~[testing only] [WIP-draft] CBT connector: Expanding bigtableio with read functionality.~~ [testing only] [WIP-draft] Apr 2, 2020

mf2199 changed the title ~~[testing only] [WIP-draft]~~ [testing only - NOT READY FOR REVIEW] [WIP-draft] Apr 2, 2020

mf2199 changed the title ~~[testing only - NOT READY FOR REVIEW] [WIP-draft]~~ [WIP-draft, testing only - NOT READY FOR REVIEW] Apr 2, 2020

mf2199 marked this pull request as ready for review April 2, 2020 18:02

mf2199 added 9 commits April 2, 2020 14:07

Reverted all changes

e7749e3

Added imports

5b844ed

Added 'blank' _BigtableReadFn

5069f97

[WIP]

be53e99

[WIP]

43adf26

[WIP]

10b95d8

[WIP]

f935dc8

[WIP]

6a8c05c

[everything added back, BEFORE ReadFromBigTable class]

956f2d7

mf2199 closed this Apr 20, 2020

mf2199 reopened this May 8, 2020

mf2199 added 2 commits May 8, 2020 18:22

Code transfer from apache#8457

254913c

Merge branch 'master' into bigtableio-read

fe06659

mf2199 mentioned this pull request May 8, 2020

[BEAM-3342] Create a Cloud Bigtable IO connector for Python #8457

Closed

3 tasks

mf2199 closed this Jun 6, 2020

chamikaramj reopened this Jun 7, 2020

mf2199 added 10 commits June 18, 2020 23:24

fix: formatting + naming

2b0ef81

fis: lint errors

a4bd7c6

fix: isort + yapf

bd32b36

fix: isort (again?)

87cf0ea

fix: NameError: name 'ValueRangeFilter' is not defined

90aace4

fix:

854a132

fix: ...missing a call to unittest.main()

3d59f01

cleanup: refactoring

d588042

fix: format

5744f4a

fix: TypeError: %d format: a number is required, not NoneType (a …

2a8ab0d

…build engine artefact?)

chamikaramj reviewed Jul 22, 2020

View reviewed changes

fix: reverted sink deletion for backwards compatibility

886fa75

mf2199 added 4 commits September 28, 2020 10:35

chore: TestPipeline added to IT [read] test + refactoring

bb8bcbf

fix: format

f2fa450

chore: format fix + refactor

bbeae20

fix: lint

fbd2d83

stale bot added the stale label Dec 13, 2020

stale bot closed this Dec 25, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BEAM-3342] CBT `Read` IO Connector Wrapper #11295

[BEAM-3342] CBT `Read` IO Connector Wrapper #11295

mf2199 commented Apr 2, 2020 •

edited

chamikaramj commented May 8, 2020

chamikaramj commented May 8, 2020

chamikaramj commented Jun 7, 2020

chamikaramj commented Jun 7, 2020

chamikaramj commented Jun 7, 2020

aaltay commented Jun 19, 2020

mf2199 commented Jun 21, 2020 •

edited

chamikaramj commented Jul 22, 2020

chamikaramj commented Jul 22, 2020 •

edited

chamikaramj Jul 22, 2020

mf2199 Jul 30, 2020

chamikaramj Jul 22, 2020

aaltay commented Aug 13, 2020

stale bot commented Dec 13, 2020

stale bot commented Dec 25, 2020

param17 commented Mar 23, 2021

chamikaramj commented Mar 23, 2021

sachinag commented Jul 8, 2022

aaltay commented Jul 14, 2022

chamikaramj commented Jul 14, 2022


		"""BigTable connector

		This module implements writing to BigTable tables.


		def parse_commane_line_arguments():
		parser = argparse.ArgumentParser()

[BEAM-3342] CBT Read IO Connector Wrapper #11295

[BEAM-3342] CBT Read IO Connector Wrapper #11295

Conversation

mf2199 commented Apr 2, 2020 • edited

Post-Commit Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

chamikaramj commented May 8, 2020

chamikaramj commented May 8, 2020

chamikaramj commented Jun 7, 2020

chamikaramj commented Jun 7, 2020

chamikaramj commented Jun 7, 2020

aaltay commented Jun 19, 2020

mf2199 commented Jun 21, 2020 • edited

chamikaramj commented Jul 22, 2020

chamikaramj commented Jul 22, 2020 • edited

chamikaramj Jul 22, 2020

Choose a reason for hiding this comment

mf2199 Jul 30, 2020

Choose a reason for hiding this comment

chamikaramj Jul 22, 2020

Choose a reason for hiding this comment

aaltay commented Aug 13, 2020

stale bot commented Dec 13, 2020

stale bot commented Dec 25, 2020

param17 commented Mar 23, 2021

chamikaramj commented Mar 23, 2021

sachinag commented Jul 8, 2022

aaltay commented Jul 14, 2022

chamikaramj commented Jul 14, 2022

[BEAM-3342] CBT `Read` IO Connector Wrapper #11295

[BEAM-3342] CBT `Read` IO Connector Wrapper #11295

mf2199 commented Apr 2, 2020 •

edited

mf2199 commented Jun 21, 2020 •

edited

chamikaramj commented Jul 22, 2020 •

edited