fixes #12591: add BigTable #15122

sushi30 · 2024-02-09T11:05:38Z

Describe your changes:

Fixes #12591. Implements the BigTable metadata connector.

Documentation will be submitted in a separate PR.

Summary

Implemented the metadata ingestion connector in metadata.py,connection.py,models.py`.
Added bytes type support for datalake_utils.
Added debug message when records return unsupported types.
Added support for table constraints for NoSQL databases.
Implemented the required jsonschemas.
Implemented ClassConverter`

Type of change:

Improvement

Checklist:

I have read the CONTRIBUTING document.
My PR title is Fixes <issue-number>: <short explanation>
I have commented on my code, particularly in hard-to-understand areas.
For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

github-actions · 2024-02-09T11:13:20Z

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

github-actions · 2024-02-09T11:25:03Z

Jest test Coverage

UI tests summary

Lines	Statements	Branches	Functions
	58.8% (30165/51300)	39.04% (12104/31003)	38.79% (3680/9488)

1. docstrings 2. tests 3. created a Row BaseModel 4. implemented a ClassConverter

github-actions · 2024-02-12T08:47:20Z

The Java checkstyle failed.

Please run mvn spotless:apply in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Java code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

github-actions · 2024-02-12T08:50:16Z

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

github-actions · 2024-02-12T09:01:00Z

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

- removed TODO - changed headers' year to 2024 for new files - fixed typos

github-actions · 2024-02-12T09:11:39Z

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

github-actions · 2024-02-12T10:15:42Z

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

pmbrull

Awesome @sushi30! Thanks for the great job. Left just a few comments if you have the time

pmbrull · 2024-02-12T11:21:39Z

ingestion/src/metadata/ingestion/source/database/bigtable/metadata.py

+
+        # ths instances and tables are cached to avoid making redundant requests to the API.
+        self.instances: Dict[str, Tuple[Instance]] = {}
+        self.tables: Dict[str, Dict[str, Tuple[Table]]] = {}


nit - since this is quite a hard typing, would it make sense to have a custom pydantic model?

I changed the types to be aliased so it is clear how the dictionaries are formed. If you still think a pydantic Model is better suited I am open to modifying but I need an example of how to model it either in the project or some draft.

thanks, works with the aliases 👍

pmbrull · 2024-02-12T11:22:40Z

ingestion/src/metadata/ingestion/source/database/bigtable/metadata.py

+        return cls(config, metadata)
+
+    def get_configured_database(self) -> Optional[str]:
+        return None


maybe I'm just out of context on the NoSQL common class, but a comment/docstring explaining why this is None would be great 🙏

added a docstring. this is an edge case for a MultiDB source without a default DB.

pmbrull · 2024-02-12T11:23:49Z

ingestion/src/metadata/ingestion/source/database/bigtable/metadata.py

+        project_id = self.context.database
+        try:
+            # the first element is a list of instances
+            # the second element is another collection (seems empty) and I do not know what is its purpose


fair enough 😅 sometimes these client have weird things

pmbrull · 2024-02-12T11:25:19Z

ingestion/src/metadata/ingestion/source/database/bigtable/metadata.py

+        try:
+            instance = [
+                i for i in self.instances[project_id] if i.instance_id == schema_name
+            ][0]


oh this feels a bit magic

abstracted it to a method. should be clearer now.

pmbrull · 2024-02-12T11:26:16Z

ingestion/tests/unit/topology/database/test_bigtable.py

+                        "type": "service_account",
+                        "projectId": "my-gcp-project",
+                        "privateKeyId": "private_key_id",
+                        "privateKey": "-----BEGIN RSA PRIVATE KEY-----\nMIIEpQIBAAKCAQEAw3vHG9fDIkcYB0xi2Mv4fS2gUzKR9ZRrcVNeKkqGFTT71AVB\nOzgIqYVe8b2aWODuNye6sipcrqTqOt05Esj+sxhk5McM9bE2RlxXC5QH/Bp9zxMP\n/Yksv9Ov7fdDt/loUk7sTXvI+7LDJfmRYU6MtVjyyLs7KpQIB2xBWEToU1xZY+v0\ndRC1NA+YWc+FjXbAiFAf9d4gXkYO8VmU5meixVh4C8nsjokEXk0T/HEItpZCxadk\ndZ7LKUE/HDmWCO2oNG6sCf4ET2crjSdYIfXuREopX1aQwnk7KbI4/YIdlRz1I369\nAz3+Hxlf9lLJVH3+itN4GXrR9yWWKWKDnwDPbQIDAQABAoIBAQC3X5QuTR7SN8iV\niBUtc2D84+ECSmza5shG/UJW/6N5n0Mf53ICgBS4GNEwiYCRISa0/ILIgK6CcVb7\nsuvH8F3kWNzEMui4TO0x4YsR5GH9HkioCCS224frxkLBQnL20HIIy9ok8Rpe6Zjg\nNZUnp4yczPyqSeA9l7FUbTt69uDM2Cx61m8REOpFukpnYLyZGbmNPYmikEO+rq9r\nwNID5dkSeVuQYo4MQdRavOGFUWvUYXzkEQ0A6vPyraVBfolESX8WaLNVjic7nIa3\nujdSNojnJqGJ3gslntcmN1d4JOfydc4bja4/NdNlcOHpWDGLzY1QnaDe0Koxn8sx\nLT9MVD2NAoGBAPy7r726bKVGWcwqTzUuq1OWh5c9CAc4N2zWBBldSJyUdllUq52L\nWTyva6GRoRzCcYa/dKLLSM/k4eLf9tpxeIIfTOMsvzGtbAdm257ndMXNvfYpxCfU\nK/gUFfAUGHZ3MucTHRY6DTkJg763Sf6PubA2fqv3HhVZDK/1HGDtHlTPAoGBAMYC\npdV7O7lAyXS/d9X4PQZ4BM+P8MbXEdGBbPPlzJ2YIb53TEmYfSj3z41u9+BNnhGP\n4uzUyAR/E4sxrA2+Ll1lPSCn+KY14WWiVGfWmC5j1ftdpkbrXstLN8NpNYzrKZwx\njdR0ZkwvZ8B5+kJ1hK96giwWS+SJxJR3TohcQ18DAoGAJSfmv2r//BBqtURnHrd8\nwq43wvlbC8ytAVg5hA0d1r9Q4vM6w8+vz+cuWLOTTyobDKdrG1/tlXrd5r/sh9L0\n15SIdkGm3kPTxQbPNP5sQYRs8BrV1tEvoao6S3B45DnEBwrdVN42AXOvpcNGoqE4\nuHpahyeuiY7s+ZV8lZdmxSsCgYEAolr5bpmk1rjwdfGoaKEqKGuwRiBX5DHkQkxE\n8Zayt2VOBcX7nzyRI05NuEIMrLX3rZ61CktN1aH8fF02He6aRaoE/Qm9L0tujM8V\nNi8WiLMDeR/Ifs3u4/HAv1E8v1byv0dCa7klR8J257McJ/ID4X4pzcxaXgE4ViOd\nGOHNu9ECgYEApq1zkZthEQymTUxs+lSFcubQpaXyf5ZC61cJewpWkqGDtSC+8DxE\nF/jydybWuoNHXymnvY6QywxuIooivbuib6AlgpEJeybmnWlDOZklFOD0abNZ+aNO\ndUk7XVGffCakXQ0jp1kmZA4lGsYK1h5dEU5DgXqu4UYJ88Vttax2W+Y=\n-----END RSA PRIVATE KEY-----\n",


I hope this is a made-up key 😬

it's a real key. but it doesnt open aynthing... it has to be real otherwise the model fails on validation.

1. added missing docstrings. 2. abstracted the _find_instance method. 3. aliased the IDs used in the BigTable connection

github-actions · 2024-02-12T15:36:30Z

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

github-actions · 2024-02-12T15:42:58Z

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

sonarcloud · 2024-02-12T16:06:07Z

Quality Gate passed for 'open-metadata-ui'

Issues
0 New issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

sonarcloud · 2024-02-12T16:09:47Z

Quality Gate passed for 'open-metadata-ingestion'

Issues
0 New issues

Measures
0 Security Hotspots
74.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

pmbrull

Thanks @sushi30, congrats on the first connector! LGTM

feat(connector): add BigTable

db7bfd1

sushi30 requested review from a team, harshach, karanh37, chirag-madlani and Sachin-chaurasiya as code owners February 9, 2024 11:05

sushi30 temporarily deployed to test February 9, 2024 11:05 — with GitHub Actions Inactive

github-actions bot added documentation Improvements or additions to documentation UI UI specific issues Ingestion backend safe to test Add this label to run secure Github workflows on PRs labels Feb 9, 2024

sushi30 added 2 commits February 12, 2024 09:32

bigtable work

1203be9

1. docstrings 2. tests 3. created a Row BaseModel 4. implemented a ClassConverter

docs moved to separate PR

0f81127

sushi30 changed the title ~~feat(connector): add BigTable~~ fixes 12591: add BigTable Feb 12, 2024

format files

95f5f2f

minor cosmetic

a5fee9b

- removed TODO - changed headers' year to 2024 for new files - fixed typos

sushi30 changed the title ~~fixes 12591: add BigTable~~ fixes #12591: add BigTable Feb 12, 2024

sushi30 mentioned this pull request Feb 12, 2024

MINOR: BigTable connector docs #15145

Merged

5 tasks

format

bf6b0b1

pmbrull reviewed Feb 12, 2024

View reviewed changes

sushi30 added 3 commits February 12, 2024 13:55

formatting and comments

d238e14

1. added missing docstrings. 2. abstracted the _find_instance method. 3. aliased the IDs used in the BigTable connection

added comment regarding private key

716b4bd

added comments regarding column families

530d941

enclose get_schema_name_list in try/except/else

d467111

sushi30 added 2 commits February 12, 2024 16:45

format

32ea9bf

streamlined get_schema_name_list to include all logic in the try block

1a5115d

pmbrull approved these changes Feb 13, 2024

View reviewed changes

IceS2 approved these changes Feb 13, 2024

View reviewed changes

chirag-madlani approved these changes Feb 13, 2024

View reviewed changes

pmbrull merged commit aeb5fbe into main Feb 13, 2024
20 checks passed

pmbrull deleted the big-table-connector branch February 13, 2024 07:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fixes #12591: add BigTable #15122

fixes #12591: add BigTable #15122

sushi30 commented Feb 9, 2024 •

edited

github-actions bot commented Feb 9, 2024

github-actions bot commented Feb 9, 2024

github-actions bot commented Feb 12, 2024

github-actions bot commented Feb 12, 2024

github-actions bot commented Feb 12, 2024

github-actions bot commented Feb 12, 2024

github-actions bot commented Feb 12, 2024

pmbrull left a comment

pmbrull Feb 12, 2024

sushi30 Feb 12, 2024

pmbrull Feb 13, 2024

pmbrull Feb 12, 2024

sushi30 Feb 12, 2024

pmbrull Feb 12, 2024

pmbrull Feb 12, 2024

sushi30 Feb 12, 2024

pmbrull Feb 12, 2024

sushi30 Feb 12, 2024

github-actions bot commented Feb 12, 2024

github-actions bot commented Feb 12, 2024

sonarcloud bot commented Feb 12, 2024

sonarcloud bot commented Feb 12, 2024

pmbrull left a comment

fixes #12591: add BigTable #15122

fixes #12591: add BigTable #15122

Conversation

sushi30 commented Feb 9, 2024 • edited

Describe your changes:

Summary

Type of change:

Checklist:

github-actions bot commented Feb 9, 2024

github-actions bot commented Feb 9, 2024

Jest test Coverage

UI tests summary

github-actions bot commented Feb 12, 2024

github-actions bot commented Feb 12, 2024

github-actions bot commented Feb 12, 2024

github-actions bot commented Feb 12, 2024

github-actions bot commented Feb 12, 2024

pmbrull left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Feb 12, 2024

github-actions bot commented Feb 12, 2024

sonarcloud bot commented Feb 12, 2024

Quality Gate passed for 'open-metadata-ui'

sonarcloud bot commented Feb 12, 2024

Quality Gate passed for 'open-metadata-ingestion'

pmbrull left a comment

Choose a reason for hiding this comment

sushi30 commented Feb 9, 2024 •

edited