Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add Database.list_tables method #219

Merged
merged 15 commits into from Mar 1, 2021

Conversation

tswast
Copy link
Contributor

@tswast tswast commented Jan 29, 2021

Adds Table-management methods required by PSO data validation tool.

Fixes #156 馃

@product-auto-label product-auto-label bot added the api: spanner label Jan 29, 2021
@google-cla google-cla bot added the cla: yes label Jan 29, 2021
Copy link
Contributor

@larkee larkee left a comment

LG, just a couple design details to work out regarding the Table methods 馃憤

google/cloud/spanner_v1/database.py Outdated Show resolved Hide resolved
google/cloud/spanner_v1/table.py Show resolved Hide resolved
google/cloud/spanner_v1/table.py Outdated Show resolved Hide resolved
self.table_id = table_id
self._database = database

def get_schema(self):
Copy link
Contributor

@larkee larkee Feb 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think instead of having get_schema, it would make more sense to match the rest of the library and the BigQuery implementation by having this as a property. What do you think?

Copy link
Contributor Author

@tswast tswast Feb 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would be more consistent. My main worry with a property is it might be less clear that table.schema makes an API call compared to table.get_schema().

Copy link
Contributor

@larkee larkee Feb 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is a good point. For the other resource objects (i.e. Instance, Backup, Database) we store local copies of properties and have a reload() method to make an RPC call to update them. I think it may make sense to stick to this design, especially if we decide to support create, update, and delete on the Table object.

I would initialize a private schema attribute as None and then call reload in schema if it is None. This means the property would make one API call at most. Then, if the user is expecting the schema to change frequently, they can all reload themselves to ensure the schema remains in sync with production. WDYT?

Copy link
Contributor Author

@tswast tswast Feb 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would initialize a private schema attribute as None and then call reload in schema if it is None. This means the property would make one API call at most. Then, if the user is expecting the schema to change frequently, they can all reload themselves to ensure the schema remains in sync with production. WDYT?

I think that's a good approach. In BigQuery we didn't have this logic, so customers would surprisingly get None values, which caused all sorts of issues. I'll make this a property and add a reload method.

Copy link
Contributor Author

@tswast tswast Feb 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated. Also added a exists() method to match Database.reload()'s NotFound exception behavior.

@tswast tswast force-pushed the issue156-list_tables branch from 134f1c8 to 40a60e3 Compare Feb 1, 2021
The NUMERIC column is not included in the emulator system tests.
@tswast tswast marked this pull request as ready for review Feb 1, 2021
@tswast tswast requested a review from as a code owner Feb 1, 2021
@tswast
Copy link
Contributor Author

@tswast tswast commented Feb 1, 2021

I'm marking this as ready for review, though I suspect I'll need to add some unit tests for coverage.

@larkee
Copy link
Contributor

@larkee larkee commented Feb 2, 2021

The new Table type will also need documentation similar to database-api and database-usage so that documentation for the type and how to use it can be generated.

@tswast
Copy link
Contributor Author

@tswast tswast commented Feb 2, 2021

Docs added:

image

image


.. code:: python
table = google.cloud.spanner_v1.table.Table(
Copy link
Contributor

@larkee larkee Feb 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a table() method on the Database object which creates the Table object to be consistent with the other classes e.g.

client = spanner.Client()
instance = client.instance(instance_id)
database = instance.database(database_id)
table = database.table(table_id)

Copy link
Contributor Author

@tswast tswast Feb 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done! Thanks for your patience while I was OOO.

Edit: added in the following commit: 1250d8a

@tswast tswast requested a review from larkee Feb 17, 2021
@tswast
Copy link
Contributor Author

@tswast tswast commented Feb 22, 2021

@larkee I've made the requested changes and synced with latest master branch. Should be ready for another round of review.

@tswast
Copy link
Contributor Author

@tswast tswast commented Feb 23, 2021

Test failure

______________________ TestBackupAPI.test_backup_workflow ______________________

self = <tests.system.test_system.TestBackupAPI testMethod=test_backup_workflow>

    def test_backup_workflow(self):
        from datetime import datetime
        from datetime import timedelta
        from pytz import UTC

        instance = Config.INSTANCE
        backup_id = "backup_id" + unique_resource_id("_")
        expire_time = datetime.utcnow() + timedelta(days=3)
        expire_time = expire_time.replace(tzinfo=UTC)

        # Create backup.
        backup = instance.backup(
            backup_id,
            database=self._db,
            expire_time=expire_time,
            version_time=self.database_version_time,
        )
        operation = backup.create()
        self.to_delete.append(backup)

        # Check metadata.
        metadata = operation.metadata
        self.assertEqual(backup.name, metadata.name)
        self.assertEqual(self._db.name, metadata.database)
        operation.result()

        # Check backup object.
        backup.reload()
        self.assertEqual(self._db.name, backup._database)
        self.assertEqual(expire_time, backup.expire_time)
        self.assertIsNotNone(backup.create_time)
        self.assertEqual(self.database_version_time, backup.version_time)
        self.assertIsNotNone(backup.size_bytes)
        self.assertIsNotNone(backup.state)

        # Update with valid argument.
        valid_expire_time = datetime.utcnow() + timedelta(days=7)
        valid_expire_time = valid_expire_time.replace(tzinfo=UTC)
        backup.update_expire_time(valid_expire_time)
        self.assertEqual(valid_expire_time, backup.expire_time)

        # Restore database to same instance.
        restored_id = "restored_db" + unique_resource_id("_")
        database = instance.database(restored_id)
        self.to_drop.append(database)
        operation = database.restore(source=backup)
        restored_db = operation.result()
>       self.assertEqual(
            self.database_version_time, restored_db.restore_info.backup_info.create_time
        )
E       AssertionError: datetime.datetime(2021, 2, 22, 21, 31, 33[16 chars]UTC>) != DatetimeWithNanoseconds(2021, 2, 22, 21, [40 chars].utc)

appears to be a problem with master branch. I'll try merging from upstream again.

larkee
larkee approved these changes Feb 26, 2021
Copy link
Contributor

@larkee larkee left a comment

LGTM other than a minor docs fix 馃憤

docs/table-usage.rst Outdated Show resolved Hide resolved
@larkee larkee added the kokoro:force-run label Mar 1, 2021
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run label Mar 1, 2021
@larkee larkee merged commit 28bde8c into googleapis:master Mar 1, 2021
11 checks passed
@tswast tswast deleted the issue156-list_tables branch Mar 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: spanner cla: yes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants