Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(spanner): add batch_create_session calls to session pools #9488

Merged
merged 7 commits into from Oct 23, 2019
Merged

feat(spanner): add batch_create_session calls to session pools #9488

merged 7 commits into from Oct 23, 2019

Conversation

larkee
Copy link
Contributor

@larkee larkee commented Oct 17, 2019

Currently, session pools are filled by creating single sessions. For large session pool sizes (>1000) this can be incredibly slow as it requires >1000 requests to be made to the server sequentially. Using batch_create_session calls to fill the pools will reduce the number of requests by a factor of 100 which will greatly reduce the start up time.

The BurstyPool does not use batch_create_session as it is designed to create sessions on an "as needed" basis rather than filling a queue.

@googlebot googlebot added the cla: yes This human has signed the Contributor License Agreement. label Oct 17, 2019
@larkee
Copy link
Contributor Author

larkee commented Oct 18, 2019

Build failure is due to a known flaky test.

@tseaver
Copy link
Contributor

tseaver commented Oct 22, 2019

@larkee I have a PR pending review, #9393, which fixes that flaky test.

@crwilcox crwilcox added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Oct 22, 2019
@crwilcox
Copy link
Contributor

Failure seems to be in an unrelated area.

_________ TestSessionAPI.test_transaction_batch_update_and_execute_dml _________

self = <tests.system.test_system.TestSessionAPI testMethod=test_transaction_batch_update_and_execute_dml>

    def test_transaction_batch_update_and_execute_dml(self):
        retry = RetryInstanceState(_has_all_ddl)
        retry(self._db.reload)()

        session = self._db.session()
        session.create()
        self.to_delete.append(session)

        with session.batch() as batch:
            batch.delete(self.TABLE, self.ALL)

        insert_statements = list(self._generate_insert_statements())
        update_statements = [
            (
                "UPDATE contacts SET email = @email " "WHERE contact_id = @contact_id;",
                {"contact_id": 1, "email": "phreddy@example.com"},
                {"contact_id": Type(code=INT64), "email": Type(code=STRING)},
            )
        ]

        delete_statement = "DELETE contacts WHERE TRUE;"

        def unit_of_work(transaction, self):
            rows = list(transaction.read(self.TABLE, self.COLUMNS, self.ALL))
            self.assertEqual(rows, [])

            status, row_counts = transaction.batch_update(
                insert_statements + update_statements
            )
            self.assertEqual(status.code, 0)  # XXX: where are values defined?
            self.assertEqual(len(row_counts), len(insert_statements) + 1)
            for row_count in row_counts:
                self.assertEqual(row_count, 1)

            row_count = transaction.execute_update(delete_statement)

            self.assertEqual(row_count, len(insert_statements))

>       session.run_in_transaction(unit_of_work, self)

tests/system/test_system.py:861:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
google/cloud/spanner_v1/session.py:299: in run_in_transaction
    return_value = func(txn, *args, **kw)
tests/system/test_system.py:852: in unit_of_work
    self.assertEqual(status.code, 0)  # XXX: where are values defined?
E   AssertionError: 10 != 0

@larkee larkee merged commit 55a920e into googleapis:master Oct 23, 2019
@larkee larkee deleted the batch-create-sessions-manual branch October 23, 2019 00:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla: yes This human has signed the Contributor License Agreement. kokoro:force-run Add this label to force Kokoro to re-run the tests.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants