Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bigtable: Create table in list_table test, deadline exceed error fix #8879

Merged

Conversation

sangramql
Copy link
Contributor

@sangramql sangramql commented Aug 1, 2019

Issue #8479
Bigtable: 'test_bigtable_list_tables' snippet flakes with '504 Deadline Exceeded'.

Issue:
As per my understanding this used to occur intermittently when Google cloud takes longer than expected to create instance or table. So we added cool down time before creating table in an instance.
We used to observe this issue sometimes while working on bigtable issues.

Fix:
This fix is added to avoid creating and deleting table for single test.
Instead, common table can be created and used in tests which require table specific tests, except table creation test.

Fix tested with following approaches:
This issue is not consistent in production i suppose. And I was not able to reproduce even once.
I tried different approach to hit this issue,

  1. Created multiple tables(>100) in test function.
  2. Created instance and w/o cool down time immediately created multiple tables (> 100) in class setup.
  3. Tried to created multiple instances (>5) and table in an instance, but got the following error
    after creating 6 instances. But didn’t get ‘deadline exceeded’ error though.
  4. Executed parallel threads of snippets.py from nox.

Cool down time (timeout=100) code was avoided to hit the issue, but error didn't occur.
---code---
operation = Config.INSTANCE.create(clusters=[cluster])
#We want to make sure the operation completes.
#operation.result(timeout=100) #this code is commented to reproduce issue.
Config.TABLE = Config.INSTANCE.table(TABLE_ID) Config.TABLE.create()
---/code---

@googlebot googlebot added the cla: yes This human has signed the Contributor License Agreement. label Aug 1, 2019
@AVaksman AVaksman added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 1, 2019
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 1, 2019
bigtable/docs/snippets.py Outdated Show resolved Hide resolved
bigtable/docs/snippets.py Outdated Show resolved Hide resolved
@sangramql
Copy link
Contributor Author

Please review @tseaver @crwilcox

bigtable/docs/snippets.py Outdated Show resolved Hide resolved
@tseaver tseaver merged commit 2da870e into googleapis:master Aug 8, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants