This repository has been archived by the owner on Aug 25, 2023. It is now read-only.
Async copy job is unable to copy table with id longer than 500 chars #24
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Pull Request Test Coverage Report for Build 381
💛 - Coveralls |
marcin-kolda
suggested changes
Aug 3, 2018
| # but still valid. | ||
| # | ||
| # The con is - if task_name is None, | ||
| # that means it's not unique and same task may be invoked more then once. |
There was a problem hiding this comment.
and the same backup may be created more than once
| # what protect us from failures where source table name is very long, | ||
| # but still valid. | ||
| # | ||
| # The con is - if task_name is None, |
| def test_return_none_if_calculated_name_is_too_long(self): | ||
| # given | ||
| task_name_suffix = "" |
There was a problem hiding this comment.
maybe more Pythonic way:
task_name_suffix = "x" * 501
There was a problem hiding this comment.
LOL, thanks :D
marcin-kolda
suggested changes
Aug 3, 2018
README.md
Outdated
| * Modification of table metadata (including table description) qualifies table to be backed up at the next cycle. It can be a problem for partitioned tables, where such change updates last modified time in every partition. Then BBQ will backup all partitions again, even though there was no actually change in partition data, | ||
| * There's 10,000 [copy jobs per project per day limit](https://cloud.google.com/bigquery/quotas#copy_jobs), which you may hit on the first day. This limit can be increased by Google Support, | ||
| * Data in table streaming buffer will be backed up on the next run, once the buffer is flushed. BBQ uses [copy-job](https://cloud.google.com/bigquery/docs/managing-tables#copy-table) for creating backups and *"Records in the streaming buffer are not considered when a copy or extract job runs"* (check [Life of a BigQuery streaming insert](https://cloud.google.com/blog/big-data/2017/06/life-of-a-bigquery-streaming-insert) for more details), | ||
| * BBQ may backup more than once tables which name is longer than 400 characters. |
There was a problem hiding this comment.
When table name is longer than 400 characters, then in rare cases BBQ may backup tables more than once. Such backup duplicates are automatically removed by retention process.
marcin-kolda
approved these changes
Aug 3, 2018
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.