This repository has been archived by the owner on Aug 25, 2023. It is now read-only.
YACHT-1295: documented not supported use cases #136
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Pull Request Test Coverage Report for Build 1100
💛 - Coveralls |
Pull Request Test Coverage Report for Build 1184
💛 - Coveralls |
radkomateusz
suggested changes
Apr 5, 2019
radkomateusz
suggested changes
Apr 19, 2019
radkomateusz
previously approved these changes
May 6, 2019
MZatorski
suggested changes
May 7, 2019
NOT_SUPPORTED_USE_CASES.md
Outdated
| @@ -0,0 +1,42 @@ | |||
| # Rare cases in which backups are not supported by BBQ | |||
There was a problem hiding this comment.
Known (rare) cases where BBQ backups may not work as expected
NOT_SUPPORTED_USE_CASES.md
Outdated
| #### Prerequisites | ||
| * Part of data needs to be deleted | ||
| * Deleted part is whole BigQuery internal chunk of data | ||
| * There is no further changes to the partition |
There was a problem hiding this comment.
There are no further data changes in that partition
NOT_SUPPORTED_USE_CASES.md
Outdated
|
|
||
| ## Data stored in [__UNPARTITIONED__ partition](https://cloud.google.com/bigquery/docs/querying-partitioned-tables#ingestion-time_partitioned_tables_unpartitioned_partition) for a long time | ||
|
|
||
| When data is slowly streamed to BigQuery partitioned table, then that data might be moved from `__UNPARTITIONED__` partition into correct one after several hours. `lastModifiedTime` is set to time of streaming (which might be several hours ago), not when data was moved between partitions. This results in not backing up new part of data, because BBQ looks at time of last backup. |
There was a problem hiding this comment.
Replace:
This results in not backing up new part of data, because BBQ looks at time of last backup.
with:
BBQ might not discover this "late-streamed" data and skip the backup.
NOT_SUPPORTED_USE_CASES.md
Outdated
| * Ingestion-time partitioned table | ||
| * Data is streamed to the BigQuery without specifying `partitionId` | ||
| * Ingestion is very slow so that `__UNPARTITIONED__` partition store data for more than 24 hours | ||
| * There is no further changes to the partition |
There was a problem hiding this comment.
<There are no further changes to the partition
NOT_SUPPORTED_USE_CASES.md
Outdated
|
|
||
| ## Backing up empty partition | ||
|
|
||
| Due to asynchronous nature of scheduling backup for table/partition it is possible that: |
NOT_SUPPORTED_USE_CASES.md
Outdated
|
|
||
| Due to asynchronous nature of scheduling backup for table/partition it is possible that: | ||
| 1. Source data is modified and backup is scheduled for given table/partition | ||
| 1. Between scheduling copy-job task and this task execution, the data is deleted manually or by partition expiration |
There was a problem hiding this comment.
Data is deleted manually (or by partition expiration) after copy-job is scheduled but before task execution
NOT_SUPPORTED_USE_CASES.md
Outdated
|
|
||
| #### Prerequisites | ||
| * Backup is scheduled for given table/partition, i.e. `lastModifiedTime` is modified | ||
| * Between scheduling copy-job task and this task execution, the data is deleted manually or by partition expiration |
There was a problem hiding this comment.
Data is deleted manually (or by partition expiration) after copy-job is scheduled but before task execution
MZatorski
approved these changes
May 8, 2019
radkomateusz
approved these changes
May 8, 2019
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.