Update BQ row limit to 100MB and make it more efficient. #417
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Pull Request Test Coverage Report for Build 1453
💛 - Coveralls |
allieychen
approved these changes
Nov 23, 2018
Contributor
allieychen
left a comment
allieychen
left a comment
There was a problem hiding this comment.
It looks awesome! LGTM! Can you also update https://github.com/googlegenomics/gcp-variant-transforms/blob/master/docs/bigquery_schema.md as well?
| # on sampling rather than exact byte size. | ||
| _MAX_BIGQUERY_ROW_SIZE_BYTES = 90 * 1024 * 1024 | ||
| # Maximum number of calls to sample for BigQuery row size estimate. | ||
| _MAX_NUM_CALL_SAMPLES = 5 |
Contributor
There was a problem hiding this comment.
If I understand correctly, this is actually the min number of calls (we always sample 5 calls or 5+1 calls). Maybe NUM_CALL_SAMPLES is good enough.
Contributor
Author
There was a problem hiding this comment.
Good point! Done!
arostamianfar
commented
Nov 26, 2018
Contributor
Author
arostamianfar
left a comment
arostamianfar
left a comment
There was a problem hiding this comment.
Thanks! Updated the doc as well! Although we need to remove that page completely once the cloud page is updated.
| # on sampling rather than exact byte size. | ||
| _MAX_BIGQUERY_ROW_SIZE_BYTES = 90 * 1024 * 1024 | ||
| # Maximum number of calls to sample for BigQuery row size estimate. | ||
| _MAX_NUM_CALL_SAMPLES = 5 |
Contributor
Author
There was a problem hiding this comment.
Good point! Done!
This was referenced Nov 26, 2018
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
BQ row limit has changed from 10MB to 100MB. I also made this method more efficient as it turns out jsonifying every single call/variant is a very expensive operation! This PTransform is now 50% faster! The sampling logic can be improved (e.g. randomizing the samples rather than picking particular locations, or dynamically choosing sample size based on input), but I realistically don't think anyone is going to hit the 100MB limit anytime soon and this method can be good enough even then.
Tested:
huge_testslater.