-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[bigquery] Increase Storage Write API max request size #741
Conversation
// Storage Write API is limited to 10 MB, let's start out conservative and use 80% of that. | ||
maxRequestByteSize = 10_000_000 * .8 | ||
// Storage Write API is limited to 10 MiB, subtract 50 KiB to account for request overhead. | ||
maxRequestByteSize = (10 * 1024 * 1024) - (50 * 1024) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is 50 kb fixed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After a bunch of testing the cutoff seemed to be around 30 KiB so I decided to go with 50 KiB to be safe.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tested it it with a pretty wide table so I think it should be fine, but if we do see errors we can increase it to 100 KiB or higher.
@@ -223,7 +222,6 @@ func (s *Store) Dedupe(tableID sql.TableIdentifier, primaryKeys []string, includ | |||
} | |||
|
|||
func LoadBigQuery(cfg config.Config, _store *db.Store) (*Store, error) { | |||
cfg.BigQuery.LoadDefaultValues() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice
i c
…On Mon, Jun 17, 2024 at 10:09 PM Nathan ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In clients/bigquery/bigquery.go
<#741 (comment)>:
> @@ -34,13 +34,12 @@ const (
describeNameCol = "column_name"
describeTypeCol = "data_type"
describeCommentCol = "description"
- // Storage Write API is limited to 10 MB, let's start out conservative and use 80% of that.
- maxRequestByteSize = 10_000_000 * .8
+ // Storage Write API is limited to 10 MiB, subtract 50 KiB to account for request overhead.
+ maxRequestByteSize = (10 * 1024 * 1024) - (50 * 1024)
After a bunch of testing the cutoff seemed to be around 30 KiB so I
decided to go with 50 KiB to be safe.
—
Reply to this email directly, view it on GitHub
<#741 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABBVGKG6ALHTWWMRBPCT57DZH66G3AVCNFSM6AAAAABJPHXB2SVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDCMRUGQ3TOOJTG4>
.
You are receiving this because your review was requested.Message ID:
***@***.***>
|
Also kill
batchSize
config.