Skip to content

Commit

Permalink
docs: clarify size limitations for AppendRowsRequest (#283)
Browse files Browse the repository at this point in the history
* docs: clarify size limitations for AppendRowsRequest
chore: add preferred_min_stream_count to CreateReadSessionRequest
chore: add write_stream to AppendRowsResponse

PiperOrigin-RevId: 463602530

Source-Link: googleapis/googleapis@d33b3fa

Source-Link: googleapis/googleapis-gen@90995f6
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiOTA5OTVmNjQzM2QwZWNkMjkwZjE4NjE2OGNlOTU3ZDZhMGRiOWM2OCJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
  • Loading branch information
gcf-owl-bot[bot] and gcf-owl-bot[bot] committed Aug 1, 2022
1 parent c1a36c9 commit 7bd13e7
Show file tree
Hide file tree
Showing 7 changed files with 107 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -256,6 +256,16 @@ message CreateReadSessionRequest {
// determine an upper bound OR set this a size for the maximum "units of work"
// it can gracefully handle.
int32 max_stream_count = 3;

// The minimum preferred stream count. This parameter can be used to inform
// the service that there is a desired lower bound on the number of streams.
// This is typically a target parallelism of the client (e.g. a Spark
// cluster with N-workers would set this to a low multiple of N to ensure
// good cluster utilization).
//
// The system will make a best effort to provide at least this number of
// streams, but in some cases might provide less.
int32 preferred_min_stream_count = 4;
}

// Request message for `ReadRows`.
Expand Down Expand Up @@ -395,6 +405,9 @@ message CreateWriteStreamRequest {
// Due to the nature of AppendRows being a bidirectional streaming RPC, certain
// parts of the AppendRowsRequest need only be specified for the first request
// sent each time the gRPC network connection is opened/reopened.
//
// The size of a single AppendRowsRequest must be less than 10 MB in size.
// Requests larger than this return an error, typically `INVALID_ARGUMENT`.
message AppendRowsRequest {
// ProtoData contains the data rows and schema when constructing append
// requests.
Expand Down Expand Up @@ -495,6 +508,10 @@ message AppendRowsResponse {
// appended. The API will return row level error info, so that the caller can
// remove the bad rows and retry the request.
repeated RowError row_errors = 4;

// The target of the append operation. Matches the write_stream in the
// corresponding request.
string write_stream = 5;
}

// Request message for `GetWriteStreamRequest`.
Expand Down
12 changes: 12 additions & 0 deletions packages/google-cloud-bigquery-storage/protos/protos.d.ts

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

47 changes: 46 additions & 1 deletion packages/google-cloud-bigquery-storage/protos/protos.js

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 8 additions & 0 deletions packages/google-cloud-bigquery-storage/protos/protos.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,16 @@ function main(parent, readSession) {
* it can gracefully handle.
*/
// const maxStreamCount = 1234
/**
* The minimum preferred stream count. This parameter can be used to inform
* the service that there is a desired lower bound on the number of streams.
* This is typically a target parallelism of the client (e.g. a Spark
* cluster with N-workers would set this to a low multiple of N to ensure
* good cluster utilization).
* The system will make a best effort to provide at least this number of
* streams, but in some cases might provide less.
*/
// const preferredMinStreamCount = 1234

// Imports the Storage library
const {BigQueryReadClient} = require('@google-cloud/bigquery-storage').v1;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
"segments": [
{
"start": 25,
"end": 68,
"end": 78,
"type": "FULL"
}
],
Expand All @@ -42,6 +42,10 @@
{
"name": "max_stream_count",
"type": "TYPE_INT32"
},
{
"name": "preferred_min_stream_count",
"type": "TYPE_INT32"
}
],
"resultType": ".google.cloud.bigquery.storage.v1.ReadSession",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -376,6 +376,15 @@ export class BigQueryReadClient {
* Typically, clients should either leave this unset to let the system to
* determine an upper bound OR set this a size for the maximum "units of work"
* it can gracefully handle.
* @param {number} request.preferredMinStreamCount
* The minimum preferred stream count. This parameter can be used to inform
* the service that there is a desired lower bound on the number of streams.
* This is typically a target parallelism of the client (e.g. a Spark
* cluster with N-workers would set this to a low multiple of N to ensure
* good cluster utilization).
*
* The system will make a best effort to provide at least this number of
* streams, but in some cases might provide less.
* @param {object} [options]
* Call options. See {@link https://googleapis.dev/nodejs/google-gax/latest/interfaces/CallOptions.html|CallOptions} for more details.
* @returns {Promise} - The promise which resolves to an array.
Expand Down

0 comments on commit 7bd13e7

Please sign in to comment.