Skip to content

Commit

Permalink
Add documentation for ingesting JSON to a ManagedStream
Browse files Browse the repository at this point in the history
  • Loading branch information
agrawal-siddharth committed Dec 12, 2023
1 parent 2020edf commit 14ad21f
Showing 1 changed file with 16 additions and 0 deletions.
16 changes: 16 additions & 0 deletions bigquery/storage/managedwriter/doc.go
Original file line number Diff line number Diff line change
Expand Up @@ -262,5 +262,21 @@ https://pkg.go.dev/google.golang.org/api/option#WithGRPCConnectionPool
A reasonable upper bound for the connection pool size is the number of concurrent writers for explicit stream
plus the configured size of the multiplex pool.
# Writing JSON Data
As an example, you can refer to this integration test that demonstrates writing JSON data to a stream:
https://github.com/googleapis/google-cloud-go/blob/7a46b5428f239871993d66be2c7c667121f60a6f/bigquery/storage/managedwriter/integration_test.go#L397
This integration test assumes the destination table already exists. In addition, it relies upon having a definition of
a BigQuery schema that is compatible with this table (for this example the schema is defined here:
https://github.com/googleapis/google-cloud-go/blob/2020edff24e3ffe127248cf9a90c67593c303e18/bigquery/storage/managedwriter/testdata/schemas.go#L31).
Given the schema, this test first utilizes the function setupDynamicDescriptors() to derive both a MessageDescriptor
and DescriptorProto from the schema. This function is defined here:
https://github.com/googleapis/google-cloud-go/blob/7a46b5428f239871993d66be2c7c667121f60a6f/bigquery/storage/managedwriter/integration_test.go#L100
The test initializes the ManagedStream it will write to with the derived DescriptorProto. The test then iterates
through each of the JSON rows to be written. For each row, it first dynamically creates an empty Message based on
the derived MessageDescriptor. Then it loads the JSON row into the Message. Finally it generates protocol buffer
bytes from the Message. These bytes are then sent to the ManagedStream within an AppendRows request.
*/
package managedwriter

0 comments on commit 14ad21f

Please sign in to comment.