-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
docs: BigQuery data plane description #182
docs: BigQuery data plane description #182
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can use this as a starting point, but I think the concept of streaming query results as pages is not 100% where it should be.
Further comments inline.
|
||
## Data source | ||
|
||
The data source supports queries with [named parameters](https://cloud.google.com/bigquery/docs/parameterized-queries). To set the values of the named parameters, the `BigQueryRequestParams` generated by `BigQueryRequestParamsProvider.provideSourceParams` (invoked from `BigQueryDataSourceFactory.createSource`) includes also the sink address passed when the transfer process is started by the consumer. The sink address must provide the values of the parameters in the query in the form: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The data source supports queries with named parameters
mentioning that they are analogous to SQL query parameters may let people understand it even better.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will update the doc, thanks!
…of single Part in the sink
Agree, the concept of multi part should more about parallelizing, this is an optimization that can be taken over once the functionality is overall confirmed. |
What this PR changes/adds
Added description of the first implementation for BQ data plane.