Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: read_session optional to ReadRowsStream.rows() #228

Merged
merged 5 commits into from Jul 9, 2021

Conversation

tswast
Copy link
Contributor

@tswast tswast commented Jul 8, 2021

The schema from the first ReadRowsResponse message can be used to decode
messages, instead.

Note: to_arrow() and to_dataframe() do not work on an empty stream unless a
read_session has been passed in, as the schema is not available. This should
not affect google-cloud-bigquery and pandas-gbq, as those packages use the
lower-level message->dataframe/arrow methods.

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes #168 馃

The schema from the first `ReadRowsResponse` message can be used to decode
messages, instead.

Note: `to_arrow()` and `to_dataframe()` do not work on an empty stream unless a
`read_session` has been passed in, as the schema is not available. This should
not affect `google-cloud-bigquery` and `pandas-gbq`, as those packages use the
lower-level message->dataframe/arrow methods.
@tswast tswast requested review from as code owners Jul 8, 2021
@tswast tswast requested review from shollyman and removed request for Jul 8, 2021
@product-auto-label product-auto-label bot added the api: bigquerystorage label Jul 8, 2021
@google-cla google-cla bot added the cla: yes label Jul 8, 2021
google/cloud/bigquery_storage_v1/reader.py Outdated Show resolved Hide resolved
google/cloud/bigquery_storage_v1/reader.py Outdated Show resolved Hide resolved
google/cloud/bigquery_storage_v1/reader.py Outdated Show resolved Hide resolved
tests/system/conftest.py Show resolved Hide resolved
tests/system/conftest.py Show resolved Hide resolved
Co-authored-by: Tres Seaver <tseaver@palladion.com>
@google-cla
Copy link

@google-cla google-cla bot commented Jul 8, 2021

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

鈩癸笍 Googlers: Go here for more info.

@google-cla google-cla bot added cla: no and removed cla: yes labels Jul 8, 2021
@tseaver
Copy link
Contributor

@tseaver tseaver commented Jul 8, 2021

@googlebot I consent.

@google-cla
Copy link

@google-cla google-cla bot commented Jul 8, 2021

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

鈩癸笍 Googlers: Go here for more info.

@tseaver
Copy link
Contributor

@tseaver tseaver commented Jul 8, 2021

@googlebot I fixed it.

@google-cla
Copy link

@google-cla google-cla bot commented Jul 8, 2021

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

鈩癸笍 Googlers: Go here for more info.

@tseaver
Copy link
Contributor

@tseaver tseaver commented Jul 8, 2021

@tswast I give up on the CLA bit.

@google-cla
Copy link

@google-cla google-cla bot commented Jul 8, 2021

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

鈩癸笍 Googlers: Go here for more info.

@tswast
Copy link
Contributor Author

@tswast tswast commented Jul 8, 2021

@tswast I give up on the CLA bit.

No worries. I'll manually override the bot.

@google-cla
Copy link

@google-cla google-cla bot commented Jul 8, 2021

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

鈩癸笍 Googlers: Go here for more info.

@tswast tswast added cla: yes and removed cla: no labels Jul 8, 2021
@tswast tswast requested a review from tseaver Jul 8, 2021
tseaver
tseaver approved these changes Jul 8, 2021
@@ -421,6 +417,7 @@ def test_to_dataframe_empty_w_scalars_avro(class_under_test):
avro_blocks = _bq_to_avro_blocks([], avro_schema)
reader = class_under_test(avro_blocks, mock_gapic_client, "", 0, {})

# Read session is needed to get a schema for empty streams.
Copy link
Contributor

@shollyman shollyman Jul 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we think this is enough of a nuisance to request that empty streams always have at least one first response where the row data is null or empty?

Copy link
Contributor Author

@tswast tswast Jul 9, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe? I'm not sure how often it'll come up now that they don't do automatic rebalancing.

I'm actually considering deprecating this method altogether, as it didn't turn out to be all that necessary. #230

@tswast tswast merged commit 4f56029 into googleapis:master Jul 9, 2021
9 checks passed
@tswast tswast deleted the issue168-read_session branch Jul 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquerystorage cla: yes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants