Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: preserve timestamp microsecond precision with rows from REST API #402

Merged
merged 7 commits into from
Dec 4, 2020

Conversation

HemangChothani
Copy link
Contributor

@HemangChothani HemangChothani commented Nov 25, 2020

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes #395

@google-cla google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Nov 25, 2020
@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery API. label Nov 25, 2020
@@ -82,7 +82,18 @@ def _timestamp_from_json(value, field):
"""Coerce 'value' to a datetime, if set or not nullable."""
if _not_null(value, field):
# value will be a float in seconds, to microsecond precision, in UTC.
return _datetime_from_microseconds(1e6 * float(value))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will need to be modified to parse from an rfc timestamp string.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, i didn't get the point. Before passing params["formatOptions.useInt64Timestamp"] = True in client.list_row, received a value in string with decimal in _timestamp_from_json method , but after passing it, received a value in string without decimal and call the method _datetime_from_microseconds(int(value)) which returns datetime object.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I see. Yes, that's expected.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this case, we should update the comment.

Also, are there cases where floating point values are still passed in? If not, we should clean this up now. If so, let's add a TODO to identify those cases and file an issue to clean them up.

elif value.isdigit():
value = int(value)
else:
value = 1e6 * float(value)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this case needed? Couldn't the above isdigit() check be removed and we always fall back to converting to integer? I don't expect floating point values to be serialized as strings in JSON.

@HemangChothani HemangChothani marked this pull request as ready for review December 2, 2020 15:11
@HemangChothani HemangChothani requested a review from a team December 2, 2020 15:11
return _datetime_from_microseconds(1e6 * float(value))
# value will be a integer in seconds, to microsecond precision, in UTC.

# if value is not None:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove commented out code.

@tswast tswast changed the title feat: add formatOption default tru for tablelist and query result fix: preserve timestamp microsecond precision with rows from REST API Dec 4, 2020
@tswast tswast merged commit 04510a7 into googleapis:master Dec 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/python-bigquery API. cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Use DataFormatOptions to download timestamps with full precision
2 participants