Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Accessing blob.custom_time errors on 0 microsecond timestamp #363

Closed
rconnaughton opened this issue Jan 22, 2021 · 2 comments
Closed

Accessing blob.custom_time errors on 0 microsecond timestamp #363

rconnaughton opened this issue Jan 22, 2021 · 2 comments

Comments

@rconnaughton
Copy link

@rconnaughton rconnaughton commented Jan 22, 2021

Environment details

  • OS type and version: centos 6
  • Python version: 3.7
  • google-cloud-storage version: 1.34.0

Steps to reproduce

  1. Load a blob
  2. Set the custom_time field to a datetime object with 0 microseconds
  3. patch the object with the changes, and then reload the object
  4. Access the custom_time field and observe the value error

Code example

blob = Blob(mybucket, objectname)
blob.reload()
blob.custom_time = datetime(year=2000, month = 1, day = 13, hour = 13, minute = 1, second = 0, microsecond = 0)
blob.patch()
blob.reload()
blob.custom_time
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 3773, in custom_time
    return _rfc3339_to_datetime(value)_rfc3339_to_datetime
  File "/usr/local/lib/python3.7/site-packages/google/cloud/_helpers.py", line 286, in _rfc3339_to_datetime
    return datetime.datetime.strptime(dt_str, _RFC3339_MICROS).replace(tzinfo=UTC)
  File "/usr/local/lib/python3.7/_strptime.py", line 577, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
  File "/usr/local/lib/python3.7/_strptime.py", line 359, in _strptime
    (data_string, format))
ValueError("time data '2000-01-13T13:01:00Z' does not match format '%Y-%m-%dT%H:%M:%S.%fZ'")

As compared to a working example:

blob.custom_time = datetime(year=2000, month = 1, day = 13, hour = 14, minute = 2, second = 0, microsecond = 10)
blob.patch()
blob.reload()
blob.custom_time
datetime.datetime(2000, 1, 13, 14, 2, 0, 10, tzinfo=<UTC>)

Stack trace

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 3773, in custom_time
    return _rfc3339_to_datetime(value)_rfc3339_to_datetime
  File "/usr/local/lib/python3.7/site-packages/google/cloud/_helpers.py", line 286, in _rfc3339_to_datetime
    return datetime.datetime.strptime(dt_str, _RFC3339_MICROS).replace(tzinfo=UTC)
  File "/usr/local/lib/python3.7/_strptime.py", line 577, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
  File "/usr/local/lib/python3.7/_strptime.py", line 359, in _strptime
    (data_string, format))
ValueError("time data '2000-01-13T13:01:00Z' does not match format '%Y-%m-%dT%H:%M:%S.%fZ'")
@frankyn
Copy link
Member

@frankyn frankyn commented Jan 25, 2021

Hi @rconnaughton,

Thank you for filing this issue. I was able to reproduce it and looks like Cloud Storage returns no fractional microsecond when it's 0 even though the format described in documentation is YYYY-MM-DD'T'HH:MM:SS.SS'Z'.

The issue with the client is that it always expects the returned value to include microseconds. I'm following up with the backend team to understand if this is the expectation.

I'll loop back when I learn more. Thank you for your patience!

@frankyn
Copy link
Member

@frankyn frankyn commented Feb 1, 2021

This is a bug in the library. The Storage backend team stated that when milliseconds is 0 the 0 is not returned.

I'll work on a fix this week. It will require two PRs based on my current understanding, there might be a better solution though and will research / update:

  1. Add a new format to python-cloud-core.
  2. Use the new format in case the format parsing with non-zero milliseconds is included in python-storage.

@frankyn frankyn assigned tritone and unassigned frankyn Feb 8, 2021
tritone added a commit to tritone/python-storage that referenced this issue Feb 9, 2021
Currently custom_time is not being decoded correctly if the value
has a zero in the microseconds field. This fixes the issue by
using a different helper method to decode.

Fixes googleapis#363
tritone added a commit to tritone/python-storage that referenced this issue Feb 9, 2021
Currently custom_time is not being decoded correctly if the value
has a zero in the microseconds field. This fixes the issue for
custom_time as well as elsewhere by replacing _datetime_to_rfc3339
with _rfc3339_nanos_to_datetime.

Fixes googleapis#363
gcf-merge-on-green bot pushed a commit that referenced this issue Feb 10, 2021
Currently custom_time is not being decoded correctly if the value
has a zero in the microseconds field. This fixes the issue for
custom_time as well as elsewhere by replacing _datetime_to_rfc3339
with _rfc3339_nanos_to_datetime.

Fixes #363
cojenco added a commit to cojenco/python-storage that referenced this issue Oct 13, 2021
Currently custom_time is not being decoded correctly if the value
has a zero in the microseconds field. This fixes the issue for
custom_time as well as elsewhere by replacing _datetime_to_rfc3339
with _rfc3339_nanos_to_datetime.

Fixes googleapis#363
cojenco added a commit to cojenco/python-storage that referenced this issue Oct 13, 2021
Currently custom_time is not being decoded correctly if the value
has a zero in the microseconds field. This fixes the issue for
custom_time as well as elsewhere by replacing _datetime_to_rfc3339
with _rfc3339_nanos_to_datetime.

Fixes googleapis#363
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

4 participants