Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Blob's custom_time is lost on upload. #372

Closed
oittaa opened this issue Feb 5, 2021 · 2 comments
Closed

Blob's custom_time is lost on upload. #372

oittaa opened this issue Feb 5, 2021 · 2 comments

Comments

@oittaa
Copy link

@oittaa oittaa commented Feb 5, 2021

Environment details

  • OS type and version: Debian Buster
  • Python version: Python 3.7.3
  • pip version: pip 18.1 from /usr/lib/python3/dist-packages/pip (python 3.7)
  • google-cloud-storage version: 1.35.1

Steps to reproduce

  1. Create a blob.
  2. Set custom_time.
  3. Upload the blob .
  4. Try to retrieve blob's custom_time.
  5. Expected result is a datatime, but the actual result is None.

If blob's custom_time is set before uploading, it is just lost. My understanding is that the metadata could be uploaded with the object's content at the same time without needing to patch it afterwards.

https://cloud.google.com/storage/docs/uploading-objects#rest-upload-objects

Code example

import datetime
from google.cloud import storage

bucket_name = "my-test-bucket"
blob_name = "test_blob"

client = storage.Client()
bucket = client.get_bucket(bucket_name)

blob = bucket.blob(blob_name)
blob.custom_time = datetime.datetime.now(datetime.timezone.utc)
print("Before upload {}".format(blob.custom_time))
blob.upload_from_string("test")
blob.reload()
print("After upload {}".format(blob.custom_time))
blob.custom_time = datetime.datetime.now(datetime.timezone.utc)
blob.patch()
blob.reload()
print("After patching {}".format(blob.custom_time))

Stack trace

Before upload 2021-02-05 10:24:08.405494+00:00
After upload None
After patching 2021-02-05 10:24:08.814159+00:00
@oittaa
Copy link
Author

@oittaa oittaa commented Feb 6, 2021

For example content_encoding works as expected.

Code

from google.cloud import storage

bucket_name = "my-test-bucket"
blob_name = "test_blob"

client = storage.Client()
bucket = client.get_bucket(bucket_name)

blob = bucket.blob(blob_name)
blob.content_encoding = "gzip"
print("Before upload {}".format(blob.content_encoding))
blob.upload_from_string("test")
blob.reload()
print("After upload {}".format(blob.content_encoding))

Output

Before upload gzip
After upload gzip

Is the reason that customTime is missing from _WRITABLE_FIELDS? EDIT: Yes it is. Adding "customTime", line fixed the issue.

_WRITABLE_FIELDS = (
"cacheControl",
"contentDisposition",
"contentEncoding",
"contentLanguage",
_CONTENT_TYPE_FIELD,
"crc32c",
"md5Hash",
"metadata",
"name",
"storageClass",
)

Which is used by _get_writable_metadata.

def _get_writable_metadata(self):
"""Get the object / blob metadata which is writable.
This is intended to be used when creating a new object / blob.
See the `API reference docs`_ for more information, the fields
marked as writable are:
* ``acl``
* ``cacheControl``
* ``contentDisposition``
* ``contentEncoding``
* ``contentLanguage``
* ``contentType``
* ``crc32c``
* ``md5Hash``
* ``metadata``
* ``name``
* ``storageClass``
For now, we don't support ``acl``, access control lists should be
managed directly through :class:`ObjectACL` methods.
"""
# NOTE: This assumes `self.name` is unicode.
object_metadata = {"name": self.name}
for key in self._changes:
if key in _WRITABLE_FIELDS:
object_metadata[key] = self._properties[key]
return object_metadata

And which is then used by _get_upload_arguments.

def _get_upload_arguments(self, content_type):
"""Get required arguments for performing an upload.
The content type returned will be determined in order of precedence:
- The value passed in to this method (if not :data:`None`)
- The value stored on the current blob
- The default value ('application/octet-stream')
:type content_type: str
:param content_type: Type of content being uploaded (or :data:`None`).
:rtype: tuple
:returns: A triple of
* A header dictionary
* An object metadata dictionary
* The ``content_type`` as a string (according to precedence)
"""
headers = _get_encryption_headers(self._encryption_key)
object_metadata = self._get_writable_metadata()
content_type = self._get_content_type(content_type)
return headers, object_metadata, content_type

@tritone
Copy link
Contributor

@tritone tritone commented Feb 8, 2021

I was able to reproduce this and discovered custom_time was missing from attributes to set during blob creation. Will send a PR with a fix.

tritone added a commit to tritone/python-storage that referenced this issue Feb 8, 2021
The custom_time property was incorrectly not included as a field
that could be set on object upload.

Fixes googleapis#372
@tritone tritone closed this in #374 Feb 8, 2021
tritone added a commit that referenced this issue Feb 8, 2021
The custom_time property was incorrectly not included as a field
that could be set on object upload.

Fixes #372
oittaa added a commit to oittaa/flask-caching that referenced this issue Feb 13, 2021
oittaa added a commit to oittaa/avif-converter that referenced this issue Feb 13, 2021
cojenco added a commit to cojenco/python-storage that referenced this issue Oct 13, 2021
The custom_time property was incorrectly not included as a field
that could be set on object upload.

Fixes googleapis#372
cojenco added a commit to cojenco/python-storage that referenced this issue Oct 13, 2021
The custom_time property was incorrectly not included as a field
that could be set on object upload.

Fixes googleapis#372
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

4 participants