-
Notifications
You must be signed in to change notification settings - Fork 39
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Resumable uploads errors if (optional) name is omitted #147
Comments
Thanks for the great test case! Unfortunately
I made a quick fix. #148 Could you test with it and see if If you don't want, or can't, build from the source, the
|
Must be a CDN problem or something, because GitHub doesn't have any issues fetching the packages, but I get some self-signed certificate every time. pypa/pip#10963 |
Thanks for the quick look (and very odd on pypi 🙃)! Now I got a import gcsfs
import json
fs = gcsfs.GCSFileSystem()
fs.mkdir("test")
gcsfs.GCSFileSystem().touch("test/a") # I think this one just doesn't have the name in either the body nor query param
# UnboundLocalError: local variable 'object_id' referenced before assignment
with fs.open("test/b", "w") as file:
json.dump(5, file) # this just writes the requests data with b"5"
# AttributeError: 'bytes' object has no attribute 'get' For the optional file name, I'm wondering if GCS just generates a uuid or similar upload_id and gets the name on subsequent requests? It looks like we use the object name from the first request for the |
Hmm... From the API documentation I understood that you have to pass either the query parameter or the metadata, but you shouldn't omit both. I need to take a closer look how
...
|
Ok, I've tried to figure out what's going on. I'm about 99% sure that This should work now. import os
import gcp_storage_emulator.server
import gcsfs
from google.cloud import storage
server = gcp_storage_emulator.server.create_server("localhost", 9023, in_memory=True)
server.start()
host, port = server._api._httpd.socket.getsockname()
os.environ["STORAGE_EMULATOR_HOST"] = f"http://{host}:{port}"
try:
gcs_fs = gcsfs.GCSFileSystem()
gcs_fs.mkdir("test_bucket")
gcs_fs.touch("test_bucket/file")
with gcs_fs.open("test_bucket/file", "w") as file:
file.write("testing")
client = storage.Client()
bucket = client.bucket("test_bucket")
for blob in bucket.list_blobs():
content = blob.download_as_bytes()
print(f"Blob [{blob.name}]: {content}")
finally:
server.stop() |
Wow, things look good on my end too so I'll close out the issue! Thank you for the super quick fixes! 🙌 |
Thanks for the great package!
Describe the bug
When initiating a resumable upload, the
name
query parameter is optional:However,
gcp-storage-emulator
expects to get some data on the first request (request.data.get("name")
), causing:To Reproduce
A minimal, complete, and reproducible example to reproduce the behavior.
Or, with
curl
to a running instance w/ existing bucket:Expected behavior
No error, but to redirect to a unique generated upload ID.
System (please complete the following information):
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: