-
Notifications
You must be signed in to change notification settings - Fork 87
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Chunked upload support #1468
Chunked upload support #1468
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMHO file
and print
and maybe end of line need to be changed then 👍
udata/core/dataset/api_fields.py
Outdated
@@ -187,4 +187,4 @@ | |||
description='The web page URL for this dataset', readonly=True), | |||
'score': fields.Float( | |||
description='The internal match score', required=True), | |||
}) | |||
}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🤔
udata/core/storages/api.py
Outdated
|
||
def on_upload_status(status): | ||
'''Not an error, just raised when chunk is processed''' | ||
print('in handler', status.ok, status.error) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove?
udata/core/storages/api.py
Outdated
def handle_upload(storage, prefix=None): | ||
args = upload_parser.parse_args() | ||
is_chunk = args['totalparts'] > 1 | ||
file = args['file'] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reserved name in Python 2
|
||
|
||
def extension(filename): | ||
'''Properly extract the extension from filename''' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cool, we had problems with that 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we'll still have problems with that kind of filenames though Capture d'écran 2018-02-13 16.03.07.png
. Maybe check against a list of known extensions and if found do not try to parse a compounded one? But it could get complicated.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I didn't changed anything, I just grouped an existing function.
I think I can improve the behavior, but in another PR
udata/tests/api/test_datasets_api.py
Outdated
'partbyteoffset': 0, | ||
'totalfilesize': parts, | ||
'totalparts': parts, | ||
'chunksize': 1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add a comma here if we you add one on the last item of the dict below ;-)
udata/tests/test_storages.py
Outdated
'partbyteoffset': 0, | ||
'totalfilesize': parts, | ||
'totalparts': parts, | ||
'chunksize': 1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
,
Changes done |
This PR adds chunked upload support, ie. uploads are no more limited by max_body_size and udata is able to handle big files upload.
This is is a first pass only adding chunked and concurrent upload support.
What can be improved in other PRs: