Extend upload endpoint to accept both types - clks and clknblocks #503
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hardbyte
requested changes
Feb 17, 2020
wilko77
reviewed
Feb 18, 2020
hardbyte
approved these changes
Feb 19, 2020
Comment on lines
63
to
66
| # check if the following combinations are true | ||
| # - uses_blocking is False AND 'clks' element in upload JSON | ||
| # - uses_blocking if True AND 'clknblocks' element in upload JSON | ||
| # otherwise, return safe_fail_request |
There was a problem hiding this comment.
Nitpick: This might as well be a docstring. (code comments like this don't come up IDE tooltips)
| @@ -216,8 +220,7 @@ def project_clks_post(project_id): | |||
| # However, as connexion is very, very strict about input validation when it comes to json, it will always | |||
| # consume the stream first to validate it against the spec. Thus the backflip to fully reading the CLks as | |||
| # json into memory. -> issue #184 | |||
|
|
|||
| receipt_token, raw_file = upload_json_clk_data(dp_id, get_json(), span) | |||
| receipt_token, raw_file = upload_json_clk_data(dp_id, get_json(), span, uses_blocking) | |||
There was a problem hiding this comment.
Another nitpick. Utility arguments such as span should come after the useful ones.
Comment on lines
355
to
356
| Convert user provided encodings from json array of base64 data into | ||
| a newline separated file of base64 data. |
There was a problem hiding this comment.
I think this docstring is now out of date?
Comment on lines
380
to
374
| logger.info('Writing uploaded {} JSON to {}'.format(element.upper(), tmp.name)) | ||
| json.dump(clk_json, tmp) | ||
|
|
||
| logger.info(f"Received {count} encodings. Uploading {fmt_bytes(num_bytes)} to object store") |
There was a problem hiding this comment.
This now looks like a debug log statement - the tmp.name won't actually be around after the tmp file goes out of scope.
Could be worth logging the object store filename at info level in here though.
Closed
|
Closing after rebasing into |
Merged
This was referenced Apr 30, 2020
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
By introducting blocking in
anonlink-client, we also want to updateentity-serviceto support blocking.In this pull request, we extended the upload endpoint i.e.
upload_json_clk_datato accept both uploading types:clkselementclknblockselementwith a focus on error handling.
Note that we also changed the way that writes to MINIO from
put_objecttofput_objectto avoid large in memory bytes conversion.Currently it will fail CI since it is only the start of this update to support blocking and more modifications are required to make the whole entity service work with blocking.
Close #501