Simplify upload handling #246
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
boydgreenfield
force-pushed
the
ng-simpler-uploads
branch
2 times, most recently
from
a3ea193
to
ef5d757
Compare
|
Ha... |
boydgreenfield
force-pushed
the
ng-simpler-uploads
branch
11 times, most recently
from
32a9f20
to
a345b61
Compare
boydgreenfield
force-pushed
the
ng-simpler-uploads
branch
from
a345b61
to
74a50e9
Compare
This commit also reworks the passing of the click.progressbar to the uploader and requires it's passed as an object vs. boolean from the top invoking call. This better supports both library and CLI usage.
Specifically we add support for "canceling" and "finalizign" statuses in the click.progressbar element. Note that we could generalize this further by subclassing the click internal class, but it's already a sufficiently gross hack that we're probably better off *not* further extending the click internals.
…s threads We do this via the use of the `atexit` module, which will handle any program exits. Note we have to use our own `atexit_unregister` function because `atexit.unregister` is unavailable in Python 2.7.
Notably, remove the use of boolean return values and rely exclusively on exception hierarchy and retry-able vs. non-retryable exceptions. This includes minor flow control logic changes (exceptions vs. return values), but no major changes to the upload code paths.
Effectively, we do the following here: 1. Always check for /status if available when uploading via a proxy 2. If /status has a `complete: false` field, poll up to 15 minutes 3. Raise a RetryableUploadException or fail based on /status code (2) specifically fixes bugs we were encountering wherein the Python client could send all of the data and either: A. Send a file with an error but not be listening for the reply, experience a ConnectionError, and then retry automatically even if the file was invalid and the proxy returned a 400 B. Send data too fast for the proxy to compress (with data getting buffered in the ALB/nginx) and then abort *despite the proxy ultimately completing successfully* (A) led to bad files not being caught by the proxy. (B) could lead to duplicate uploads to to erroneous retry requests. Both of these fixes bring our implementation into line with the frontend JS code on app.onecodex.com.
boydgreenfield
force-pushed
the
ng-simpler-uploads
branch
from
74a50e9
to
cf5fdd7
Compare
No longer display "INFO" for logging.info messages when invoked via the CLI. We can now further configure these for both Python 2 and Python 3.
Closed
6 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uses
concurrent.futuresto simplify threading greatly.Todos:
KeyboardInterruptworks properly (once only!)