Fixes for Immediate Uploaders #1418

Merged
merged 1 commit into from Feb 10, 2017

Conversation

Projects
None yet
2 participants
@ssalinas
Member

ssalinas commented Feb 9, 2017

@PtrTeixeira after looking at the code for this for a while I realized a few things:

  • It's quite messy to deal with the single uploader metric counter when the immediate vs regular uploaders are treated so differently
  • We weren't actually able to guarantee that the files eventually got uploaded in the exception case. i.e. if there is an exception from S3, we probably need to retry that still.

So, I updated this to have a map to keep track of the immediate uploaders and have a separate metric counter for them as well. We submit the callables to the executor service immediately, but rather than working with the callbacks, we check the results on our regular polling interval, restarting them if there are exceptions (only retrying for exception, not for no files found right now)

This probably still needs a few tweaks, can you take it from here on this one?

@PtrTeixeira

This comment has been minimized.

Show comment
Hide comment
@PtrTeixeira

PtrTeixeira Feb 9, 2017

Contributor

👀

Contributor

PtrTeixeira commented Feb 9, 2017

👀

@ssalinas ssalinas modified the milestone: 0.14.0 Feb 9, 2017

@ssalinas ssalinas merged commit b4c92bc into s3uploader-upload-immediately Feb 10, 2017

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@ssalinas ssalinas deleted the immediate_upload_fixes branch Jun 8, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment