Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes for Immediate Uploaders #1418

merged 1 commit into from Feb 10, 2017


Copy link

@ssalinas ssalinas commented Feb 9, 2017

@PtrTeixeira after looking at the code for this for a while I realized a few things:

  • It's quite messy to deal with the single uploader metric counter when the immediate vs regular uploaders are treated so differently
  • We weren't actually able to guarantee that the files eventually got uploaded in the exception case. i.e. if there is an exception from S3, we probably need to retry that still.

So, I updated this to have a map to keep track of the immediate uploaders and have a separate metric counter for them as well. We submit the callables to the executor service immediately, but rather than working with the callbacks, we check the results on our regular polling interval, restarting them if there are exceptions (only retrying for exception, not for no files found right now)

This probably still needs a few tweaks, can you take it from here on this one?

Copy link

PtrTeixeira commented Feb 9, 2017


@ssalinas ssalinas modified the milestone: 0.14.0 Feb 9, 2017
@ssalinas ssalinas merged commit b4c92bc into s3uploader-upload-immediately Feb 10, 2017
@ssalinas ssalinas deleted the immediate_upload_fixes branch Jun 8, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
None yet

Successfully merging this pull request may close these issues.

None yet

2 participants