Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes for Immediate Uploaders #1418

Merged
merged 1 commit into from Feb 10, 2017

Conversation

@ssalinas
Copy link
Member

ssalinas commented Feb 9, 2017

@PtrTeixeira after looking at the code for this for a while I realized a few things:

  • It's quite messy to deal with the single uploader metric counter when the immediate vs regular uploaders are treated so differently
  • We weren't actually able to guarantee that the files eventually got uploaded in the exception case. i.e. if there is an exception from S3, we probably need to retry that still.

So, I updated this to have a map to keep track of the immediate uploaders and have a separate metric counter for them as well. We submit the callables to the executor service immediately, but rather than working with the callbacks, we check the results on our regular polling interval, restarting them if there are exceptions (only retrying for exception, not for no files found right now)

This probably still needs a few tweaks, can you take it from here on this one?

@PtrTeixeira
Copy link
Contributor

PtrTeixeira commented Feb 9, 2017

👀

@ssalinas ssalinas modified the milestone: 0.14.0 Feb 9, 2017
@ssalinas ssalinas merged commit b4c92bc into s3uploader-upload-immediately Feb 10, 2017
2 checks passed
2 checks passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
@ssalinas ssalinas deleted the immediate_upload_fixes branch Jun 8, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants
You can’t perform that action at this time.