Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Uploader refactoring and additional attempts for immediate uploaders #1724

Merged
merged 6 commits into from Feb 26, 2018

Conversation

@ssalinas
Copy link
Member

@ssalinas ssalinas commented Feb 20, 2018

@darcatron I'm folding #1714 into this as well to avoid too many merge conflicts. Will close the other when they are both in hs_qa

There were a few cases where an immediate uploader could miss files. Particularly, if an uploader was immediate, but then something attempted to recreate it (wrote the file again), before the original one expired and was removed. This updates the uploader driver to:

  • Trigger additional immediately upload attempts when the metadata is written to
  • Check for any remaining files matching the glob on immediate uploader expiration, uploading them if some are found
  • Refactor a number of things to java8-ify them and store immediate uploaders in a way that better matches regular uploaders
private final List<S3UploadMetadata> immediateUploadMetadata;
private final ReentrantLock lock;
private final ConcurrentMap<SingularityUploader, Future<Integer>> immediateUploaders;
private final Map<S3UploadMetadata, SingularityUploader> metadataToimmediateUploader;

This comment has been minimized.

@darcatron

darcatron Feb 20, 2018
Contributor

nit metadataToimmediateUploader -> metadataToImmediateUploader (capitalize immediate)

This comment has been minimized.

@ssalinas

ssalinas Feb 20, 2018
Author Member

fixed

LOG.debug("Retrying immediate uploader {}", uploaderMetadata);
performImmediateUpload(uploader);
} else {
LOG.debug("Uploader for metadata {} not found to retry, recreating", uploaderMetadata);

This comment has been minimized.

@darcatron

darcatron Feb 20, 2018
Contributor

not sure I understand this debug line. What is being re-created?

This comment has been minimized.

@ssalinas

ssalinas Feb 20, 2018
Author Member

Ah, copy-pasta error. the ,recreating should be removed, if we don't find the uploader in the map it means it somehow got added to both toRemove and toRetry, which theoretically should never happen, but better than throwing a NPE, updated the message

SingularityUploader uploader = metadataToimmediateUploader.remove(uploaderMetadata);
if (uploader != null) {
LOG.debug("Retrying immediate uploader {}", uploaderMetadata);
performImmediateUpload(uploader);

This comment has been minimized.

@darcatron

darcatron Feb 20, 2018
Contributor

For this retry loop, the uploader is removed from metadataToImmeditateUploader and when it's called to upload immediately, it'll be added to the immediteUploadersFutures map. During the next run of checkUploads, won't the uploader be missing from metadataToImmediateUploader and cause the check to skip adding the uploadedFiles to the total count? Instead it'll be moved to the toRemove list and hit the 30 second check

This comment has been minimized.

@ssalinas

ssalinas Feb 20, 2018
Author Member

good catch. It ends up working out that it still retries, but that should be a .get so we can do the full retry -> toRemove loop . Fixed

ssalinas added 3 commits Feb 20, 2018
@darcatron
Copy link
Contributor

@darcatron darcatron commented Feb 20, 2018

lgtm 🚢

@ssalinas ssalinas added this to the 0.20.0 milestone Feb 20, 2018
@ssalinas ssalinas added the hs_stable label Feb 21, 2018
@ssalinas ssalinas merged commit 48f5435 into master Feb 26, 2018
2 checks passed
2 checks passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
@ssalinas ssalinas deleted the uploader_fixes branch Feb 26, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants
You can’t perform that action at this time.