Wire up overrides for the S3 uploader path. #1692
Conversation
| @@ -40,6 +42,7 @@ public SingularityRunNowRequest(@JsonProperty("message") Optional<String> messag | |||
| @JsonProperty("runId") Optional<String> runId, | |||
| @JsonProperty("commandLineArgs") Optional<List<String>> commandLineArgs, | |||
| @JsonProperty("resources") Optional<Resources> resources, | |||
| @JsonProperty("s3UploaderAdditionalFiles") List<SingularityS3UploaderFile> s3UploaderAdditionalFiles, | |||
ssalinas
Jan 9, 2018
Member
remember this can be used from the client, so adding args will break the build for folks
remember this can be used from the client, so adding args will break the build for folks
baconmania
Jan 10, 2018
Author
Contributor
👍 good catch. Restored the constructor from the most recent release.
| if (configuration.getS3ConfigurationOptional().isPresent()) { | ||
| uploaderAdditionalFiles.addAll(configuration.getS3ConfigurationOptional().get().getS3UploaderAdditionalFiles()); | ||
| } | ||
| uploaderAdditionalFiles.addAll(task.getPendingTask().getS3UploaderAdditionalFiles()); |
ssalinas
Jan 9, 2018
Member
is it possible for two of these to overlap? The original use for this was to override where a specific file was being sent. Maybe we need to dedupe on the directory/glob in the additional file settings? If the uploader tries to upload it twice, one instance of that will fail when the first one done tries to delete it
is it possible for two of these to overlap? The original use for this was to override where a specific file was being sent. Maybe we need to dedupe on the directory/glob in the additional file settings? If the uploader tries to upload it twice, one instance of that will fail when the first one done tries to delete it
baconmania
Jan 10, 2018
Author
Contributor
Hmm, that's a tricky one. Since we don't have a priori information about the actual set of files in the sandbox, it's nontrivial to determine if the glob patterns overlap. We could just compare the pattern strings directly, but that would of course not cover all potential overlaps.
Hmm, that's a tricky one. Since we don't have a priori information about the actual set of files in the sandbox, it's nontrivial to determine if the glob patterns overlap. We could just compare the pattern strings directly, but that would of course not cover all potential overlaps.
ssalinas
Jan 10, 2018
Member
I would argue that some protection is better than none. Maybe we at least check for exact matches and call it good? That or another way to ensure it is to only take the override if it is set and ignore the executor settings
I would argue that some protection is better than none. Maybe we at least check for exact matches and call it good? That or another way to ensure it is to only take the override if it is set and ignore the executor settings
baconmania
Jan 10, 2018
Author
Contributor
After digging on this, it looks like a ConcurrentMap is used to ensure that two uploaders don't simultaneously try to handle the same file.
This means we won't have uploader errors due to conflicts, although it does leave a race condition in place. The way to solve this in the future would be to ensure that upload overrides for a file use a matching directory and glob strings as the corresponding global config, and then just compare those fields directly instead of trying to compute glob overlaps.
After digging on this, it looks like a ConcurrentMap is used to ensure that two uploaders don't simultaneously try to handle the same file.
This means we won't have uploader errors due to conflicts, although it does leave a race condition in place. The way to solve this in the future would be to ensure that upload overrides for a file use a matching directory and glob strings as the corresponding global config, and then just compare those fields directly instead of trying to compute glob overlaps.
|
|
No description provided.