Wire up overrides for the S3 uploader path. - #1692
Conversation
| @JsonProperty("runId") Optional<String> runId, | ||
| @JsonProperty("commandLineArgs") Optional<List<String>> commandLineArgs, | ||
| @JsonProperty("resources") Optional<Resources> resources, | ||
| @JsonProperty("s3UploaderAdditionalFiles") List<SingularityS3UploaderFile> s3UploaderAdditionalFiles, |
There was a problem hiding this comment.
remember this can be used from the client, so adding args will break the build for folks
There was a problem hiding this comment.
👍 good catch. Restored the constructor from the most recent release.
| if (configuration.getS3ConfigurationOptional().isPresent()) { | ||
| uploaderAdditionalFiles.addAll(configuration.getS3ConfigurationOptional().get().getS3UploaderAdditionalFiles()); | ||
| } | ||
| uploaderAdditionalFiles.addAll(task.getPendingTask().getS3UploaderAdditionalFiles()); |
There was a problem hiding this comment.
is it possible for two of these to overlap? The original use for this was to override where a specific file was being sent. Maybe we need to dedupe on the directory/glob in the additional file settings? If the uploader tries to upload it twice, one instance of that will fail when the first one done tries to delete it
There was a problem hiding this comment.
Hmm, that's a tricky one. Since we don't have a priori information about the actual set of files in the sandbox, it's nontrivial to determine if the glob patterns overlap. We could just compare the pattern strings directly, but that would of course not cover all potential overlaps.
There was a problem hiding this comment.
I would argue that some protection is better than none. Maybe we at least check for exact matches and call it good? That or another way to ensure it is to only take the override if it is set and ignore the executor settings
There was a problem hiding this comment.
After digging on this, it looks like a ConcurrentMap is used to ensure that two uploaders don't simultaneously try to handle the same file.
This means we won't have uploader errors due to conflicts, although it does leave a race condition in place. The way to solve this in the future would be to ensure that upload overrides for a file use a matching directory and glob strings as the corresponding global config, and then just compare those fields directly instead of trying to compute glob overlaps.
|
🚢 |
2 participants
No description provided.