Skip to content

Commit

Permalink
Fixed foreign file system detection
Browse files Browse the repository at this point in the history
This commit fixed the upload of files located on a file system
provider different from the one where the workflow work directory
is located.

Checking the file system instance is a bad strategy
because there could be multiple file system instances for the
same provider. For example the Google Storage file system
provider creates a new file system instance for each bucket.
Therefore the FilePorter would create a copy of file located
in a bucket X when the pipeline work directory is located in
the bucket Y.

Checking instead the file system provider scheme prevent to
copy files co-located in the same provider.
  • Loading branch information
pditommaso committed Dec 16, 2018
1 parent 66d4bc7 commit 1489cc7
Show file tree
Hide file tree
Showing 2 changed files with 6 additions and 2 deletions.
2 changes: 1 addition & 1 deletion src/main/groovy/nextflow/file/FilePorter.groovy
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ class FilePorter {
for( Map.Entry<String,Path> entry : filesMap ) {
def name = entry.getKey()
def path = entry.getValue()
if( path.fileSystem == stagingDir.fileSystem )
if( path.scheme == stagingDir.scheme )
continue
// copy the path with a thread pool
actions << { new NamePathPair(name, stageForeignFile(path)) } as Callable<NamePathPair>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1482,12 +1482,16 @@ class FilesEx {
static String toUriString( Path path ) {
if(!path)
return null
def scheme = path.getFileSystem().provider().scheme
final scheme = getScheme(path)
if( scheme == 'file' )
return path.toString()
if( scheme == 's3')
return "$scheme:/$path".toString()
else
return path.toUri().toString()
}

static String getScheme(Path path) {
path.getFileSystem().provider().getScheme()
}
}

0 comments on commit 1489cc7

Please sign in to comment.