Check to exist parent directory before uploading files

[sakama]

I got org.apache.commons.vfs2.FileSystemException: Could not create folder "sftp://user@example.com/parent" failure. So I fixed it.

This failure happens when

• Input source have multiple files
• Using LocalExecutor/ScatterExecutor
  • max_threads=8 / output tasks 4 = input tasks 2 * 2
• Remote parent directory doesn't exists

Environment is

• Embulk v0.8.6
• Current master/HEAD of embulk-output-sftp

My Changes

• Check before uploading files
• Create remote directory if not exists

Stacktrace

[@sakama embulk]$ embulk run -I ~/data/embulk-output-sftp/lib gcs_sftp.yml
2016-03-09 14:57:35.771 +0900: Embulk v0.8.6
2016-03-09 14:57:37.279 +0900 [INFO] (0001:transaction): Loaded plugin embulk-input-gcs (0.1.13)
2016-03-09 14:57:37.307 +0900 [INFO] (0001:transaction): Loaded plugin embulk/output/sftp from a load path
2016-03-09 14:57:39.688 +0900 [INFO] (0001:transaction): Using local thread executor with max_threads=8 / output tasks 4 = input tasks 2 * 2
2016-03-09 14:57:39.734 +0900 [INFO] (0001:transaction): {done:  0 / 2, running: 0}
2016-03-09 14:57:39.842 +0900 [INFO] (0019:task-0000): set identity: org.embulk.spi.unit.LocalFile@c6a5d79
2016-03-09 14:57:39.842 +0900 [INFO] (0020:task-0001): set identity: org.embulk.spi.unit.LocalFile@37ce8e5
2016-03-09 14:57:42.118 +0900 [ERROR] (0020:task-0001): Could not create folder "sftp://user1@example.com/home/user1/parent".
2016-03-09 14:57:42.418 +0900 [INFO] (0019:task-0000): new sftp file: sftp://user1@example.com/home/user1/parent/file.00_20151020.csv
2016-03-09 14:57:42.427 +0900 [INFO] (0019:task-0000): set identity: org.embulk.spi.unit.LocalFile@17f6ab
2016-03-09 14:57:44.497 +0900 [INFO] (0019:task-0000): new sftp file: sftp://user1@example.com/home/user1/parent/file.10_20151020.csv
2016-03-09 14:57:45.973 +0900 [INFO] (0001:transaction): {done:  2 / 2, running: 0}
2016-03-09 14:57:45.973 +0900 [INFO] (0001:transaction): {done:  2 / 2, running: 0}
org.embulk.exec.PartialExecutionException: java.lang.RuntimeException: org.apache.commons.vfs2.FileSystemException: Could not create folder "sftp://user1@example.com/home/user1/parent".
    at org.embulk.exec.BulkLoader$LoaderState.buildPartialExecuteException(org/embulk/exec/BulkLoader.java:363)
    at org.embulk.exec.BulkLoader.doRun(org/embulk/exec/BulkLoader.java:572)
    at org.embulk.exec.BulkLoader.access$000(org/embulk/exec/BulkLoader.java:33)
    at org.embulk.exec.BulkLoader$1.run(org/embulk/exec/BulkLoader.java:374)
    at org.embulk.exec.BulkLoader$1.run(org/embulk/exec/BulkLoader.java:370)
    at org.embulk.spi.Exec.doWith(org/embulk/spi/Exec.java:25)
    at org.embulk.exec.BulkLoader.run(org/embulk/exec/BulkLoader.java:370)
    at org.embulk.EmbulkEmbed.run(org/embulk/EmbulkEmbed.java:180)
    at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:497)
    at RUBY.run(/Users/satoshi/.embulk/bin/embulk!/embulk/runner.rb:84)
    at RUBY.run(/Users/satoshi/.embulk/bin/embulk!/embulk/command/embulk_run.rb:306)
    at RUBY.<top>(/Users/satoshi/.embulk/bin/embulk!/embulk/command/embulk_main.rb:2)
    at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:937)
    at RUBY.(root)(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1)
    at Users.satoshi.$_dot_embulk.bin.embulk.embulk.command.embulk_bundle.<top>(file:/Users/satoshi/.embulk/bin/embulk!/embulk/command/embulk_bundle.rb:51)
    at java.lang.invoke.MethodHandle.invokeWithArguments(java/lang/invoke/MethodHandle.java:625)
    at org.embulk.cli.Main.main(org/embulk/cli/Main.java:23)
Caused by: java.lang.RuntimeException: org.apache.commons.vfs2.FileSystemException: Could not create folder "sftp://user1@example.com/home/user1/parent".
    at com.google.common.base.Throwables.propagate(Throwables.java:160)
    at org.embulk.output.sftp.SftpFileOutput.nextFile(SftpFileOutput.java:163)
    at org.embulk.spi.util.FileOutputOutputStream.nextFile(FileOutputOutputStream.java:34)
    at org.embulk.spi.util.LineEncoder.nextFile(LineEncoder.java:93)
    at org.embulk.standards.CsvFormatterPlugin.open(CsvFormatterPlugin.java:116)
    at org.embulk.spi.FileOutputRunner.open(FileOutputRunner.java:140)
    at org.embulk.exec.LocalExecutorPlugin$ScatterTransactionalPageOutput.openOutputs(LocalExecutorPlugin.java:448)
    at org.embulk.exec.LocalExecutorPlugin$ScatterExecutor.runInputTask(LocalExecutorPlugin.java:281)
    at org.embulk.exec.LocalExecutorPlugin$ScatterExecutor.access$000(LocalExecutorPlugin.java:212)
    at org.embulk.exec.LocalExecutorPlugin$ScatterExecutor$1.call(LocalExecutorPlugin.java:257)
    at org.embulk.exec.LocalExecutorPlugin$ScatterExecutor$1.call(LocalExecutorPlugin.java:253)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
    Suppressed: java.lang.NullPointerException
        at org.embulk.output.sftp.SftpFileOutput.closeCurrentFile(SftpFileOutput.java:217)
        at org.embulk.output.sftp.SftpFileOutput.close(SftpFileOutput.java:195)
        at org.embulk.spi.CloseResource.close(CloseResource.java:34)
        at org.embulk.spi.FileOutputRunner.open(FileOutputRunner.java:147)
        ... 9 more
Caused by: org.apache.commons.vfs2.FileSystemException: Could not create folder "sftp://user1@example.com/home/user1/parent".
    at org.apache.commons.vfs2.provider.AbstractFileObject.createFolder(AbstractFileObject.java:437)
    at org.apache.commons.vfs2.provider.AbstractFileObject.getOutputStream(AbstractFileObject.java:1411)
    at org.apache.commons.vfs2.provider.DefaultFileContent.getOutputStream(DefaultFileContent.java:479)
    at org.apache.commons.vfs2.provider.DefaultFileContent.getOutputStream(DefaultFileContent.java:457)
    at org.embulk.output.sftp.SftpFileOutput.nextFile(SftpFileOutput.java:158)
    ... 13 more
Caused by: 4: Failure
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2846)
    at com.jcraft.jsch.ChannelSftp.mkdir(ChannelSftp.java:2155)
    at org.apache.commons.vfs2.provider.sftp.SftpFileObject.doCreateFolder(SftpFileObject.java:210)
    at org.apache.commons.vfs2.provider.AbstractFileObject.createFolder(AbstractFileObject.java:426)
    ... 17 more

[civitaspo]

thx!