Skip to content

Conversation

sonots
Copy link
Member

@sonots sonots commented Mar 16, 2016

This possibly fixes several connection issues such as below:

(1) Could not write to

2016-03-14T09:10:37 [WARN]         org.apache.commons.vfs2.FileSystemException: Could not write to \"sftp://xxxx/xxxxx\".\n\tat
2016-03-14T09:10:37 [WARN]         com.google.common.base.Throwables.propagate(Throwables.java:160)\n\tat 
2016-03-14T09:10:37 [WARN]         org.embulk.output.sftp.SftpFileOutput.nextFile(SftpFileOutput.java:122)\n\tat
2016-03-14T09:10:37 [WARN]         org.embulk.spi.util.FileOutputOutputStream.nextFile(FileOutputOutputStream.java:34)\n\tat

I implemented retry for this.

(2) Possible write failure

I did not meet with this on the production, but it possibly can happen. Fixed to retry writing.

NOTE: Retry writing may produce data duplication, users should always check number of rows coming from input, and number of rows written into output. I advise that users do such check always even if they do not turn on retry feature.

(3) Could not close the output stream for file

2016-03-14T12:10:43 [WARN] Caused by: java.lang.RuntimeException: org.apache.commons.vfs2.FileSystemException: Could not close the output stream for file "xxxx.zip".
2016-03-14T12:10:43 [WARN] \tat com.google.common.base.Throwables.propagate(Throwables.java:160)
2016-03-14T12:10:43 [WARN] \tat org.embulk.output.sftp.SftpFileOutput.closeCurrentFile(SftpFileOutput.java:181)
2016-03-14T12:10:43 [WARN] \tat org.embulk.output.sftp.SftpFileOutput.close(SftpFileOutput.java:152)

InputStream may close OutputStream. We can simply ignore errors on closing if output stream or file are already closed.

Extra

I also added Vagrantfile so that example can easily be ran.

civitaspo added a commit that referenced this pull request Mar 16, 2016
@civitaspo civitaspo merged commit e5b3296 into embulk:master Mar 16, 2016
@civitaspo
Copy link
Member

good! thx so much!!!!

@sonots sonots deleted the retry branch March 16, 2016 08:21
@civitaspo civitaspo mentioned this pull request Mar 16, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants