Socket error when transferring to S3 - excon #351

Closed
jbrook opened this Issue Aug 21, 2012 · 17 comments

10 participants

@jbrook

When backup is trying to transfer one of my files to an S3 bucket, I get the following error:

An Error occured which has caused this Backup to abort before completion.
[2012/08/21 09:51:26][error]   Reason: Excon::Errors::SocketError
[2012/08/21 09:51:26][error]   Broken pipe
[2012/08/21 09:51:26][error] 
[2012/08/21 09:51:26][error] Backtrace:
[2012/08/21 09:51:26][error]   /usr/lib/ruby/1.8/openssl/buffering.rb:179:in `syswrite'
[2012/08/21 09:51:26][error]   /usr/lib/ruby/1.8/openssl/buffering.rb:179:in `do_write'
[2012/08/21 09:51:26][error]   /usr/lib/ruby/1.8/openssl/buffering.rb:193:in `write'
[2012/08/21 09:51:26][error]   /usr/lib/ruby/gems/1.8/gems/excon-0.14.3/lib/excon/connection.rb:249:in `request_kernel'

Previously to this one file does get successfully transferred:

[2012/08/21 09:49:23][message] Syncer::Cloud::S3 started the syncing process:
[2012/08/21 09:49:23][message]   Concurrency: false Level: 2
[2012/08/21 09:49:25][message]   Generating checksums for '/opt/graphite/storage/graphite.db'
[2012/08/21 09:49:30][message]   [transferring] 'files/graphite.db/opt/graphite/storage/graphite.db'
[2012/08/21 09:49:52][message]   [removing] 'files/graphite.db/opt/graphite/storage/graphite.db'
[2012/08/21 09:49:54][message]   Generating checksums for '/opt/graphite/storage/whisper'

The failed file is 6.5Mb in size.

It is running on EC2 Ubuntu with Ruby 1.8.7, backup 3.0.25, fog 1.4.0 and excon 0.14.3.

It looks like it was an issue that was fixed in excon at some point - excon/excon#121

Any help would be much appreciated.

Thanks
James

@jbrook

I don't think that's it. The first smaller file is transferred without a problem.

@fullofcaffeine

I'm also getting this error. My first backup ran fine (two days ago), but now I can't do it anymore.

Here's the backtrace:

[2012/10/21 00:03:13][message] Storage::S3 started transferring '2012.10.21.00.00.04.my_backup.tar' to bucket 'mine'
[2012/10/21 00:47:14][error] ModelError: Backup for fullofcaffeine_backup (my_backup) Failed!
[2012/10/21 00:47:14][error] An Error occured which has caused this Backup to abort before completion.
[2012/10/21 00:47:14][error] Reason: Excon::Errors::SocketError
[2012/10/21 00:47:14][error] Broken pipe
[2012/10/21 00:47:14][error]
[2012/10/21 00:47:14][error] Backtrace:
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/openssl/buffering.rb:375:in syswrite_nonblock'
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/openssl/buffering.rb:375:in
write_nonblock'
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/excon-0.14.3/lib/excon/socket.rb:118:in write'
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/excon-0.14.3/lib/excon/connection.rb:249:in
request_kernel'
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/excon-0.14.3/lib/excon/connection.rb:101:in request'
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/fog-1.4.0/lib/fog/core/connection.rb:20:in
request'
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/fog-1.4.0/lib/fog/aws/storage.rb:392:in request'
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/fog-1.4.0/lib/fog/aws/requests/storage/put_object.rb:43:in
put_object'
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup-3.0.25/lib/backup/storage/s3.rb:68:in block (2 levels) in transfer!'
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup-3.0.25/lib/backup/storage/s3.rb:67:in
open'
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup-3.0.25/lib/backup/storage/s3.rb:67:in block in transfer!'
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup-3.0.25/lib/backup/storage/base.rb:79:in
block in files_to_transfer_for'
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup-3.0.25/lib/backup/storage/base.rb:78:in each'
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup-3.0.25/lib/backup/storage/base.rb:78:in
files_to_transfer_for'
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup-3.0.25/lib/backup/storage/s3.rb:63:in transfer!'
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup-3.0.25/lib/backup/storage/base.rb:33:in
perform!'
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup-3.0.25/lib/backup/model.rb:254:in each'
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup-3.0.25/lib/backup/model.rb:254:in
block in perform!'
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup-3.0.25/lib/backup/model.rb:252:in each'
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup-3.0.25/lib/backup/model.rb:252:in
perform!'
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup-3.0.25/lib/backup/cli/utility.rb:74:in block in perform'
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup-3.0.25/lib/backup/cli/utility.rb:65:in
each'
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup-3.0.25/lib/backup/cli/utility.rb:65:in perform'
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/thor-0.15.4/lib/thor/task.rb:27:in
run'
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/thor-0.15.4/lib/thor/invocation.rb:120:in invoke_task'
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/thor-0.15.4/lib/thor.rb:275:in
dispatch'
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/thor-0.15.4/lib/thor/base.rb:425:in start'
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup-3.0.25/bin/backup:11:in
'
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/bin/backup:19:in load'
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/bin/backup:19:in

'
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/bin/ruby_noexec_wrapper:14:in eval'
[2012/10/21 00:47:14][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/bin/ruby_noexec_wrapper:14:in
'
[2012/10/21 00:47:14][warning] CleanerError: Cleanup Warning
[2012/10/21 00:47:14][warning] The temporary backup folder '/Users/fullofcaffeine/Backup/.tmp'
[2012/10/21 00:47:14][warning] appears to contain the backup files which were to be stored:
[2012/10/21 00:47:14][warning] /Users/fullofcaffeine/Backup/.tmp/2012.10.21.00.00.04.my_backup.tar

Any ideas?

Cheers,

Marcelo.

@fullofcaffeine

I'm now sure it's not related to the region. Yesterday's backup completed successfully, but today's backup failed with a slightly different error:

[2012/10/23 11:04:36][error] ModelError: Backup for fullofcaffeine_backup (my_backup) Failed!
[2012/10/23 11:04:36][error] An Error occured which has caused this Backup to abort before completion.
[2012/10/23 11:04:36][error] Reason: Excon::Errors::SocketError
[2012/10/23 11:04:36][error] getaddrinfo: nodename nor servname provided, or not known

That's pretty annoying and it affects the reliability of the backup system, so any ideas on what might be wrong are welcome :)

@mmzoo

I have the same issue - before it just happend randomly but right now the error occurs each time. I suspect that it's dependent on the excon and fog gem versions.

@mmzoo

I upgraded from fog 1.1.0 to 1.1.2 and that did the trick.

@fullofcaffeine

I have fog 1.4.0 (and Backup 3.0.25) running on Ruby 1.9.3-p194, and I'm still having issues. Last night's backup failed again:

[2012/11/14 04:52:28][error] ModelError: Backup for fullofcaffeine_backup (my_backup) Failed!
[2012/11/14 04:52:28][error] An Error occured which has caused this Backup to abort before completion.
[2012/11/14 04:52:28][error] Reason: Excon::Errors::SocketError
[2012/11/14 04:52:28][error] Broken pipe
[2012/11/14 04:52:28][error]
[2012/11/14 04:52:28][error] Backtrace:
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/openssl/buffering.rb:375:in >syswrite_nonblock'
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/openssl/buffering.rb:375:in >
write_nonblock'
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/excon->0.14.3/lib/excon/socket.rb:118:in write'
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/excon->0.14.3/lib/excon/connection.rb:249:in
request_kernel'
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/excon->0.14.3/lib/excon/connection.rb:101:in request'
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/fog->1.4.0/lib/fog/core/connection.rb:20:in
request'
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/fog-1.4.0/lib/fog/aws/storage.rb:392:in >request'
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/fog->1.4.0/lib/fog/aws/requests/storage/put_object.rb:43:in
put_object'
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup->3.0.25/lib/backup/storage/s3.rb:68:in block (2 levels) in transfer!'
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup->3.0.25/lib/backup/storage/s3.rb:67:in
open'
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup->3.0.25/lib/backup/storage/s3.rb:67:in block in transfer!'
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup->3.0.25/lib/backup/storage/base.rb:79:in
block in files_to_transfer_for'
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup->3.0.25/lib/backup/storage/base.rb:78:in each'
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup->3.0.25/lib/backup/storage/base.rb:78:in
files_to_transfer_for'
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup->3.0.25/lib/backup/storage/s3.rb:63:in transfer!'
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup->3.0.25/lib/backup/storage/base.rb:33:in
perform!'
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup->3.0.25/lib/backup/model.rb:254:in each'
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup->3.0.25/lib/backup/model.rb:254:in
block in perform!'
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup->3.0.25/lib/backup/model.rb:252:in each'
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup->3.0.25/lib/backup/model.rb:252:in
perform!'
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup->3.0.25/lib/backup/cli/utility.rb:74:in block in perform'
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup->3.0.25/lib/backup/cli/utility.rb:65:in
each'
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup->3.0.25/lib/backup/cli/utility.rb:65:in perform'
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/thor-0.15.4/lib/thor/task.rb:27:in
run'
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/thor->0.15.4/lib/thor/invocation.rb:120:in invoke_task'
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/thor-0.15.4/lib/thor.rb:275:in
dispatch'
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/thor-0.15.4/lib/thor/base.rb:425:in >start'
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/gems/backup-3.0.25/bin/backup:11:in
(required)>'
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/bin/backup:19:in load'
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/bin/backup:19:in

'
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/bin/ruby_noexec_wrapper:14:in eval'
[2012/11/14 04:52:28][error] /Users/fullofcaffeine/.rvm/gems/ruby-1.9.3-p194@backup/bin/ruby_noexec_wrapper:14:in
'

The backup file has around 4GB. The S3 region is us-east-1.

Any ideas? This is getting really annoying, to the point of almost making give up on Backup for S3 backups, which is a pity since it used to work so well, and I love how it works.

@fullofcaffeine

Does anyone know if S3 have any kind of file size limit? If so, does Backup offer any file-splitting facilities? I have upgraded to Backup 3.0.27 just now, I will try tonight and see how did the backup go in the morning and post the results here.

Cheers,

Marcelo.

@maxim

Started getting this problem now too, backing up every 2 hours, last 3 backups failed with the original poster's error (Broken pipe).

@maxim

Disregard me, I forgot to update aws credentials.

@johntdyer

Getting this w/ Fog 1.8, Ruby 1.9.3 and Back 3.0.27... Anyone resolved this yet?

@santaux

I have the same problem with Ruby 1.9.3 and fog 1.4.0

@Antiarchitect

backup 3.1.3 - the same.

@bjensen

I also get this issue. backup 3.1.3 fog 1.9.0 and excon 0.17.0

@joshcutler

I get this as well, backup 3.2.0, Fog 1.9, excon 0.17.0

@ghost

These SocketErrors are most likely due to some disruption in your connection.
I've updated the S3 Storage in v3.4.0 to use multipart uploading and retry requests when errors occur. This should take care of those problems.
The S3 Syncer will take a bit more work to incorporate these changes.

@ghost

As of 3.7.0, the S3 Syncer supports retrying failed operations which should eliminate these issues.
This is also now true for the CloudFiles Syncer and Storage as well.

@ghost ghost closed this Jul 25, 2013
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment