New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Excon::Errors::SocketError: Broken pipe (Errno::EPIPE) - while uploading more than 1MB file #1731

Closed
hide2me opened this Issue Apr 4, 2013 · 9 comments

Comments

Projects
None yet
3 participants
@hide2me

hide2me commented Apr 4, 2013

while uploading more than 1MB file. Even same error after gave region value.

plz suggest upload large file to S3

gem details
excon-0.20.1
formatador-0.2.4
net-ssh-2.6.6
net-scp-1.1.0
nokogiri-1.5.9
ruby-hmac-0.4.0
fog-1.10.0

Here code

connection = Fog::Storage.new({:provider => 'AWS', :aws_access_key_id => 'my key ', :aws_secret_access_key =>'my key'})
directory = connection.directories.get('git-44')
a=directory.files.create(:key =>"288.pdf",:body =>File.open('/Users/hide/Downloads/Folx/rails_testing.pdf'),:public => true)

Excon::Errors::SocketError: Broken pipe (Errno::EPIPE)
from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/openssl/buffering.rb:375:in syswrite_nonblock' from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/openssl/buffering.rb:375:inwrite_nonblock'
from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/excon-0.20.1/lib/excon/socket.rb:156:in write' from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/excon-0.20.1/lib/excon/ssl_socket.rb:82:inwrite'
from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/excon-0.20.1/lib/excon/connection.rb:167:in request_call' from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/excon-0.20.1/lib/excon/middlewares/mock.rb:79:inrequest_call'
from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/excon-0.20.1/lib/excon/middlewares/instrumentor.rb:22:in request_call' from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/excon-0.20.1/lib/excon/middlewares/base.rb:15:inrequest_call'
from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/excon-0.20.1/lib/excon/middlewares/base.rb:15:in request_call' from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/excon-0.20.1/lib/excon/connection.rb:223:inrequest'
from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/excon-0.20.1/lib/excon/middlewares/idempotent.rb:12:in error_call' from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/excon-0.20.1/lib/excon/middlewares/base.rb:10:inerror_call'
from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/excon-0.20.1/lib/excon/connection.rb:239:in rescue in request' from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/excon-0.20.1/lib/excon/connection.rb:200:inrequest'
from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/excon-0.20.1/lib/excon/middlewares/idempotent.rb:12:in error_call' from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/excon-0.20.1/lib/excon/middlewares/base.rb:10:inerror_call'
from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/excon-0.20.1/lib/excon/connection.rb:239:in rescue in request' from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/excon-0.20.1/lib/excon/connection.rb:200:inrequest'
from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/excon-0.20.1/lib/excon/middlewares/idempotent.rb:12:in error_call' from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/excon-0.20.1/lib/excon/middlewares/base.rb:10:inerror_call'
from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/excon-0.20.1/lib/excon/connection.rb:239:in rescue in request' from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/excon-0.20.1/lib/excon/connection.rb:200:inrequest'
from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/fog-1.10.0/lib/fog/core/connection.rb:21:in request' from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/fog-1.10.0/lib/fog/aws/storage.rb:401:inrequest'
from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/fog-1.10.0/lib/fog/aws/requests/storage/put_object.rb:40:in put_object' from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/fog-1.10.0/lib/fog/aws/models/storage/file.rb:212:insave'
from /usr/local/rvm/gems/ruby-1.9.3-p194/gems/fog-1.10.0/lib/fog/core/collection.rb:52:in `create'

@geemus

This comment has been minimized.

Show comment
Hide comment
@geemus

geemus Apr 4, 2013

Member

@hide-me - hmm. Is it always failing or just sometimes? You mentioned setting the region, does that make it fail less often? Is it this particular file for some reason or just files greater than 1MB in general?

Member

geemus commented Apr 4, 2013

@hide-me - hmm. Is it always failing or just sometimes? You mentioned setting the region, does that make it fail less often? Is it this particular file for some reason or just files greater than 1MB in general?

@ttdonovan

This comment has been minimized.

Show comment
Hide comment
@ttdonovan

ttdonovan Apr 5, 2013

@geemus @hide-me I seem to be experiencing the same issue using Fog 1.10. I'm attempting to run a Sidekiq worker that fetch records from a database, generate a XML file and place that file in AWS. My first attempt was with files around 20MB then I reduced the number for records fetched form the database so the file is much smaller now.

@geemus what other information could I provide to help in troubleshooting the issue?

      storage = Fog::Storage.new({
        provider: 'AWS',
        aws_access_key_id: ENV['AWS_ACCESS_KEY_ID'],
        aws_secret_access_key: ENV['AWS_SECRET_ACCESS_KEY']
      })

      # catalog_manager.builder returns an instance of Nokogiri::XML::Builder
      file = storage.directories.get("feeds").files.create({
        key: "catalog/#{file_name}",
        body: catalog_manager.builder.to_xml,
        multipart_chunk_size: 5242880
      })

ttdonovan commented Apr 5, 2013

@geemus @hide-me I seem to be experiencing the same issue using Fog 1.10. I'm attempting to run a Sidekiq worker that fetch records from a database, generate a XML file and place that file in AWS. My first attempt was with files around 20MB then I reduced the number for records fetched form the database so the file is much smaller now.

@geemus what other information could I provide to help in troubleshooting the issue?

      storage = Fog::Storage.new({
        provider: 'AWS',
        aws_access_key_id: ENV['AWS_ACCESS_KEY_ID'],
        aws_secret_access_key: ENV['AWS_SECRET_ACCESS_KEY']
      })

      # catalog_manager.builder returns an instance of Nokogiri::XML::Builder
      file = storage.directories.get("feeds").files.create({
        key: "catalog/#{file_name}",
        body: catalog_manager.builder.to_xml,
        multipart_chunk_size: 5242880
      })
@hide2me

This comment has been minimized.

Show comment
Hide comment
@hide2me

hide2me Apr 5, 2013

@geemus - Always its failing , try with pdf, image & video files.

Even I have added timeout

connection = Fog::Storage.new({:provider => 'AWS', :aws_access_key_id => '', :aws_secret_access_key =>'',:persistent=>false,:connection_options=>{:write_timeout=>3600,:read_timeout=>3600,:connect_timeout=>3600}})

I have to move pdf, image & video to s3 , for now I'm using s3-cmd tool for moving files.

hide2me commented Apr 5, 2013

@geemus - Always its failing , try with pdf, image & video files.

Even I have added timeout

connection = Fog::Storage.new({:provider => 'AWS', :aws_access_key_id => '', :aws_secret_access_key =>'',:persistent=>false,:connection_options=>{:write_timeout=>3600,:read_timeout=>3600,:connect_timeout=>3600}})

I have to move pdf, image & video to s3 , for now I'm using s3-cmd tool for moving files.

@geemus

This comment has been minimized.

Show comment
Hide comment
@geemus

geemus Apr 5, 2013

Member

@hide-me if you could run it with ENV['EXCON_DEBUG']=true I suspect that could be helpful, just make sure to redact anything private (as you have above). That should give us a more detailed view into what is going on here.

Member

geemus commented Apr 5, 2013

@hide-me if you could run it with ENV['EXCON_DEBUG']=true I suspect that could be helpful, just make sure to redact anything private (as you have above). That should give us a more detailed view into what is going on here.

@ttdonovan

This comment has been minimized.

Show comment
Hide comment
@ttdonovan

ttdonovan Apr 6, 2013

@hide-me by setting the correct 'region' to 'us-west-1' in the Fog::Storage.new params appear to fix the issues I was having.

ttdonovan commented Apr 6, 2013

@hide-me by setting the correct 'region' to 'us-west-1' in the Fog::Storage.new params appear to fix the issues I was having.

@geemus

This comment has been minimized.

Show comment
Hide comment
@geemus

geemus Apr 8, 2013

Member

@ttdonovan - thanks for the update, that lines up with what I've heard from others.

Member

geemus commented Apr 8, 2013

@ttdonovan - thanks for the update, that lines up with what I've heard from others.

@hide2me

This comment has been minimized.

Show comment
Hide comment
@hide2me

hide2me Apr 10, 2013

Hi @geemus and @ttdonovan

Thanks for update, but region is not working other than 'us-east-1'

Excon::Errors::MovedPermanently: Expected(200) <=> Actual(301 Moved Permanently)

1.9.3p392 :001 > require 'fog'
=> true
1.9.3p392 :002 > connection = Fog::Storage.new({:provider => 'AWS', :aws_access_key_id => '<>', :aws_secret_access_key =>'<>,:region=>'us-west-1'})

=> #<Fog::Storage::AWS::Real:2158782920 @use_iam_profile=nil @aws_access_key_id="<>" @aws_session_token=nil @aws_credentials_expire_at=nil @connection_options={} @endpoint=nil @region="us-west-1" @host="s3-us-west-1.amazonaws.com" @path="/" @Persistent=false @port=443 @scheme="https" @connection=#<Fog::Connection:0x00000101305848 @excon=#<Excon::Connection:1041f6760 @DaTa={:chunk_size=>1048576, :connect_timeout=>60, :headers=>{"User-Agent"=>"fog/1.10.0"}, :idempotent=>false, :instrumentor_name=>"excon", :middlewares=>[Excon::Middleware::Expects, Excon::Middleware::Idempotent, Excon::Middleware::Instrumentor, Excon::Middleware::Mock], :mock=>false, :nonblock=>true, :read_timeout=>60, :retry_limit=>4, :ssl_ca_file=>"/Users/hide/.rvm/gems/ruby-1.9.3-p392@rails-3.1.3-microsite/gems/excon-0.20.1/data/cacert.pem", :ssl_verify_peer=>true, :uri_parser=>URI, :write_timeout=>60, :host=>"s3-us-west-1.amazonaws.com", :path=>"/", :port=>"443", :query=>nil, :scheme=>"https", :user=>nil, :password=>nil, :family=>0} @socket_key="s3-us-west-1.amazonaws.com:443">, @Persistent=false>>

1.9.3p392 :003 > directory = connection.directories

=> <Fog::Storage::AWS::Directories
[
<Fog::Storage::AWS::Directory
key="git-44",
creation_date=2013-04-03 12:33:48 UTC
>
] >

1.9.3p392 :004 > directory = connection.directories.get('git-44')

Excon::Errors::MovedPermanently: Expected(200) <=> Actual(301 Moved Permanently)

hide2me commented Apr 10, 2013

Hi @geemus and @ttdonovan

Thanks for update, but region is not working other than 'us-east-1'

Excon::Errors::MovedPermanently: Expected(200) <=> Actual(301 Moved Permanently)

1.9.3p392 :001 > require 'fog'
=> true
1.9.3p392 :002 > connection = Fog::Storage.new({:provider => 'AWS', :aws_access_key_id => '<>', :aws_secret_access_key =>'<>,:region=>'us-west-1'})

=> #<Fog::Storage::AWS::Real:2158782920 @use_iam_profile=nil @aws_access_key_id="<>" @aws_session_token=nil @aws_credentials_expire_at=nil @connection_options={} @endpoint=nil @region="us-west-1" @host="s3-us-west-1.amazonaws.com" @path="/" @Persistent=false @port=443 @scheme="https" @connection=#<Fog::Connection:0x00000101305848 @excon=#<Excon::Connection:1041f6760 @DaTa={:chunk_size=>1048576, :connect_timeout=>60, :headers=>{"User-Agent"=>"fog/1.10.0"}, :idempotent=>false, :instrumentor_name=>"excon", :middlewares=>[Excon::Middleware::Expects, Excon::Middleware::Idempotent, Excon::Middleware::Instrumentor, Excon::Middleware::Mock], :mock=>false, :nonblock=>true, :read_timeout=>60, :retry_limit=>4, :ssl_ca_file=>"/Users/hide/.rvm/gems/ruby-1.9.3-p392@rails-3.1.3-microsite/gems/excon-0.20.1/data/cacert.pem", :ssl_verify_peer=>true, :uri_parser=>URI, :write_timeout=>60, :host=>"s3-us-west-1.amazonaws.com", :path=>"/", :port=>"443", :query=>nil, :scheme=>"https", :user=>nil, :password=>nil, :family=>0} @socket_key="s3-us-west-1.amazonaws.com:443">, @Persistent=false>>

1.9.3p392 :003 > directory = connection.directories

=> <Fog::Storage::AWS::Directories
[
<Fog::Storage::AWS::Directory
key="git-44",
creation_date=2013-04-03 12:33:48 UTC
>
] >

1.9.3p392 :004 > directory = connection.directories.get('git-44')

Excon::Errors::MovedPermanently: Expected(200) <=> Actual(301 Moved Permanently)

@hide2me

This comment has been minimized.

Show comment
Hide comment
@hide2me

hide2me Apr 10, 2013

Thanks @geemus and @ttdonovan

Based on AWS account region various, even its specific to bucket. I have used "us-west-2" region, its working fine.

connection = Fog::Storage.new({:provider => 'AWS', :aws_access_key_id => '', :aws_secret_access_key =>' ',:region=>'us-west-2'})

Screen Shot 2013-04-10 at 2 21 51 PM

hide2me commented Apr 10, 2013

Thanks @geemus and @ttdonovan

Based on AWS account region various, even its specific to bucket. I have used "us-west-2" region, its working fine.

connection = Fog::Storage.new({:provider => 'AWS', :aws_access_key_id => '', :aws_secret_access_key =>' ',:region=>'us-west-2'})

Screen Shot 2013-04-10 at 2 21 51 PM

@hide2me hide2me closed this Apr 10, 2013

@geemus

This comment has been minimized.

Show comment
Hide comment
@geemus

geemus Apr 10, 2013

Member

@hide - great, thanks for the update!

On Wed, Apr 10, 2013 at 3:57 AM, hide notifications@github.com wrote:

Thanks @geemus https://github.com/geemus and @ttdonovanhttps://github.com/ttdonovan

Based on AWS account region various. I have used "us-west-2" region, its
working fine.

connection = Fog::Storage.new({:provider => 'AWS', :aws_access_key_id =>
'', :aws_secret_access_key =>' ',:region=>'us-west-2'})

[image: Screen Shot 2013-04-10 at 2 21 51 PM]https://f.cloud.github.com/assets/956372/361217/7eec5d2c-a1bc-11e2-9d7a-dd49492fe813.png


Reply to this email directly or view it on GitHubhttps://github.com/fog/fog/issues/1731#issuecomment-16162625
.

Member

geemus commented Apr 10, 2013

@hide - great, thanks for the update!

On Wed, Apr 10, 2013 at 3:57 AM, hide notifications@github.com wrote:

Thanks @geemus https://github.com/geemus and @ttdonovanhttps://github.com/ttdonovan

Based on AWS account region various. I have used "us-west-2" region, its
working fine.

connection = Fog::Storage.new({:provider => 'AWS', :aws_access_key_id =>
'', :aws_secret_access_key =>' ',:region=>'us-west-2'})

[image: Screen Shot 2013-04-10 at 2 21 51 PM]https://f.cloud.github.com/assets/956372/361217/7eec5d2c-a1bc-11e2-9d7a-dd49492fe813.png


Reply to this email directly or view it on GitHubhttps://github.com/fog/fog/issues/1731#issuecomment-16162625
.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment