getting -sometimes- Twitter::Error::ClientError: end of file reached

[BobWalsh]

in a rails c session in an app with the Twitter gem configured with my keys

1.9.3p392 :111 > thetweets = Twitter.search('git', :count => 10, :lang => "en", :result_type => "recent") 
 => #<Twitter::SearchResults:0x007f8e8ad4f178 @attrs={:statuses=>[{:metadata=>{:result_type=>"recent", :iso_language_code=>"en"}, :created_at=>"Wed Mar 20 02:34:16 +0000 2013", :id=>314203204525125633, :id_str=>"314203204525125633", :text=>"@LGwenn :D Let's git this thing done!", :source=>"<a href=\"http://www.metrotwit.com/\" rel=\"nofollow\">MetroTwit</a>", :truncated=>false,

but a nearly identical request gets very different results:

1.9.3p392 :168 > thetweets = Twitter.search('1git', :count => 10, :lang => "en", :result_type => "recent") 
**Twitter::Error::ClientError: end of file reached**
    from /Users/bob/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/openssl/buffering.rb:174:in `sysread_nonblock'
    from /Users/bob/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/openssl/buffering.rb:174:in `read_nonblock'
    from /Users/bob/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/protocol.rb:141:in `rbuf_fill'
    from /Users/bob/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/protocol.rb:122:in `readuntil'
    from /Users/bob/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/protocol.rb:132:in `readline'
    from /Users/bob/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/http.rb:2770:in `read_chunked'
    from /Users/bob/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/http.rb:2750:in `read_body_0'
    from /Users/bob/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/http.rb:2710:in `read_body'
    from /Users/bob/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/http.rb:1029:in `block in get'
    from /Users/bob/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/http.rb:1322:in `block (2 levels) in transport_request'
    from /Users/bob/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/http.rb:2671:in `reading_body'
    from /Users/bob/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/http.rb:1321:in `block in transport_request'
    from /Users/bob/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/http.rb:1316:in `catch'
    from /Users/bob/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/http.rb:1316:in `transport_request'
    from /Users/bob/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/http.rb:1293:in `request'
    from /Users/bob/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/http.rb:1286:in `block in request'
... 8 levels...
    from /Users/bob/.rvm/gems/ruby-1.9.3-p392@Rails3.2.12/gems/faraday-0.8.6/lib/faraday/request/url_encoded.rb:14:in `call'
    from /Users/bob/.rvm/gems/ruby-1.9.3-p392@Rails3.2.12/gems/faraday-0.8.6/lib/faraday/request/multipart.rb:13:in `call'
    from /Users/bob/.rvm/gems/ruby-1.9.3-p392@Rails3.2.12/gems/twitter-4.6.1/lib/twitter/request/multipart_with_file.rb:14:in `call'
    from /Users/bob/.rvm/gems/ruby-1.9.3-p392@Rails3.2.12/gems/faraday-0.8.6/lib/faraday/connection.rb:247:in `run_request'
    from /Users/bob/.rvm/gems/ruby-1.9.3-p392@Rails3.2.12/gems/faraday-0.8.6/lib/faraday/connection.rb:100:in `get'
    from /Users/bob/.rvm/gems/ruby-1.9.3-p392@Rails3.2.12/gems/twitter-4.6.1/lib/twitter/client.rb:81:in `request'
    from /Users/bob/.rvm/gems/ruby-1.9.3-p392@Rails3.2.12/gems/twitter-4.6.1/lib/twitter/client.rb:64:in `get'
    from /Users/bob/.rvm/gems/ruby-1.9.3-p392@Rails3.2.12/gems/twitter-4.6.1/lib/twitter/api/utils.rb:82:in `object_from_response'
    from /Users/bob/.rvm/gems/ruby-1.9.3-p392@Rails3.2.12/gems/twitter-4.6.1/lib/twitter/api/search.rb:32:in `search'
    from /Users/bob/.rvm/gems/ruby-1.9.3-p392@Rails3.2.12/gems/twitter-4.6.1/lib/twitter.rb:52:in `method_missing'
    from (irb):168
    from /Users/bob/.rvm/gems/ruby-1.9.3-p392@Rails3.2.12/gems/railties-3.2.13/lib/rails/commands/console.rb:47:in `start'
    from /Users/bob/.rvm/gems/ruby-1.9.3-p392@Rails3.2.12/gems/railties-3.2.13/lib/rails/commands/console.rb:8:in `start'
    from /Users/bob/.rvm/gems/ruby-1.9.3-p392@Rails3.2.12/gems/railties-3.2.13/lib/rails/commands.rb:41:in `<top (required)>'
    from script/rails:6:in `require'
    from script/rails:6:in `<main>'1.9.3p392 :169 > 

What is Twitter::Error::ClientError: end of file reached?

[veesahni]

Take a look at the suggestions made by @sferik in #354

[dewski]

I've had an app in production using the Twitter gem for a little over 6 months now with no issues, and all of a sudden yesterday I started noticing we're getting Twitter::Error::ClientError EOF reached consistently.

We have multiple tokens we rotate through and build new Twitter::Client's, and when I noticed the EOF error, switching to a new token would work fine. Once confirming a new token works, if I switch back to the old token that causes the EOF error, it works again. Weird. I have an example below for that behavior under Rotating the token used.

Using the suggestions in #354 yield these results (tokens hidden for security):

Using HTTPS

token = {:token=>"xx", :secret=>"xx"}
c = Twitter::Client.new(
  :oauth_token => token[:token],
  :oauth_token_secret => token[:secret]
)

=> #<Twitter::Client:0x007f8fc79b4098 @consumer_key="consumer_key", @consumer_secret="consumer_secret", @oauth_token="xx", @oauth_token_secret="xx", @endpoint="https://api.twitter.com", @connection_options={:headers=>{:accept=>"application/json", :user_agent=>"Twitter Ruby Gem 4.6.2"}, :request=>{:open_timeout=>5, :timeout=>10}, :ssl=>{:verify=>false}}, @identity_map=false, @middleware=#<Faraday::Builder:0x007f8fc2f160b8 @handlers=[Twitter::Request::MultipartWithFile, Faraday::Request::Multipart, Faraday::Request::UrlEncoded, Twitter::Response::RaiseError, FaradayMiddleware::Instrumentation, Twitter::Response::ParseJson, Twitter::Response::RaiseError, Faraday::Adapter::NetHttp, Faraday::Adapter::Excon]>>

c.user 'github'
Twitter::Error::ClientError: end of file reached
  from /opt/boxen/rbenv/versions/1.9.3-p231-tcs-github-1.0.11/lib/ruby/1.9.1/openssl/buffering.rb:174:in `sysread_nonblock'
  from /opt/boxen/rbenv/versions/1.9.3-p231-tcs-github-1.0.11/lib/ruby/1.9.1/openssl/buffering.rb:174:in `read_nonblock'
  from /opt/boxen/rbenv/versions/1.9.3-p231-tcs-github-1.0.11/lib/ruby/1.9.1/net/protocol.rb:141:in `rbuf_fill'
  from /opt/boxen/rbenv/versions/1.9.3-p231-tcs-github-1.0.11/lib/ruby/1.9.1/net/protocol.rb:122:in `readuntil'
  from /opt/boxen/rbenv/versions/1.9.3-p231-tcs-github-1.0.11/lib/ruby/1.9.1/net/protocol.rb:132:in `readline'
  from /opt/boxen/rbenv/versions/1.9.3-p231-tcs-github-1.0.11/lib/ruby/1.9.1/net/http.rb:2770:in `read_chunked'
  from /opt/boxen/rbenv/versions/1.9.3-p231-tcs-github-1.0.11/lib/ruby/1.9.1/net/http.rb:2750:in `read_body_0'
  from /opt/boxen/rbenv/versions/1.9.3-p231-tcs-github-1.0.11/lib/ruby/1.9.1/net/http.rb:2710:in `read_body'
  from /opt/boxen/rbenv/versions/1.9.3-p231-tcs-github-1.0.11/lib/ruby/1.9.1/net/http.rb:1029:in `block in get'
  from /opt/boxen/rbenv/versions/1.9.3-p231-tcs-github-1.0.11/lib/ruby/1.9.1/net/http.rb:1322:in `block (2 levels) in transport_request'
  from /opt/boxen/rbenv/versions/1.9.3-p231-tcs-github-1.0.11/lib/ruby/1.9.1/net/http.rb:2671:in `reading_body'
  from /opt/boxen/rbenv/versions/1.9.3-p231-tcs-github-1.0.11/lib/ruby/1.9.1/net/http.rb:1321:in `block in transport_request'
  from /opt/boxen/rbenv/versions/1.9.3-p231-tcs-github-1.0.11/lib/ruby/1.9.1/net/http.rb:1316:in `catch'
  from /opt/boxen/rbenv/versions/1.9.3-p231-tcs-github-1.0.11/lib/ruby/1.9.1/net/http.rb:1316:in `transport_request'
  from /opt/boxen/rbenv/versions/1.9.3-p231-tcs-github-1.0.11/lib/ruby/1.9.1/net/http.rb:1293:in `request'
  from /Users/dewski/github/xx/vendor/gems/ruby/1.9.1/gems/rest-client-1.6.7/lib/restclient/net_http_ext.rb:51:in `request'
... 14 levels...
  from /Users/dewski/github/xx/vendor/gems/ruby/1.9.1/gems/faraday-0.8.6/lib/faraday/response.rb:8:in `call'
  from /Users/dewski/github/xx/vendor/gems/ruby/1.9.1/gems/faraday-0.8.6/lib/faraday/request/url_encoded.rb:14:in `call'
  from /Users/dewski/github/xx/vendor/gems/ruby/1.9.1/gems/faraday-0.8.6/lib/faraday/request/multipart.rb:13:in `call'
  from /Users/dewski/github/xx/vendor/gems/ruby/1.9.1/gems/twitter-4.6.2/lib/twitter/request/multipart_with_file.rb:14:in `call'
  from /Users/dewski/github/xx/vendor/gems/ruby/1.9.1/gems/faraday-0.8.6/lib/faraday/connection.rb:247:in `run_request'
  from /Users/dewski/github/xx/vendor/gems/ruby/1.9.1/gems/faraday-0.8.6/lib/faraday/connection.rb:100:in `get'
  from /Users/dewski/github/xx/vendor/gems/ruby/1.9.1/gems/twitter-4.6.2/lib/twitter/client.rb:81:in `request'
  from /Users/dewski/github/xx/vendor/gems/ruby/1.9.1/gems/twitter-4.6.2/lib/twitter/client.rb:64:in `get'
  from /Users/dewski/github/xx/vendor/gems/ruby/1.9.1/gems/twitter-4.6.2/lib/twitter/api/utils.rb:82:in `object_from_response'
  from /Users/dewski/github/xx/vendor/gems/ruby/1.9.1/gems/twitter-4.6.2/lib/twitter/api/users.rb:291:in `user'
  from (irb):19
  from /Users/dewski/github/xx/vendor/gems/ruby/1.9.1/gems/railties-3.2.13/lib/rails/commands/console.rb:47:in `start'
  from /Users/dewski/github/xx/vendor/gems/ruby/1.9.1/gems/railties-3.2.13/lib/rails/commands/console.rb:8:in `start'
  from /Users/dewski/github/xx/vendor/gems/ruby/1.9.1/gems/railties-3.2.13/lib/rails/commands.rb:41:in `<top (required)>'
  from script/rails:6:in `require'
  from script/rails:6:in `<main>'>

Using HTTP

token = {:token=>"xx", :secret=>"xx"}
c = Twitter::Client.new(
  :oauth_token => token[:token],
  :oauth_token_secret => token[:secret],
  :endpoint => "http://api.twitter.com"
)

=> #<Twitter::Client:0x007f8fc272ae08 @consumer_key="consumer_key", @consumer_secret="consumer_secret", @oauth_token="xx", @oauth_token_secret="xx", @endpoint="http://api.twitter.com", @connection_options={:headers=>{:accept=>"application/json", :user_agent=>"Twitter Ruby Gem 4.6.2"}, :request=>{:open_timeout=>5, :timeout=>10}, :ssl=>{:verify=>false}}, @identity_map=false, @middleware=#<Faraday::Builder:0x007f8fc2f160b8 @handlers=[Twitter::Request::MultipartWithFile, Faraday::Request::Multipart, Faraday::Request::UrlEncoded, Twitter::Response::RaiseError, FaradayMiddleware::Instrumentation, Twitter::Response::ParseJson, Twitter::Response::RaiseError, Faraday::Adapter::NetHttp, Faraday::Adapter::Excon]>>

c.user 'github'
Twitter::Error::ClientError: end of file reached
  from /opt/boxen/rbenv/versions/1.9.3-p231-tcs-github-1.0.11/lib/ruby/1.9.1/net/protocol.rb:141:in `read_nonblock'
  from /opt/boxen/rbenv/versions/1.9.3-p231-tcs-github-1.0.11/lib/ruby/1.9.1/net/protocol.rb:141:in `rbuf_fill'
  from /opt/boxen/rbenv/versions/1.9.3-p231-tcs-github-1.0.11/lib/ruby/1.9.1/net/protocol.rb:122:in `readuntil'
  from /opt/boxen/rbenv/versions/1.9.3-p231-tcs-github-1.0.11/lib/ruby/1.9.1/net/protocol.rb:132:in `readline'
  from /opt/boxen/rbenv/versions/1.9.3-p231-tcs-github-1.0.11/lib/ruby/1.9.1/net/http.rb:2770:in `read_chunked'
  from /opt/boxen/rbenv/versions/1.9.3-p231-tcs-github-1.0.11/lib/ruby/1.9.1/net/http.rb:2750:in `read_body_0'
  from /opt/boxen/rbenv/versions/1.9.3-p231-tcs-github-1.0.11/lib/ruby/1.9.1/net/http.rb:2710:in `read_body'
  from /opt/boxen/rbenv/versions/1.9.3-p231-tcs-github-1.0.11/lib/ruby/1.9.1/net/http.rb:1029:in `block in get'
  from /opt/boxen/rbenv/versions/1.9.3-p231-tcs-github-1.0.11/lib/ruby/1.9.1/net/http.rb:1322:in `block (2 levels) in transport_request'
  from /opt/boxen/rbenv/versions/1.9.3-p231-tcs-github-1.0.11/lib/ruby/1.9.1/net/http.rb:2671:in `reading_body'
  from /opt/boxen/rbenv/versions/1.9.3-p231-tcs-github-1.0.11/lib/ruby/1.9.1/net/http.rb:1321:in `block in transport_request'
  from /opt/boxen/rbenv/versions/1.9.3-p231-tcs-github-1.0.11/lib/ruby/1.9.1/net/http.rb:1316:in `catch'
  from /opt/boxen/rbenv/versions/1.9.3-p231-tcs-github-1.0.11/lib/ruby/1.9.1/net/http.rb:1316:in `transport_request'
  from /opt/boxen/rbenv/versions/1.9.3-p231-tcs-github-1.0.11/lib/ruby/1.9.1/net/http.rb:1293:in `request'
  from /Users/dewski/github/xx/vendor/gems/ruby/1.9.1/gems/rest-client-1.6.7/lib/restclient/net_http_ext.rb:51:in `request'
  from /opt/boxen/rbenv/versions/1.9.3-p231-tcs-github-1.0.11/lib/ruby/1.9.1/net/http.rb:1286:in `block in request'
... 13 levels...
  from /Users/dewski/github/xx/vendor/gems/ruby/1.9.1/gems/faraday-0.8.6/lib/faraday/response.rb:8:in `call'
  from /Users/dewski/github/xx/vendor/gems/ruby/1.9.1/gems/faraday-0.8.6/lib/faraday/request/url_encoded.rb:14:in `call'
  from /Users/dewski/github/xx/vendor/gems/ruby/1.9.1/gems/faraday-0.8.6/lib/faraday/request/multipart.rb:13:in `call'
  from /Users/dewski/github/xx/vendor/gems/ruby/1.9.1/gems/twitter-4.6.2/lib/twitter/request/multipart_with_file.rb:14:in `call'
  from /Users/dewski/github/xx/vendor/gems/ruby/1.9.1/gems/faraday-0.8.6/lib/faraday/connection.rb:247:in `run_request'
  from /Users/dewski/github/xx/vendor/gems/ruby/1.9.1/gems/faraday-0.8.6/lib/faraday/connection.rb:100:in `get'
  from /Users/dewski/github/xx/vendor/gems/ruby/1.9.1/gems/twitter-4.6.2/lib/twitter/client.rb:81:in `request'
  from /Users/dewski/github/xx/vendor/gems/ruby/1.9.1/gems/twitter-4.6.2/lib/twitter/client.rb:64:in `get'
  from /Users/dewski/github/xx/vendor/gems/ruby/1.9.1/gems/twitter-4.6.2/lib/twitter/api/utils.rb:82:in `object_from_response'
  from /Users/dewski/github/xx/vendor/gems/ruby/1.9.1/gems/twitter-4.6.2/lib/twitter/api/users.rb:291:in `user'
  from (irb):13
  from /Users/dewski/github/xx/vendor/gems/ruby/1.9.1/gems/railties-3.2.13/lib/rails/commands/console.rb:47:in `start'
  from /Users/dewski/github/xx/vendor/gems/ruby/1.9.1/gems/railties-3.2.13/lib/rails/commands/console.rb:8:in `start'
  from /Users/dewski/github/xx/vendor/gems/ruby/1.9.1/gems/railties-3.2.13/lib/rails/commands.rb:41:in `<top (required)>'
  from script/rails:6:in `require'
  from script/rails:6:in `<main>'

Rotating the token used

token = {:token=>"first_token", :secret=>"first_token"}
connection = Twitter::Client.new(
  :oauth_token => token[:token],
  :oauth_token_secret => token[:secret]
)
=> #<Twitter::Client:0x007f8fc272ae08 @consumer_key="consumer_key", @consumer_secret="consumer_secret", @oauth_token="first_token", @oauth_token_secret="first_token", @endpoint="http://api.twitter.com", @connection_options={:headers=>{:accept=>"application/json", :user_agent=>"Twitter Ruby Gem 4.6.2"}, :request=>{:open_timeout=>5, :timeout=>10}, :ssl=>{:verify=>false}}, @identity_map=false, @middleware=#<Faraday::Builder:0x007f8fc2f160b8 @handlers=[Twitter::Request::MultipartWithFile, Faraday::Request::Multipart, Faraday::Request::UrlEncoded, Twitter::Response::RaiseError, FaradayMiddleware::Instrumentation, Twitter::Response::ParseJson, Twitter::Response::RaiseError, Faraday::Adapter::NetHttp, Faraday::Adapter::Excon]>>

connection.user 'github'
Twitter::Error::ClientError: end of file reached

TokenPool.refresh # => {:token=>"second_token", :secret=>"second_token"}
token = TokenPool.get # => {:token=>"second_token", :secret=>"second_token"}
connection = Twitter::Client.new(
  :oauth_token => token[:token],
  :oauth_token_secret => token[:secret]
)
=> #<Twitter::Client:0x00000004e65880 @consumer_key="consumer_key", @consumer_secret="consumer_secret", @oauth_token="second_token", @oauth_token_secret="second_token", @endpoint="https://api.twitter.com", @connection_options={:headers=>{:accept=>"application/json", :user_agent=>"Twitter Ruby Gem 4.6.2"}, :request=>{:open_timeout=>5, :timeout=>10}, :ssl=>{:verify=>false}}, @identity_map=false, @middleware=#<Faraday::Builder:0x00000001f55518 @handlers=[Twitter::Request::MultipartWithFile, Faraday::Request::Multipart, Faraday::Request::UrlEncoded, Twitter::Response::RaiseError, FaradayMiddleware::Instrumentation, Twitter::Response::ParseJson, Twitter::Response::RaiseError, Faraday::Adapter::NetHttp]>>

begin
  connection.user('github')
rescue ::Twitter::Error::ClientError => exception
  p exception.wrapped_exception
  p exception.rate_limit
end

=> #<Twitter::User:0x00000004eb5290>

TokenPool.lock 'first_token' # => true
token = TokenPool.get # => {:token=>"first_token", :secret=>"first_token"}
connection = Twitter::Client.new(
  :oauth_token => token[:token],
  :oauth_token_secret => token[:secret]
)
=> #<Twitter::Client:0x00000005158420 @consumer_key="consumer_key", @consumer_secret="consumer_secret", @oauth_token="first_token", @oauth_token_secret="first_token", @endpoint="https://api.twitter.com", @connection_options={:headers=>{:accept=>"application/json", :user_agent=>"Twitter Ruby Gem 4.6.2"}, :request=>{:open_timeout=>5, :timeout=>10}, :ssl=>{:verify=>false}}, @identity_map=false, @middleware=#<Faraday::Builder:0x00000001f55518 @handlers=[Twitter::Request::MultipartWithFile, Faraday::Request::Multipart, Faraday::Request::UrlEncoded, Twitter::Response::RaiseError, FaradayMiddleware::Instrumentation, Twitter::Response::ParseJson, Twitter::Response::RaiseError, Faraday::Adapter::NetHttp]>>

begin
  connection.user('github')
rescue ::Twitter::Error::ClientError => exception
  p exception.wrapped_exception
  p exception.rate_limit
end

=> #<Twitter::User:0x00000005d50a98>

Catching the error

Using both HTTPS and HTTP endpoints yield:

begin
  c.user('github')
rescue Twitter::Error::ClientError => exception
  p exception.wrapped_exception # => #<Faraday::Error::ConnectionFailed>
  p exception.rate_limit # => #<Twitter::RateLimit:0x00000004e27e18 @attrs={}>
end

Seems odd all of this would work for months then all of a sudden break, could this actually be something on Twitter's side? When logging into dev.twitter.com you'll see that they're testing their blackouts of 1.0:

March 20, 2013 The next API v1 blackout test is TODAY, March 20th, 2013 from 3PM-4PM PST. | Read more →

[veesahni]

I had implemented an exponential backoff (with notification only after a certain threshold of sequential failures) which did hide the errors for the last few weeks until last night.

Given updates in #369, #370 and #354, it seems a number of people have been experiencing the same error since last night.

[BobWalsh]

dewski, thanks for the code - may have to implement my own "pool". Have reported it to Twitter here: https://dev.twitter.com/issues/927 (unreviewed, so you might not see it, so here it is again):

Please have a look at:

#370

#369

and

#354

Various solutions (disable HTTPS, change default HTTP adaptor via setting a custom middleware stack, using gem 'faraday', '0.9.0.pre' [https://github.com//issues/354#issuecomment-14140759], and switching between tokens [https://github.com//issues/370#issuecomment-15191639] have been tried. They sometimes work.

As I am writing up this issue, here's the identical results I'm getting on two different macs (a imac 2009, os upgraded repeatedly and a MBP Retina, bought 2 months ago):

thetweets = Twitter.search('git', :count => 10, :lang => "en", :result_type => "recent") -- works

thetweets = Twitter.search('1git', :count => 10, :lang => "en", :result_type => "recent") -- Twitter::Error::ClientError: end of file reached

Ran the same on other similar terms - "got" worked, "1got" did not.

BTW, All searches returned results in twitter.com in browser.

I can't see how, especially when switching tokens and repeating the request worked that this can be some sort of error out here. ---> [https://github.com//issues/370#issuecomment-15191639]

Regards,
Bob Walsh
bob.walsh@47hats.com

[sferik]

Seems odd all of this would work for months then all of a sudden break, could this actually be something on Twitter's side? When logging into dev.twitter.com you'll see that they're testing their blackouts of 1.0:

March 20, 2013 The next API v1 blackout test is TODAY, March 20th, 2013 from 3PM-4PM PST. | Read more →

This error is occurring on both API v1.0 and v1.1. Also, it’s currently 1 PM PDT, so I don’t believe this is related to the blackout.

[asanderson15]

Am relatively new to using this gem, but I am having the same issue. Today is the first time I have seen this error.

It seems like it is only happening when I add parameters (in my case geocode) beyond just the search query term. I haven't been able to duplicate the issue with only query terms and no params, but maybe the gem (or Twitter) is treating those searches differently.

[sferik]

@BobWalsh @dewski @veesahni @asanderson15 @timhaines @stuliston @jfredson @joekain (CCing everyone from #354 and #369): I believe I've discovered the root cause of this issue. It appears to be caused by using a version of Ruby that was built against an older version of OpenSSL. I discovered this while trying to reproduce this bug in different Ruby environments. I was able to successfully reproduce the issue on 1.8.7-p371, 1.9.2-p320, and 1.9.3-p392, however I could not reproduce the issue on Ruby 2.0.0-p0. The difference is that I built all the earlier versions of Ruby against my system OpenSSL:

$ openssl version
OpenSSL 0.9.8r 8 Feb 2011

However, Ruby 2.0.0-p0 requires a newer version of OpenSSL, so I installed OpenSSL 1.0.1e via homebrew and built it with the flag:

--with-openssl-dir=`brew --prefix openssl`

Can someone else please verify this hypothesis? Either try reproducing the problem on Ruby 2.0.0 (which requires a modern OpenSSL version) or manually build an earlier version of Ruby with the --with-openssl-dir flag set.

[dewski]

I am unable to test this out on the production/staging environment where this is happening as that is on Heroku, but I'll try to recompile locally and get back with any results.

[sferik]

@dewski You can check what version of OpenSSL is installed on Heroku with the command:

heroku run openssl version

As of February 24, Heroku officially supports Ruby 2.0.0-p0, which should be built against a relatively modern version of OpenSSL.

[dewski]

$ heroku run openssl version --app xx
Running `openssl version` attached to terminal... up, run.6604
OpenSSL 0.9.8k 25 Mar 2009

[dewski]

@sferik what's the end command you're running to compile ruby? I'm not having any luck with straight ./configure or ruby-build.

[skyshard]

In a heroku dyno running ruby 2.0.0p0:

~ $ openssl version
OpenSSL 0.9.8k 25 Mar 2009
~ $ ruby -v
ruby 2.0.0p0 (2013-02-24 revision 39474) [x86_64-linux]
~ $ rails c
(startup output snipped)

irb(main):002:0> OpenSSL::OPENSSL_VERSION
=> "OpenSSL 0.9.8k 25 Mar 2009"

looks like a custom buildpack might be needed for new heroku rubies?

[sferik]

Apparently I was wrong about Ruby 2.0.0 requiring a recent OpenSSL version. For those who were able to upgrade Ruby 2.0.0, did that solve the problem or are you still seeing the error?

@dewski I’m installing Ruby via ruby-build. You can clone the repo and run ./install.sh or install it via homebrew. Then run ruby-build 2.0.0-p0 ~/local/2.0.0-p0. Has anyone tried building 1.9.3-p392 with a newer version of OpenSSL? I'm going to try this now.

[skyshard]

Upgrading to 2.0.0 (with the old openssl) didn't solve the problem for me- still consistently getting the error. Will try rebuilding next

[iko]

I am able to reproduce this only when querying non-existent users.

Using ruby-2.0.0-p0 and openssl version 1.0.1e

[yasuoza]

Me too...

$ ruby -v 
ruby 2.0.0p0 (2013-02-24 revision 39474) [x86_64-darwin12.2.1]

[1] pry(main)> OpenSSL::OPENSSL_VERSION
=> "OpenSSL 1.0.1e 11 Feb 2013"

[2] pry(main)> Twitter.search('1git', :count => 10, :lang => "en", :result_type => "recent")
Twitter::Error::ClientError: end of file reached
from /Users/Yasu/.rbenv/versions/2.0.0-p0/lib/ruby/2.0.0/openssl/buffering.rb:174:in `sysread_nonblock'

[asanderson15]

I recompiled 1.9.3-p392 with OpenSSL 1.0.1e and confirmed the right version was running in the rails console, but still having the same problem where some terms return results while others consistently get the EOF error.

It doesn't seem to be entirely intermittent. Some terms consistently work (eg git, foo), while others consistently don't (eg, 1git, 'foo bar'). And I am only able to reproduce the issue when including params like :geocode.

[sferik]

Thanks for rebuilding your Ruby and confirming the bug, everyone. I’m not sure why it’s working for me on Ruby 2.0.0 with the latest version of OpenSSL. I’ve opened an issue with Twitter to look into the problem on their end. I’ll keep investigating on my end too. In the mean time, please use one of the workarounds described in #354.

[dangerp]

Looks like a twitter dev has suggested a few options here: https://dev.twitter.com/discussions/15989

[waynepan]

Posted a comment to @dangerp's link (waiting approval)... to me it might a Twitter issue? Faraday sends "Connection: close" header by default.

$ curl --compressed --header "Connection: close" http://api.twitter.com/1.1/users/show.json?user_id=33978
curl: (18) transfer closed with outstanding read data remaining
{"errors":[{"message":"Bad Authentication data","code":215}]}

$ curl --compressed http://api.twitter.com/1.1/users/show.json?user_id=33978
{"errors":[{"message":"Bad Authentication data","code":215}]}

Any way to do the following w/o monkey patching? https://gist.github.com/waynepan/5218264

[johnbarratt]

I've just added an updated post to the Twitter discussion, this appears to be a bug on Twitter's side from what I can see. Essentially for chunked responses they are not sending the final '0' chunk, and so ruby is raising an EOF error because of that. I have posted a full explaination and work around on the thread :

https://dev.twitter.com/discussions/15989

Not sure if ruby should be made to be tolerant of this, perhaps depends how many rouge servers there are out there, and if there are any potential serious side effects, such as blindly accepting a partial transfer, thinking it is a full transfer.

[dewski]

@johnbarratt great job digging deep down into it. Looking forward to the response on Twitter's side!

[awvidmer]

+1 dewski re @johnbarratt : not doing a thing until Twitter responds. Hoping to have Twitter on our sites magically start working again.... :)

[mikestanley]

Man I'm glad others have run into this same issue. Kudos @johnbarratt for the diligence. that saved me some work. I couldn't get the Net HTTP patch to work, but I was able to utilize the Accept Encoding trick.

For what it's worth, I did this without a patch.

client = Twitter:Client.new(opts)
client.send(:connection).headers["Accept-Encoding"] = ""
client.search(....)

same difference in the end. I really wish this was resolved though.

I found that if the results from a search are <= 1 than this END OF FILE error is hit. otherwise it seems to work. To test this I simply add :count=>1 to the search opts and could reproduce the error every time.

[shayonj]

With so much information being exchanged all over the place, I am a little confused. I tried the HTTP patch but it didnt work for me.

UPDATE: If the hashtag or the content you are looking for hasnt yet syndicated with twitter api yet, it gives the EOL error. At least for me it does :)

[johnbarratt]

@shayonj Are you using ruby 1.9.3, and adding it in after the twitter gem loads, but before your code calls it?

In terms of what is happening, I just tried to follow up again on the thread on Twiter's forums (https://dev.twitter.com/discussions/15989) , but my latgest post has been moderated for some reason :( . There is a further example from @Waynep using curl, and a follow up from myself from yesterday that further seems to indicate the problem is on Twitter's side, though they haven't been able to verify it from their end as yet. Hopefully we will hear from them again before the day is out in SF.

[shayonj]

@johnbarratt yep i am using 1.9.3 and i tried changing the order but nothing happens.

Well I really hope they figure it out :). Thanks for taking of care of it.

On a side note if I run a query for

client.search("#holi", :lang => "en", :result_type => "recent")

It returns fine. but if I do

client.search("#shayon", :lang => "en", :result_type => "recent")

Which apparently only has 3 tweets in total. It returns the EOL error. Not sure why.

[johnbarratt]

@shayonj : If you don't care about the response being gzip encoded, you could try the workaround from @mikestanley from above:

client = Twitter:Client.new(opts)
client.send(:connection).headers["Accept-Encoding"] = ""
client.search(....)

This turns off gzip compression for the response, and works around the bug.

[shayonj]

@johnbarratt yep I used to test couple of things out. But gotta need a fix along with gzip compression :)

[johnbarratt]

FYI, Twitter can now see the error happening internally, and confirm something is broken there (see latest from @kurrik on https://dev.twitter.com/discussions/15989), so hopefully there will be a fix soon! :)

[waynepan]

Not sure if this is better, but this also fixes it. It retains gzip (less data) AFAIK.

client = Twitter:Client.new(opts)
client.send(:connection).headers["Connection"] = ""
client.search(....)

[DavidHuie]

The developers at Twitter claim this issue has been fixed: https://dev.twitter.com/discussions/15989

[rymai]

I confirm that it's fixed for me (and some others on the Twitter's thread)!

[shayonj]

same here.