EOFError: end of file reached

[timhaines]

By default the Twitter gem uses Net::HTTP right? It seems there's some sort of problem under certain unknown conditions that Net::HTTP doesn't handle responses from Twitter, and returns the 'end of file reached' error.

Here's a reproducible scenario on getting a user with a dud id, and no auth. Excon handles it well:

1.9.3p392 :001 > url = "https://api.twitter.com/1.1/users/show.json?user_id=33978"
 => "https://api.twitter.com/1.1/users/show.json?user_id=33978" 
1.9.3p392 :002 > r = Excon.get url 
 => #<Excon::Response:0x007f8abcecf510 @data={:body=>"{\"errors\":[{\"message\":\"Bad Authentication data\",\"code\":215}]}", :headers=>{"content-length"=>"61", "content-type"=>"application/json; charset=utf-8", "date"=>"Tue, 19 Mar 2013 21:38:08 UTC", "server"=>"tfe", "set-cookie"=>"guest_id=v1%3A136372908877663221; Domain=.twitter.com; Path=/; Expires=Thu, 19-Mar-2015 21:38:08 UTC"}, :status=>400, :remote_ip=>"199.59.148.20"}, @body="{\"errors\":[{\"message\":\"Bad Authentication data\",\"code\":215}]}", @headers={"content-length"=>"61", "content-type"=>"application/json; charset=utf-8", "date"=>"Tue, 19 Mar 2013 21:38:08 UTC", "server"=>"tfe", "set-cookie"=>"guest_id=v1%3A136372908877663221; Domain=.twitter.com; Path=/; Expires=Thu, 19-Mar-2015 21:38:08 UTC"}, @status=400, @remote_ip="199.59.148.20"> 
1.9.3p392 :003 > uri = URI(url)
 => #<URI::HTTPS:0x007f8abca89bb8 URL:https://api.twitter.com/1.1/users/show.json?user_id=33978> 
1.9.3p392 :004 > r2 = Net::HTTP.get(uri)
EOFError: end of file reached
    from /Users/tim/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/protocol.rb:141:in `read_nonblock'
    from /Users/tim/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/protocol.rb:141:in `rbuf_fill'
    from /Users/tim/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/protocol.rb:122:in `readuntil'
    from /Users/tim/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/protocol.rb:132:in `readline'
    from /Users/tim/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/http.rb:2562:in `read_status_line'
    from /Users/tim/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/http.rb:2551:in `read_new'
    from /Users/tim/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/http.rb:1319:in `block in transport_request'
    from /Users/tim/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/http.rb:1316:in `catch'
    from /Users/tim/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/http.rb:1316:in `transport_request'
    from /Users/tim/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/http.rb:1293:in `request'
    from /Users/tim/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/http.rb:1195:in `request_get'
    from /Users/tim/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/http.rb:455:in `block in get_response'
    from /Users/tim/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/http.rb:745:in `start'
    from /Users/tim/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/http.rb:454:in `get_response'
    from /Users/tim/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/http.rb:431:in `get'
    from (irb):4
    from /Users/tim/.rvm/gems/ruby-1.9.3-p392/gems/railties-3.2.13/lib/rails/commands/console.rb:47:in `start'
    from /Users/tim/.rvm/gems/ruby-1.9.3-p392/gems/railties-3.2.13/lib/rails/commands/console.rb:8:in `start'
    from /Users/tim/.rvm/gems/ruby-1.9.3-p392/gems/railties-3.2.13/lib/rails/commands.rb:41:in `<top (required)>'
    from script/rails:6:in `require'
    from script/rails:6:in `<main>

[timhaines]

Furthermore, changing the adapter to :excon seems to result in a different error.

In an initializer:

Twitter.middleware = Faraday::Builder.new(
  &Proc.new do |builder|
    # Specify a middleware stack here
    builder.adapter :excon
  end
)

On requesting the same non existing user:

TypeError: can't convert Symbol into Integer
    from /Users/tim/.rvm/gems/ruby-1.9.3-p194/bundler/gems/twitter-972a79f20b8a/lib/twitter/identity.rb:46:in `[]'
    from /Users/tim/.rvm/gems/ruby-1.9.3-p194/bundler/gems/twitter-972a79f20b8a/lib/twitter/identity.rb:46:in `id'
    from /Users/tim/.rvm/gems/ruby-1.9.3-p194/bundler/gems/twitter-972a79f20b8a/lib/twitter/identity.rb:35:in `initialize'
    from /Users/tim/.rvm/gems/ruby-1.9.3-p194/bundler/gems/twitter-972a79f20b8a/lib/twitter/base.rb:67:in `new'
    from /Users/tim/.rvm/gems/ruby-1.9.3-p194/bundler/gems/twitter-972a79f20b8a/lib/twitter/base.rb:67:in `fetch_or_new'
    from /Users/tim/.rvm/gems/ruby-1.9.3-p194/bundler/gems/twitter-972a79f20b8a/lib/twitter/base.rb:57:in `from_response'
    from /Users/tim/.rvm/gems/ruby-1.9.3-p194/bundler/gems/twitter-972a79f20b8a/lib/twitter/api/utils.rb:83:in `object_from_response'
    from /Users/tim/.rvm/gems/ruby-1.9.3-p194/bundler/gems/twitter-972a79f20b8a/lib/twitter/api/users.rb:291:in `user'
    from /clients/me/favstar/favstar/lib/app_based_twitter_request.rb:19:in `block in user'
    from /clients/me/favstar/favstar/lib/app_based_twitter_request.rb:37:in `try_with_bearer_token_repair'
    from /clients/me/favstar/favstar/lib/app_based_twitter_request.rb:19:in `user'
    from (irb):1
    from /Users/tim/.rvm/gems/ruby-1.9.3-p194/gems/railties-3.2.13/lib/rails/commands/console.rb:47:in `start'
    from /Users/tim/.rvm/gems/ruby-1.9.3-p194/gems/railties-3.2.13/lib/rails/commands/console.rb:8:in `start'
    from /Users/tim/.rvm/gems/ruby-1.9.3-p194/gems/railties-3.2.13/lib/rails/commands.rb:41:in `<top (required)>'
    from script/rails:6:in `require'
    from script/rails:6:in `

[stuliston]

I'm getting the same thing using ruby 2.0. Is this related to #354? I see not everyone got resolution on that one...

[stuliston]

Sorry guys, scratch that. I went through @sferik's advice from #354 again and this timeI'm getting results, meaning it's a net/http issue. The steps were:

Adding typhoeus to my gemfile:

gem 'typhoeus'

Then changing the default http adapter like so:

middleware = Proc.new do |builder|
  builder.use Twitter::Request::MultipartWithFile
  builder.use Faraday::Request::Multipart
  builder.use Faraday::Request::UrlEncoded
  builder.use Twitter::Response::RaiseError, Twitter::Error::ClientError
  builder.use Twitter::Response::ParseJson
  builder.use Twitter::Response::RaiseError, Twitter::Error::ServerError
  builder.adapter :typhoeus
end

Twitter.middleware = Faraday::Builder.new(&middleware)
Twitter.search("YOLO") # hooray, works!

Sorry for the trouble. @timhaines, I hope that works for you, too.

[timhaines]

@stuliston Thanks for the notes. Looks promising. I'll give it a go when I get to my desk.

[timhaines]

I can confirm that the advice in #354 solves my test case problems (I use excon instead of typhoeus)

@sferik I think it would be worth changing the default adapter (or getting to the root cause and making the Twitter gem work around it).

A 2nd suggestion would maybe be to update the readme to show what's required to change the adapter - i.e. the full list of middleware required. Although this would be a poor solution going forward, as it means one addition to the default and everyone needs to update. Perhaps allow of the adapter to be adjusted independently?

[sferik]

@timhaines I have mixed feelings about changing the default adapter. Net::HTTP is part of the Ruby standard library. If there is a bug in it, we should really try to understand it and fix it, rather than working around the problem by adding a dependency and changing the default adapter.

I do agree, it should be easier to override the default Faraday middleware. I'll work on a patch that does just that for the next minor version.

[timhaines]

@sferik I understand the mixed feelings on changing the default. My argument would be that getting Net::HTTP fixed is going to require a new release to ruby itself. There's likely to be many people who for some reason or another would be reluctant to update their version of Ruby for this.

I suppose having it more easily overridable would be a good solution for those not willing to update their Ruby though. However, that would still mean the Twitter gem is broken by default for those on older Ruby.

[sferik]

@timhaines This doesn't seem to be happening on every request, just certainrequests. If I change your example URL:

-https://api.twitter.com/1.1/users/show.json?user_id=33978
+http://api.twitter.com/1.1/users/show.json?user_id=33978

…the error no longer occurs. If I can better understand the circumstances under which this error occurs, then I may be able to fix it in this library, either by modifying the existing code (e.g. adding middleware to correct the error) or monkey patching Net::HTTP. Either way, this solution will not require a particular Ruby version and will not require using another HTTP adapter.

[timhaines]

@sferik You're right. Sounds good.

[BobWalsh]

you might want to see #370 (comment)