Twitter input throws exception

[PaulusTM]

I'm trying to fetch Twitter metrics with the Twitter input plugin but when I enable the full_tweet option it stops working and the log fills up with exceptions.

When I remove the full_tweet line it works ok.

If your are missing information please let me know and I'll provide it asap.

Installed packages

Name        : elasticsearch
Arch        : noarch
Version     : 1.2.1
Release     : 1
Size        : 21 M
Repo        : installed
From repo   : elasticsearch
Summary     : elasticsearch
License     : 2013, Elasticsearch
Description : Elasticsearch - Open Source, Distributed, RESTful Search Engine

Name        : logstash
Arch        : noarch
Version     : 1.4.1
Release     : 1_bd507eb
Size        : 92 M
Repo        : installed
From repo   : logstash
Summary     : An extensible logging pipeline
URL         : http://logstash.net
License     : ASL 2.0
Description : An extensible logging pipeline

Logstash.conf

input {
  twitter {
    consumer_key       => ""
    consumer_secret    => ""
    oauth_token        => ""
    oauth_token_secret => ""
    keywords           => ["velocityconf"]
    full_tweet         => true
  }
}
output {
  elasticsearch {
    host => "localhost"
  }
}

/var/log/logastash/logstash.log

{
:timestamp=>"2014-06-24T21:00:18.548000+0200", 
:message=>"Failed to flush outgoing items", 
:outgoing_count=>2, 
:exception=>java.lang.ClassCastException: org.jruby.RubySymbol cannot be cast to java.lang.String, 
:backtrace=>[
"org.elasticsearch.common.xcontent.XContentBuilder.writeMap(org/elasticsearch/common/xcontent/XContentBuilder.java:1095)",
"org.elasticsearch.common.xcontent.XContentBuilder.map(org/elasticsearch/common/xcontent/XContentBuilder.java:1015)", 
"org.elasticsearch.action.index.IndexRequest.source(org/elasticsearch/action/index/IndexRequest.java:338)", 
"org.elasticsearch.action.index.IndexRequest.source(org/elasticsearch/action/index/IndexRequest.java:327)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:606)", 
"LogStash::Outputs::Elasticsearch::Protocols::NodeClient.build_request(/opt/logstash/lib/logstash/outputs/elasticsearch/protocol.rb:217)",
 "LogStash::Outputs::Elasticsearch::Protocols::NodeClient.build_request(/opt/logstash/lib/logstash/outputs/elasticsearch/protocol.rb:217)", 
"LogStash::Outputs::Elasticsearch::Protocols::NodeClient.bulk(/opt/logstash/lib/logstash/outputs/elasticsearch/protocol.rb:205)", 
"LogStash::Outputs::Elasticsearch::Protocols::NodeClient.bulk(/opt/logstash/lib/logstash/outputs/elasticsearch/protocol.rb:205)", 
"org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)", 
"LogStash::Outputs::Elasticsearch::Protocols::NodeClient.bulk(/opt/logstash/lib/logstash/outputs/elasticsearch/protocol.rb:204)", 
"LogStash::Outputs::Elasticsearch::Protocols::NodeClient.bulk(/opt/logstash/lib/logstash/outputs/elasticsearch/protocol.rb:204)",
 "LogStash::Outputs::ElasticSearch.flush(/opt/logstash/lib/logstash/outputs/elasticsearch.rb:321)",
 "LogStash::Outputs::ElasticSearch.flush(/opt/logstash/lib/logstash/outputs/elasticsearch.rb:321)",
 "RUBY.buffer_flush(/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:219)", 
"org.jruby.RubyHash.each(org/jruby/RubyHash.java:1339)", "RUBY.buffer_flush(/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:216)",
 "RUBY.buffer_initialize(/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:112)",
 "org.jruby.RubyKernel.loop(org/jruby/RubyKernel.java:1521)",
 "RUBY.buffer_initialize(/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.17/lib/stud/buffer.rb:110)",
 "java.lang.Thread.run(java/lang/Thread.java:744)"], 
:level=>:warn
}

[electrical]

Hi,

This is a known issue and will be fixed soon.
In the meantime you can use the elasticsearch output with the http protocol.

[allenfromu]

I have the exactly same issue as Daniel, any solution for that?

[electrical]

Hi,

I believe its been fixed in the master branch and will be included in the next release.
In the meantime you can use the elasticsearch with protocol set to http

[allenfromu]

Ok, thanks, when can you download the next release btw

[electrical]

there is no ETA yet i'm afraid. will let you know when we have it.

[allenfromu]

Can you please explain what mainly cause that issue?

[electrical]

The issue is that the twitter plugin creates ruby hashes with object keys instead of string keys.

The http version of elasticsearch output creates a json structure and forces keys to be strings, the native output part doesn't do that.
We've patched it so it forces to create string keys.

[colinsurprenant]

yes, PR #1450 addresses this.

[allenfromu]

Thank you guys, it is working now

[allenfromu]

By the way guys, did you ever have this problem I listed below, I am thinking this error should be incurred from the twitter stream, but not sure, by now I just ignore it, because once this error disconnected the plugin, the plugin can restart ASAP.

Error: unexpected token at '{"created_at":"Wed Jul 23 16:31:12 +0000 2014","id":491983881927528448,"id_str":"491983881927528448","text":"\u30f4\u30a9\u30f3\u30db\u30d5\u541b\u3001\u306a\u306b\u8cb7\u3063\u3066\u308bww RT @SteeleVH This should be enough for the week I think! #nutella http://t.co/PIgJPeAFys","source":"\u003ca href="http://janetter.net/" rel="nofollow"\u003eJanetter Pro for Mac\u003c/a\u003e","truncated":false,"in_reply_to_status_id":491974578013753344,"in_reply_to_status_id_str":"491974578013753344","in_reply_to_user_id":413019236,"in_reply_to_user_id_str":"413019236","in_reply_to_screen_name":"SteeleVH","user":{"id":212921741,"id_str":"212921741","name":"yoko di \u30eb\u30c3\u30ab","screen_name":"Lucca196","location":"abito a Tokyo_JAPAN","url":"http://lucca196.cocolog-nifty.com/blog/","description":"una tifosa del ciclismo in particolare l'italiano.91\u5e74\u304b\u3089\u81ea\u8ee2\u8eca\u30ed\u30fc\u30c9\u30ec\u30fc\u30b9\u89b3\u6226\u7d99\u7d9a\u4e2d\u3002\u8d14\u5c53\u306f\u30a4\u30bf\u30ea\u30a2\u3002\u65e5\u3005\u306e\u697d\u3057\u307f\u306f\u89b3\u5287\u3001\u7f8e\u8853\u5c55\u9451\u8cde\u3001\u6620\u753b\u3001\u843d\u8a9e\u4f1a\u3001\u65c5\u884c\u3001\u5199\u771f\u3092\u64ae\u308b\u3053\u3068\u30028\u6708\u306812\u6708\u306b\u306f\u5225\u4ef6\u3067\u6771\u4eac\u30d3\u30c3\u30af\u30b5\u30a4\u30c8\u306b\u5c45\u308a\u307e\u3059\u3002","protected":false,"verified":false,"followers_count":143,"friends_count":444,"listed_count":12,"favourites_count":539,"statuses_count":10226,"created_at":"Sun Nov 07 13:03:51 +0000 2010","utc_offset":32400,"time_zone":"Tokyo","geo_enabled":false,"lang":"ja","contributors_enabled":false,"is_translator":false,"profile_background_color":"EBEBEB","profile_background_image_url":"http://pbs.twimg.com/profile_background_images/574486406/back4183694484849431937.JPEG","profile_background_image_url_https":"https://pbs.twimg.com/profile_background_images/574486406/back4183694484849431937.JPEG","profile_background_tile":true,"profile_link_color":"990000","profile_sidebar_border_color":"DFDFDF","profile_sidebar_fill_color":"F3F3F3","profile_text_color":"333333","profile_use_background_image":true,"profile_image_url":"http://pbs.twimg.com/profile_images/486341849250951168/GTp_jdhK_normal.jpeg","profile_image_url_https":"https://pbs.twimg.com/profile_images/486341849250951168/GTp_jdhK_normal.jpeg","profile_banner_url":"https://pbs.twimg.com/profile_banners/212921741/1404788600","default_profile":false,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":null,"contributors":null,"retweet_count":0,"favorite_count":0,"entities":{"hashtags":[{"text":"nutella","indices":[73,81]}],"trends":[],"urls":[],"user_mentions":[{"screen_name":"SteeleVH","name":"Steele von Hoff","id":413019236,"id_str":"413019236","indices":[19,28]}],"symbols":[],"media":[{"id":491974546573242368,"id_str":"491974546573242368","indices":[82,104],"media_url":"http://pbs.twimg.com/media/BtPYPlcIMAA_S_I.jpg","media_url_https":"https://pbs.twimg.com/media/BtPYPlcIMAA_S_I.jpg","url":"http://t.co/PIgJPeAFys","display_url":"pic.twitter.com/PIgJPeAFys","expanded_url":"http://twitter.com/SteeleVH/status/491974578013753344/photo/1","type":"photo","sizes":{"medium":{"w":600,"h":800,"resize":"fit"},"thumb":{"w":150,"h":150,"resize":"crop"},"small":{"w":340,"h":453,"resize":"fit"},"large":{"w":768,"h":1024,"resize":"fit"}},"source_status_id":491974578013753344,"source_status_id_str":"491974578013753344"}]},"favorited":false,"retweeted":false,"possibly_sensitive":false,"filter_level":"medium","lang":"ja"}' {:level=>:error}

[colinsurprenant]

@allenfromu thanks. yes, PR #1450 also addresses this issue :)

[allenfromu]

Awesome, thank you so much, so do you think this issue is ignorable if I don't handle it?

[allenfromu]

it seems like 'unexpected token error' is inevitable

[colinsurprenant]

so, there's not much we can do if the received json is invalid, this PR solves the problem of the connection reset upon invalid json.

[allenfromu]

True, thank you Colin.

On Thu, Jul 24, 2014 at 10:22 AM, Colin Surprenant <notifications@github.com

wrote:

so, there's not much we can do if the received json is invalid, this PR
solves the problem of the connection reset upon invalid json.

—
Reply to this email directly or view it on GitHub
#1471 (comment)
.

[jordansissel]

The "Unexpected token at" thing on what appears to be valid json (especially the { at the start), if I recall, is a problem with the buftok library we ship Logstash with has a bug.

[jordansissel]

Thanks to all who've filed details about this bug. We should have it fixed for Logstash 1.5 (coming soon)

[mmohrwinkel]

Hi all, I have the same Issue ("Unexpected token at"), but with the JSON codec. My config looks like this:

input {
        file {
                codec => json {}
                path => "/home/user/test1.log"
                type => "JSON Test"
                start_position => 'beginning'
               }
        }
...

I have validated my test1.log with http://jsonlint.com/, which said it was valid json. If the Problem is the buftok library, will my problem probably be gone in 1.5, too?

All the best, Max

[colinsurprenant]

@mmohrwinkel can you share your test1.log json ?

[suyograo]

Fixed in #1450 and moved to plugins here -- logstash-plugins/logstash-input-twitter@560152b