Cannot find Serializer for class: org.jruby.RubyObject

[tomsommer]

Logstash 2.0, ES 2.0

{:timestamp=>"2015-11-05T13:29:45.500000+0100", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://elasticmaster.example.com:9200/\"]', but an error occurred and it failed! Are you sure you can reach elasticsearch from this machine using the configuration provided?", :client_config=>{:hosts=>["http://elasticmaster.example.com:9200/"], :ssl=>nil, :transport_options=>{:socket_timeout=>0, :request_timeout=>0, :proxy=>nil, :ssl=>{}}, :transport_class=>Elasticsearch::Transport::Transport::HTTP::Manticore, :logger=>nil, :tracer=>nil, :reload_connections=>false, :retry_on_failure=>false, :reload_on_failure=>false, :randomize_hosts=>false}, :error_message=>"Cannot find Serializer for class: org.jruby.RubyObject", :error_class=>"JrJackson::ParseError", :backtrace=>["com/jrjackson/JrJacksonBase.java:83:in `generate'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/jrjackson-0.3.6/lib/jrjackson/jrjackson.rb:59:in `dump'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/multi_json-1.11.2/lib/multi_json/adapters/jr_jackson.rb:20:in `dump'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/multi_json-1.11.2/lib/multi_json/adapter.rb:25:in `dump'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/multi_json-1.11.2/lib/multi_json.rb:136:in `dump'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.14/lib/elasticsearch/api/utils.rb:102:in `__bulkify'", "org/jruby/RubyArray.java:2414:in `map'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.14/lib/elasticsearch/api/utils.rb:102:in `__bulkify'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.14/lib/elasticsearch/api/actions/bulk.rb:82:in `bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.1.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:56:in `bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.1.2-java/lib/logstash/outputs/elasticsearch.rb:353:in `submit'", "org/jruby/ext/thread/Mutex.java:149:in `synchronize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.1.2-java/lib/logstash/outputs/elasticsearch.rb:350:in `submit'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.1.2-java/lib/logstash/outputs/elasticsearch.rb:382:in `flush'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:219:in `buffer_flush'", "org/jruby/RubyHash.java:1342:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:216:in `buffer_flush'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:193:in `buffer_flush'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:159:in `buffer_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.1.2-java/lib/logstash/outputs/elasticsearch.rb:343:in `receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/outputs/base.rb:80:in `handle'", "(eval):1904:in `output_func'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/pipeline.rb:252:in `outputworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/pipeline.rb:169:in `start_outputs'"], :level=>:error}

This error just loops.

No attempts are made to contact ES after this

[ThomasdOtreppe]

Seems related to #294.

[guyboertje]

@tomsommer, @ThomasdOtreppe - please post your config and input data example

I have fixed this in JrJackson 0.3.7. I want to verify that the JrJackson fix also fixes this error because the error message is slightly different from that of #294

[guyboertje]

@tomsommer, @ThomasdOtreppe - OTOH if you have a dev environment setup you could try bin/bundle update to fetch the new version of JrJackson and then test your config with some data known to fail before.

[tomsommer]

/opt/logstash/vendor/bundle/jruby/1.9/bin/bundle update
-bash: /opt/logstash/vendor/bundle/jruby/1.9/bin/bundle: /home/jenkins/workspace/logstash_create_rpm_artifact_20/jdk/JDK7/label/metal-p: bad interpreter: No such file or directory

/opt/logstash/bin/plugin update
Updating logstash-codec-collectd, logstash-codec-dots, logstash-codec-edn, logstash-codec-edn_lines, logstash-codec-es_bulk, logstash-codec-fluent, logstash-codec-graphite, logstash-codec-json, logstash-codec-json_lines,                                                                                                 logstash-codec-line, logstash-codec-msgpack, logstash-codec-multiline, logstash-codec-netflow, logstash-codec-oldlogstashjson, logstash-codec-plain, logstash-codec-rubydebug, logstash-filter-anonymize, logstash-filter-che                                                                                                cksum, logstash-filter-clone, logstash-filter-csv, logstash-filter-date, logstash-filter-dns, logstash-filter-drop, logstash-filter-fingerprint, logstash-filter-geoip, logstash-filter-grok, logstash-filter-json, logstash-                                                                                                filter-kv, logstash-filter-metrics, logstash-filter-multiline, logstash-filter-mutate, logstash-filter-ruby, logstash-filter-sleep, logstash-filter-split, logstash-filter-syslog_pri, logstash-filter-throttle, logstash-fil                                                                                                ter-urldecode, logstash-filter-useragent, logstash-filter-uuid, logstash-filter-xml, logstash-input-beats, logstash-input-couchdb_changes, logstash-input-elasticsearch, logstash-input-eventlog, logstash-input-exec, logsta                                                                                                sh-input-file, logstash-input-ganglia, logstash-input-gelf, logstash-input-generator, logstash-input-graphite, logstash-input-heartbeat, logstash-input-http, logstash-input-imap, logstash-input-irc, logstash-input-kafka,                                                                                                 logstash-input-log4j, logstash-input-lumberjack, logstash-input-pipe, logstash-input-rabbitmq, logstash-input-redis, logstash-input-s3, logstash-input-snmptrap, logstash-input-sqs, logstash-input-stdin, logstash-input-sys                                                                                                log, logstash-input-tcp, logstash-input-twitter, logstash-input-udp, logstash-input-unix, logstash-input-xmpp, logstash-input-zeromq, logstash-output-cloudwatch, logstash-output-csv, logstash-output-elasticsearch, logstas                                                                                                h-output-email, logstash-output-exec, logstash-output-file, logstash-output-ganglia, logstash-output-gelf, logstash-output-graphite, logstash-output-hipchat, logstash-output-http, logstash-output-irc, logstash-output-jugg                                                                                                ernaut, logstash-output-kafka, logstash-output-lumberjack, logstash-output-nagios, logstash-output-nagios_nsca, logstash-output-null, logstash-output-opentsdb, logstash-output-pagerduty, logstash-output-pipe, logstash-out                                                                                                put-rabbitmq, logstash-output-redis, logstash-output-s3, logstash-output-sns, logstash-output-sqs, logstash-output-statsd, logstash-output-stdout, logstash-output-tcp, logstash-output-udp, logstash-output-xmpp, logstash-o                                                                                                utput-zeromq
Updated logstash-input-beats 0.9.3 to 0.9.6

[guyboertje]

@tomsommer - in your case, if there is no bundle file in /opt/logstash/bin you can't update jrjackson gem.

Otherwise you would run /opt/logstash/bin/bundle update.

[ThomasdOtreppe]

Anything I can do with a LS release, 1.5.5 or 2.0.0 (no dev environment)?

[guyboertje]

@ThomasdOtreppe - not at this time.

[tomsommer]

Any way I can get the fix? I'm suffering badly under this bug.

[guyboertje]

@tomsommer - I did ask you to post your config and data sample. I need this so I can verify the fix before I merge it.

[ThomasdOtreppe]

When do you guys plan to do a release? Pretty much all my logs are dropped because of that.

[guyboertje]

@tomsommer, @ThomasdOtreppe - I am trying to release a fix. I have repeatedly asked both of you for your configs and sample data known to result in the failure.

AS A LAST RESORT YOU CAN DO THE FOLLOWING - NOTE: THIS PROCEDURE IS NOT RECOMMENDED PRACTICE AND YOU DO IT AT YOUR OWN RISK.

Step 3 takes many minutes and requires connection to the internet.

1. in LS root folder, edit Gemfile, add gem "jrjackson", "= 0.3.7" at the end of the file
2. rename Gemfile.jruby-1.9.lock to Gemfile.jruby-1.9.lock-old
3. run bin/plugin install --no-verify

In the case of step 3 failing:

1. remove the added line in Gemfile
2. reverse the rename of Gemfile.jruby-1.9.lock
3. run bin/plugin install --no-verify

---

I would be very grateful if you would report back whether this fixes the problem or post configs and sample data - so I can merge our change to LS.

[tomsommer]

I can't provide sample data, but I'm running the indexer with the above now.

[ThomasdOtreppe]

I'll provide data as soon as it triggers the issue.

[tomsommer]

Same error here. Not fixed.

[guyboertje]

@tomsommer - please confirm that the newly generated Gemfile.jruby-1.9.lock contains the line jrjackson (0.3.7) just before the line jruby-kafka (1.4.0-java)

[tomsommer]

    jmespath (1.1.3)
    jrjackson (0.3.7)
    jruby-kafka (1.4.0-java)

[guyboertje]

@tomsommer - I need to be able to recreate the problem locally. Surely you must be able to anonymise a few lines of data and your config. Either post here or send to guy |at| elastic.co.

[tomsommer]

What debug lines do you need? Is it possible to debug the request/response to ES?

[guyboertje]

@tomsommer - from the error in the lines you posted the problem is deep in the ES ruby client code that this plugin uses. As it happens the ruby client also uses my library JrJackson to serialise the events before submission to ES. Using the mechanism I posted you have tried using the patched version 0.3.7 and are still getting the same fault signature. I will need to observe this locally before I can begin to figure out where the problem lies.

Run LS with the --debug flag and capture the output. Make sure the debug output includes the fault. Post that in a Github gist and the link to it here or email a zipped file of the debug output to my email ^^.

Alternatively, post or email a line of input data and config.

I really can't help any further without this.

[tomsommer]

Log sent in mail

[guyboertje]

@tomsommer - I looked at the logs that you sent. I understand better what the cause of this problem is.
It seems that a bad geopip lookup is returning a hash with key = 12276 and value of the LS logger instance.
a good lookup returns a usable hash

"dstip"=>"N/A", "dstip_geoip"=>{"12276"=>#<Cabin::Outputs::IO:0x2a143974 @lock=#<Mutex:0x15c80441>, @io=#<File:/var/log/logstash/logstash.log>>}}
"dstip"=>"X.X.X.55", "dstip_geoip"=>{"ip"=>"X.X.X.55", "country_code2"=>"KK", ... }

This hash {"12276"=>#<Cabin::Outputs::IO:0x2a143974 @lock=#<Mutex:0x15c80441>, @io=#<File:/var/log/logstash/logstash.log>>} is not serializeable so JrJackson raises ParseError in the bulk transfer to ES.

You should not be getting this hash from your geoip lookup for "N/A" in the "dstip" field. Please debug your geoip filter setup.

[tomsommer]

Thanks, I'll look into it.
The data is the fault of the geoip filter obviously, but it seems to me that LS shouldn't completely crash because of this. Any and all data transfers after this data is hit, is stopped. Everything goes into a stall, as you can see in the log.

[guyboertje]

I understand your concerns that LS should not stop. However, currently, there is no mechanism in the pipeline to shove the 'bad' batch of buffered Events out of the way and continue. We are working on this though.

[colinsurprenant]

good catch guys 👍