New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed to derive xcontent from org.elasticsearch.common.bytes.ChannelBufferBytesReference #8595

Closed
jlecour opened this Issue Nov 21, 2014 · 21 comments

Comments

Projects
None yet
@jlecour
Copy link
Contributor

jlecour commented Nov 21, 2014

Hi,

I've recently upgraded to 1.4.0 and I have some transient errors, like this one :

ElasticsearchParseException[Failed to derive xcontent from org.elasticsearch.common.bytes.ChannelBufferBytesReference@143d90f]

It's very strange since it happens on various indices. Some of them are very stable (a handful of update each day), some of them are very volatile (thousands of concurrent read/writes during the day, then left alone).

I can't reproduce those errors. I can't pinpoint any specific query.
I've been logging a lot and each time a query raise this error, I re-execute it and it runs well.

I've not tried to downgrade to 1.3.x yet. Maybe I will.

I've been raking my brain for many days, with quite some frustration hence this evasive bug report.

If anyone has the beginning of an idea, I'd be happy to try and give as much information as I can.

@jlecour

This comment has been minimized.

Copy link
Contributor

jlecour commented Nov 21, 2014

A couple of details to add :

I get those errors with

  • Elasticsearch 1.4.0
  • Debian stable 7.7
  • OpenJDK 7u65-2.5.1-5~deb7u1
  • Ruby 2.1.5
  • elasticsearch gem 1.0.6

I don't get any error with

  • Elasticsearch 1.3.4
  • Ruby 2.1.2

There is also a couple of things that has changed in my application. I'll try to exclude those changes.

@s1monw

This comment has been minimized.

Copy link
Contributor

s1monw commented Nov 23, 2014

can we get more of the stacktrace for this error - do you know when this happens?

@jlecour

This comment has been minimized.

Copy link
Contributor

jlecour commented Nov 25, 2014

I've found this in my Elasticsearch log :

[2014-11-25 10:03:22,940][WARN ][http.netty               ] [Bis] Caught exception while handling client http traffic, closing connection [id: 0xf14bdac7, /127.0.0.1:54741 => /127.0.0.1:9200]
java.lang.IllegalArgumentException: empty text
        at org.elasticsearch.common.netty.handler.codec.http.HttpVersion.<init>(HttpVersion.java:97)
        at org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:62)
        at org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:75)
        at org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:191)
        at org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:102)
        at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500)
        at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

PS : my log level is DEBUG and there is nothing before/after this message.

@clintongormley

This comment has been minimized.

Copy link
Member

clintongormley commented Nov 25, 2014

@jlecour I can reproduce this error if I telnet to port 9200 and send some non-HTTP message, eg `FOO\n``

Do you have some process which is pinging your server to check that it is alive?

@jlecour

This comment has been minimized.

Copy link
Contributor

jlecour commented Nov 25, 2014

@clintongormley No I don't have such activity. Also, I have this error in my Ruby client's log, after _bulk requests. But as I've said, the request that trigger such an error can be further executed as is and work perfectly.

I may have more information in a few hours.

@epackorigan

This comment has been minimized.

Copy link

epackorigan commented Jan 14, 2015

i was seeing this earlier today with a slightly malformed JSON block (trying to setup some transient settings). In my case, i had extra spaces at the beginning, before the opening bracket.

this worked:

curl -XPUT localhost:9200/_cluster/settings -d '{
    "transient" : {
        "cluster.routing.allocation.disk.threshold_enabled" : false
    }
}'

this didn't (note the extra space on the beginning of the line containing "transient")

curl -XPUT localhost:9200/_cluster/settings -d '
 { "transient" : {
        "cluster.routing.allocation.disk.threshold_enabled" : false
    }
}'
@clintongormley

This comment has been minimized.

Copy link
Member

clintongormley commented Jan 15, 2015

@epackorigan Can you recreate this?

i tried setting up two nodes and running your malfunctioning request, and all worked as expected.

@epackorigan

This comment has been minimized.

Copy link

epackorigan commented Jan 17, 2015

I will try to reproduce on Monday and reconfirm.

@epackorigan

This comment has been minimized.

Copy link

epackorigan commented Jan 22, 2015

not quite monday, but here is what i have:

I was trying to migrate data off one node, so i can service it (add ram to it).
i tried the following:

curl --silent -XPUT localhost:9200/_cluster/settings -s '{
  "transient" : {
    "cluster.routing.allocation.exclude._ip" : "10.0.0.1"
  }
}'

note: there are two spaces in front of transient, and 4 in front of cluster. i received the following error:
"error" : "ElasticsearchParseException[Failed to derive xcontent from org.elasticsearch.common.bytes.BytesArray@1]",

more variations on the spacing, or formatting of the submitted data, pretty much always yielded that same error:

curl --silent -XPUT localhost:9200/_cluster/settings?pretty -s '{ "transient" : { "cluster.routing.allocation.exclude._ip" : "10.0.0.1" } }'
{
  "error" : "ElasticsearchParseException[Failed to derive xcontent from org.elasticsearch.common.bytes.BytesArray@1]",
  "status" : 400
}

until i had exactly 4/8 spaces:

curl -XPUT localhost:9200/_cluster/settings -d '{
    "transient" : {
        "cluster.routing.allocation.exclude._ip" : "10.0.0.1"
    }
}'
{"acknowledged":true,"persistent":{},"transient":{"cluster":{"routing":{"allocation":{"exclude":{"_ip":"10.0.0.1"}}}}}}

it could be that the parser there is very picky about the format of the JSON data (i'm not even sure if that's valid JSON or not).

Note: I'm running on Debian Wheezy, with OpenJDK-1.7, ES 1.4.2 (all installed from packages)

@clintongormley

This comment has been minimized.

Copy link
Member

clintongormley commented Jan 22, 2015

@epackorigan The formatting of your JSON is just fine. It's your use of parameters which is flakey ;)

Your version which doesn't work:

curl -XPUT localhost:9200/_cluster/settings -s '{

Your version which does work:

curl -XPUT localhost:9200/_cluster/settings -d '{

With the first version, you're not passing a body at all, which is why it is failing with Failed to derive xcontent from org.elasticsearch.common.bytes.BytesArray@1

@epackorigan

This comment has been minimized.

Copy link

epackorigan commented Jan 22, 2015

Doh! sorry for the noise!

@mausch

This comment has been minimized.

Copy link

mausch commented Feb 24, 2015

I just got this error from copying and pasting a request from somewhere into Marvel/Sense, which left me with a space after the request path. I googled the error and this was the first result, which was a good thing because I could fix it right away, but perhaps a "less scary" error might help other users (I imagine this must be a quite common mistake).

@bleskes

This comment has been minimized.

Copy link
Member

bleskes commented Feb 24, 2015

@mausch that's annoying. FYI We have this noted for Sense and we'll fix it to trim the request properly (putting aside the question of a better error message

@bleskes

This comment has been minimized.

Copy link
Member

bleskes commented Mar 5, 2015

@mausch FYI - the trailing space issue is fixed in marvel 1.3.1 , released today.

@clintongormley

This comment has been minimized.

Copy link
Member

clintongormley commented Apr 26, 2015

No more info on this ticket. Closing. Feel free to reopen if you're still seeing this

@GamingCoder

This comment has been minimized.

Copy link

GamingCoder commented Jul 29, 2015

I saw this error in the ES JavaScript client and documented the problem at elastic/elasticsearch-js#245

@bamarni

This comment has been minimized.

Copy link

bamarni commented Jan 27, 2016

I'm getting the same error with version 1.7.4 and the official php client.

It is with the bulk api too, even though I get this error it looks like everything worked normally, very similar to what @jlecour reported.

[EDIT] : actually it was due to an empty document in the batch, my bad.

@ulkas

This comment has been minimized.

Copy link
Contributor

ulkas commented Mar 8, 2016

@bamarni
the same at me, also en empty body in the batch json

@serg3ant

This comment has been minimized.

Copy link

serg3ant commented Apr 25, 2017

It seem to happen when bulk is empty (no actions / documents at all)

@gdubicki

This comment has been minimized.

Copy link

gdubicki commented Sep 22, 2017

Can you please reopen and change the error message?

Passing empty data to ES API seems to be a pretty common mistake...

(especially as curl makes is it easy to make this mistake - btw: I am switching to https://github.com/jakubroztocil/httpie to hitting such issues in the future)

@javanna

This comment has been minimized.

Copy link
Member

javanna commented Sep 25, 2017

@gdubicki the error messages were improved for all the relevant APIs with #23497 (went out with 5.5).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment