unable to index with logstash with current master

[rmuir]

This was working for me as of a few weeks ago. but now I hit this:

Got error to send bulk of actions to elasticsearch server at localhost : [400] {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Malformed action/metadata line [1], expected a simple value for field [_id] but found [VALUE_NULL]"}],"type":"illegal_argument_exception","reason":"Malformed action/metadata line [1], expected a simple value for field [_id] but found [VALUE_NULL]"},"status":400} {:level=>:error}

I'm using the simple config here: https://github.com/peterskim12/elk-index-size-tests

[s1monw]

isn't this a logstash issue in the first place? I mean null is not a valid id?

[rmuir]

I don't know anything about the particulars, I just wanted to raise the issue. I'm using logstash 1.5 fwiw.

[ph]

@rmuir When it was working you were running logstash 1.4.2?

[rmuir]

When i started doing index size tests with master, I started with a 1.5 release candidate but upgraded to the official release.

Since then i semi-regularly index that test dataset with logstash into es master, because its a little sample index for debugging, etc. (in this case, I am looking at our stats api so i wanted some data there)

[ph]

@rmuir I'll have a look, lets keep this issue open, if needed I'll migrate it.

[s1monw]

@ph this likely caused by #10977 since we now barf if there is an invalid value

[s1monw]

btw. this is also in 1.6 so we better check if we wanna keep it there since it seems to break logstash?

[tlrx]

#11331 throws an exception if a "metadata" field (like _id , _type etc) contains a null value. Let me know if this must be fixed on ES side.

[colinsurprenant]

we should see if this is a regression on the logstash-output-elasticsearch plugin /cc @talevy .
I suggest we test with 1.4.2 to see if we can reproduce and this will tell us if this is a new behaviour introduced in the plugin. Note that there's been a handful of point releases of logstash-output-elasticsearch in the last weeks so it can also be a regression within the 1.5.0 release cycle.

[clintongormley]

@tlrx i think that null values should be ignored here

[ph]

@rmuir @s1monw I've tested with 1.5.0 and I can reproduce the bug, this also break logstash 1.4.2.
Even if it wasn't the intended behavior, this feel like a bit breaking for a 1.x release?

[ph]

Easy fix on our side we can remove the null keys before doing the bulk request.

[clintongormley]

@ph we should just fix this before 1.6, to ignore null values.

[ph]

@clintongormley 👍

[tlrx]

This will be fixed by #11459, sorry for the regression