"index" param appears to be ignored

[webmat]

• Version: LS 6.3.x branch, ES 6.3.0
• Operating System: Mac OS
• Config File (if you have sensitive info, please remove it):
• Sample Data: (generated)
• Steps to Reproduce:

bin/logstash --log.level debug -e "input { generator { count =>  3 } }
filter { elasticsearch {
  user => elastic password => '$ES_PWD' hosts => ['https://example.us-central1.gcp.cloud.es.io:9243/']
  index => 'cert' query => '*' fields => { 'sequence' => 'last_sequence' }
} }
output { stdout {} elasticsearch {
  user => elastic password => '$ES_PWD' hosts => ['https://example.us-central1.gcp.cloud.es.io:9243/']
  index => 'cert'
} }"

When executing this pipeline, dummy data is inserted into the "cert" index. This is an otherwise empty ES instance. It only has one other index, ".kibana".

The filter should search only in the "cert" index, according to index => 'cert'. However I'm getting an error to the effectr that the .kibana index doesn't have a @timestamp field to sort on.

[2018-06-20T14:00:31,579][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"cert", :query=>"*", :event=>#<LogStash::Event:0x530c4ab1>, :error=>#<RuntimeError: Elasticsearch query error: [{"shard"=>0, "index"=>".kibana", "node"=>"xtxlP5pNS_2vmUKeXylZ5A", "reason"=>{"type"=>"query_shard_exception", "reason"=>"No mapping found for [@timestamp] in order to sort on", "index_uuid"=>"I1jLWOTUStuiVew5Ew0AVg", "index"=>".kibana"}}]>}

[webmat]

Got this error while investigating #102

[nick-george]

I'm using Logstash version 6.1.1 with ES 6.2.4 on Oracle Linux.

I am having possibly the exact same issue using this filter to do percolation. When searching, the request seems to hit all indices. I've had a look through the code, and I can see it specifying the correct index to the ruby elasticsearch library, so I'm not sure what's going on.

In an attempt to figure out the issue, I've written a ruby script that runs under JRUBY and uses the same version of the ruby elasticsearch libraries, and is doing exactly the same query. However I haven't been able to reproduce the issue. It definitely seems to be related to this filter, or the fact that it's running in Logstash. Happy to share my config if that helps, although percolation is a bit more involved.

[nick-george]

Any movement on this? It is still happening for me, and unfortunately totally breaking my percolation through logstash.

[nick-george]

This no longer seems to be happening to me in Logstash 7.3.