How to add an elasticsearch alert query

The goal was to create a new alert for detecting error codes between a min/max range and per device.

The main point to know is that there is an added LAYER between the code and elastic search in order to hid the lower details of elastic search [at the expense of losing some elastic search query configuration]. It is difficult initially to understand this layer, this note describes how I asked a new alert query.

Initially I experimented with the query directly in elastic search,

GET /cdx_development/test/_search {

"filter" : {

"range": {

    "test.error_code": {

        "gte":  "100",

        "lte":  "400"

    }
}

},

"query": {

"filtered": {

  "query": {

    "term": {

      "device.uuid" : "y9LnSAFpyF3l"

    }
  }
}}}

this may not be needed if your query is similar to existing ones, and just use the added abstraction layer in the code: https://github.com/instedd/cdx-api-elasticsearch. An important point is that this gem is no longer maintained due to coupling, the code is not directly in the project under deps/cdx-api-elasticsearch/ and there is no need to add changes to this gem rather change the code directly in deps/cdx-api-elasticsearch/.

I used the /api/playground to add a sample with an alert: I create a cepid device/model and added this sample:

{

"event": {

"result": "MTB DETECTED MEDIUM",

"assay_name": "MTB-RIF Assay G4_noQC1_2",

"sample_ID": "A-1234-5",

"system_user": "JaneDoe",

"device_serial_number": "123456",

"end_time": "2015-04-07T20:08:41-05:00",

"start_time": "2015-04-07T18:31:20-05:00",

"event_id": "76B4AFBA09A143718CED4767ACC4EAE7",

"test_type": "0",

"error_code":"155"

} }

NOW TO ADD THE ALERT

to help debug it:

in class Api::TestsController < ApiController

…

put a breakpoint so you can change the filters value passed.

filters = params.merge(body)

p "++++++++++++++++++"+filters.to_s

binding.pry

query = TestResult.query(filters, current_user)

STEP 1: then trigger it:

rspec spec/controllers/api/tests_controller_query_spec.rb

STEP 2: change the filters variable to

filters= {"test.error_code" => "155”}

Note: you do not need to define this filter, for a standard field the elastic search code will automatically convert it.

STEP 3: In class Cdx::Api::Elasticsearch::Query to see the actual elastic search query generated, put a breakpoint at

def query(params)

binding.pry

query = elasticsearch_query

You may need to paste that into elastic search ‘sense’ to see if it works,

this generated this query {:bool=>{:must=>[{:match=>{"test.test.error_code"=>"155"}}, {:match=>{"test.institution.uuid"=>"61581214-4ab6-b677-2324-82b6e6836075"}}]}}

STEP 4 To now create a custom query for min/max error code ranges

in cdx_api_fields.ml add:

test:

• name: test.error_code

  filter_parameter_definition:

  • include_default: true

  • name: test.error_code.min

    type: range-integer

    boundary: gte

  • name: test.error_code.max

    type: range-integer

    boundary: lte

STEP 5 As there was only a range section defined for dates I needed to create the range-integer for integers

in query.rb, def process_field field_definition, filter_definition, params, conditions

add when "range-integer"

    conditions.push range: {field_definition.name => ({filter_definition["boundary"] => field_value})}

STEP 6 call the spec again but for the filters use filters= {"test.error_code.min" => "1", "test.error_code.max"=>"200"}

check the results

STEP 7: If you want to check per device then you do not need to change the alert error_code definition you just need to call the filter as

filters={"test.error_code.min" => "1", "test.error_code.max"=>"500", "device.uuid" => "y9LnSAFpyF3l"}

this generates this elastic search query

[4] pry(#)> query = elasticsearch_query => {:bool=> {:must=> [{:range=>{"test.test.error_code"=>{"gte"=>"1"}}}, {:range=>{"test.test.error_code"=>{"lte"=>"500"}}}, {:match=>{"test.device.uuid"=>"y9LnSAFpyF3l"}}, {:match=>{"test.institution.uuid"=>"3ad6e748-9d4a-2c70-d5cf-1e937649a2d7"}}]}}

STEP 8: Did a rspec for it spec/controllers/api/tests_controller_query_spec.rb

it "filters by error code" do