Cannot add search template in ES 6.3.1

[alexvollmer]

I've been testing an upgrade from Elasticsearch 5.6 to the latest (6.3.1) and ran into an issue with one of our rake tasks that creates a search template. Using v6.1.0 of the ruby client, I get the following error when attempting to use the put_template method:

D, [2018-07-17T10:08:32.043236 #17583] DEBUG -- sentry: ** [Raven] [400] {"error":{"root_cause":[{"type":"invalid_index_name_exception","reason":"Invalid index name [_search], must not start with '_', '-', or '+'","index_uuid":"_na_","index":"_search"}],"type":"invalid_index_name_exception","reason":"Invalid index name [_search], must not start with '_', '-', or '+'","index_uuid":"_na_","index":"_search"},"status":400} excluded from capture due to environment or should_capture callback
rake aborted!
Elasticsearch::Transport::Transport::Errors::BadRequest: [400] {"error":{"root_cause":[{"type":"invalid_index_name_exception","reason":"Invalid index name [_search], must not start with '_', '-', or '+'","index_uuid":"_na_","index":"_search"}],"type":"invalid_index_name_exception","reason":"Invalid index name [_search], must not start with '_', '-', or '+'","index_uuid":"_na_","index":"_search"},"status":400}
/Users/avollmer/.rvm/gems/ruby-2.4.4@webcrawl/gems/elasticsearch-transport-6.1.0/lib/elasticsearch/transport/transport/base.rb:205:in `__raise_transport_error'
/Users/avollmer/.rvm/gems/ruby-2.4.4@webcrawl/gems/elasticsearch-transport-6.1.0/lib/elasticsearch/transport/transport/base.rb:323:in `perform_request'
/Users/avollmer/.rvm/gems/ruby-2.4.4@webcrawl/gems/elasticsearch-transport-6.1.0/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
/Users/avollmer/.rvm/gems/ruby-2.4.4@webcrawl/gems/elasticsearch-transport-6.1.0/lib/elasticsearch/transport/client.rb:131:in `perform_request'
/Users/avollmer/.rvm/gems/ruby-2.4.4@webcrawl/gems/elasticsearch-api-6.1.0/lib/elasticsearch/api/actions/put_template.rb:21:in `put_template'
/Users/avollmer/Development/webcrawl/lib/es_common.rb:10:in `install_search_template'
/Users/avollmer/Development/webcrawl/rakelib/es.rake:208:in `block (2 levels) in <top (required)>'
/Users/avollmer/.rvm/gems/ruby-2.4.4@global/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => es:update_search_templates
(See full trace by running task with --trace)

According to the docs, I believe the correct approach is to issue a POST against _scripts/<template-name>, but looking at the code it is issuing a PUT against _search/template/<template-name>. Am I reading this correctly? Is there a different API call I should use instead?

[karmi]

Hi @alexvollmer, I've looked at the API specification and it should indeed be a POST request to /_search/template — I'll make the neccessary fixes.

[estolfo]

Hi @alexvollmer Is this still an issue? It looks like you are calling put_template, which should accept either POST or PUT according to this https://github.com/elastic/elasticsearch/blob/master/rest-api-spec/src/main/resources/rest-api-spec/api/indices.put_template.json
If it is still an issue, can you please provide some sample code that causes this error?
Thanks!

[estolfo]

I'm going to close this because of no response. Please comment here if it persists.

[alexvollmer]

Sorry for the late response. I'm still seeing this with elasticsearch-transport-6.1.0. Here is the error and backtrace:

Elasticsearch::Transport::Transport::Errors::BadRequest:
       [400] {"error":{"root_cause":[{"type":"invalid_index_name_exception","reason":"Invalid index name [_search], must not start with '_', '-', or '+'","index_uuid":"_na_","index":"_search"}],"type":"invalid_index_name_exception","reason":"Invalid index name [_search], must not start with '_', '-', or '+'","index_uuid":"_na_","index":"_search"},"status":400}
     # /Users/avollmer/.rvm/gems/ruby-2.4.5@webcrawl/gems/elasticsearch-transport-6.1.0/lib/elasticsearch/transport/transport/base.rb:205:in `__raise_transport_error'
     # /Users/avollmer/.rvm/gems/ruby-2.4.5@webcrawl/gems/elasticsearch-transport-6.1.0/lib/elasticsearch/transport/transport/base.rb:323:in `perform_request'
     # /Users/avollmer/.rvm/gems/ruby-2.4.5@webcrawl/gems/elasticsearch-transport-6.1.0/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
     # /Users/avollmer/.rvm/gems/ruby-2.4.5@webcrawl/gems/elasticsearch-transport-6.1.0/lib/elasticsearch/transport/client.rb:131:in `perform_request'
     # /Users/avollmer/.rvm/gems/ruby-2.4.5@webcrawl/gems/elasticsearch-api-6.1.0/lib/elasticsearch/api/actions/put_template.rb:21:in `put_template'
     # ./lib/es_common.rb:11:in `install_search_template'
     # ./spec/lib/services/searching_spec.rb:11:in `block (2 levels) in <top (required)>'

I ran tcpflow to see what was actually going over the wire and it appears this method is still issuing the request against the wrong endpoint:

::1.54706-::1.09210: PUT /_search/template/companies_search HTTP/1.1
Host: localhost:9210
Accept: */*
User-Agent: Faraday v0.9.2
Content-Type: application/json
Content-Length: 3477
Expect: 100-continue

The current source looks like it's still pointed at the wrong endpoint, if I'm reading this correctly.

[alexvollmer]

@estolfo If this change looks right, I'll turn it into a real PR.

[alexvollmer]

I also ran into a related issue with rendering search templates, for which I also have a proposed fix

[estolfo]

Sorry, I've removed my last response as I realized I was talking about the wrong template API. I'll have an update soon!

[estolfo]

@alexvollmer Would you mind providing the line of Ruby code where you are creating the template?

[alexvollmer]

@estolfo here's a code snippet for how we're loading the search template:

    res = self.client.put_template(id: template_name, body: {
      script: {
        lang: 'mustache',
        source: template_body
      }
    })

In this instance, self.client is an instance of Elasticsearch::Client and the template_body is a string that looks sort of like this:

{
  "query": {
    "function_score": {
      "query": {
        "bool": {
          "should": [
            {{#names}}
            {
              "match_phrase": {
                "searchable_name": {
                  "query": "{{ name }}",
                  "boost": "{{ weight }}"
                }
              }
            },
            {
              "match_phrase": {
                "dba_aliases": {
                  "query": "{{ name }}",
                  "boost": "{{ weight }}"
                }
              }
            }
          }
        }
      }
    }
  }
}

Thanks for re-opening the ticket. Let me know if I can provide any additional information.

[estolfo]

@alexvollmer Thanks for the response and code sample.
I think I've finally tracked down what the issues was. The endpoint for the get and put template APIs must use _scripts. I've opened pull request #575 to fix it. Would you mind testing it and letting me know if it works for you?

As for the render_template API, you can successfully send the request if you provide the id in the body of the request like this:

client.render_search_template(body: { id: 'test'})

Let me know if you test that and it works for you.
Thanks!

[alexvollmer]

@estolfo I tested your branch locally and it looks like that change fixes the issues I was seeing. As far as I'm concerned, that patch looks 👍

Thanks!

[estolfo]

Closing, as this issue is resolved by the referenced pull request #575