Backward Compatibility Error

[skukx]

Libraries

Elasticsearch -> 6.8.0
elasticsearch-persistence -> 7.0.0.pre

Summary

When using the upgraded elasticsearch-rails gem alongside an old cluster causes some errors. This may be expected since they are major version changes, but in case backward compatibility is possible I'm posting this error.

The following error is triggered when calling repository.create_index!

Elasticsearch::Transport::Transport::Errors::InternalServerError:
        [500] {"error":{"root_cause":[{"type":"class_cast_exception","reason":"java.util.ArrayList cannot be cast to java.util.Map"}],"type":"class_cast_exception","reason":"java.util.ArrayList cannot be cast to java.util.Map"},"status":500}

Stack Trace:

/.rvm/gems/ruby-2.6.3/gems/elasticsearch-transport-7.1.0/lib/elasticsearch/transport/transport/base.rb:218:in `__raise_transport_error'
/.rvm/gems/ruby-2.6.3/gems/elasticsearch-transport-7.1.0/lib/elasticsearch/transport/transport/base.rb:338:in `perform_request'
/.rvm/gems/ruby-2.6.3/gems/elasticsearch-transport-7.1.0/lib/elasticsearch/transport/transport/http/faraday.rb:37:in `perform_request'
/.rvm/gems/ruby-2.6.3/gems/elasticsearch-transport-7.1.0/lib/elasticsearch/transport/client.rb:160:in `perform_request'
/.rvm/gems/ruby-2.6.3/gems/elasticsearch-api-7.1.0/lib/elasticsearch/api/namespace/common.rb:38:in `perform_request'
/.rvm/gems/ruby-2.6.3/gems/elasticsearch-api-7.1.0/lib/elasticsearch/api/actions/indices/create.rb:95:in `create'
/.rvm/gems/ruby-2.6.3/gems/elasticsearch-model-7.0.0.pre/lib/elasticsearch/model/indexing.rb:252:in `create_index!'

This error only began after upgrading to elasticsearch-persistence 7.0.0.pre from 6.0.0. I'd also like to note that this is only when using on an older cluster (6.8.0). The library works fine with elasticsearch 7

[skukx]

Looking at the stack, this is more likely an issue for elasticsearch gem

[estolfo]

Hi @skukx
Thanks for opening this issue. The elasticsearch-rails 7.x gem is intended to be used with Elasticsearch 7.x
That said, would you mind sharing the result of calling #mapping on your repository object? It seems that the mapping created by the repository is causing the error in Elasticsearch 7. Thanks!

[skukx]

No problem.

#<Elasticsearch::Model::Indexing::Mappings:0x00007ff4f93c6e50
 @mapping=
  {:authors=>
    {:type=>:nested,
     :properties=>
      {:first_name=>{:type=>:text},
       :id=>{:type=>:keyword},
       :last_name=>{:type=>:text},
       :title=>
        {:type=>:text,
         :fields=>{:typeahead=>{:type=>:text, :analyzer=>:typeahead}}}}},
   :author_ids=>{:type=>:keyword},
   :description=>{:type=>:text, :analyzer=>:html},
   :images=>
    {:type=>:nested,
     :properties=>
      {:size=>{:type=>:keyword},
       :url=>{:type=>:keyword, :index=>false, :doc_values=>false}}},
   :language=>{:type=>:keyword},
   :popularity=>{:type=>:double},
   :producer_ids=>{:type=>:keyword},
   :producers=>
    {:type=>:nested,
     :properties=>
      {:id=>{:type=>:integer},
       :title=>
        {:type=>:text,
         :fields=>{:typeahead=>{:type=>:text, :analyzer=>:typeahead}}}}},
   :sku=>{:type=>:keyword},
   :subtitle=>
    {:type=>:text,
     :fields=>{:typeahead=>{:type=>:text, :analyzer=>:typeahead}}},
   :tags=>{:type=>:text},
   :title=>
    {:type=>:text,
     :fields=>{:typeahead=>{:type=>:text, :analyzer=>:typeahead}}},
   :book_suggest=>
    {:type=>:object, :properties=>{:title=>{:type=>:completion}}}},
 @options=
  {:dynamic_templates=>
    [{:strings=>
       {:match_mapping_type=>"string",
        :mapping=>
         {:type=>:text,
          :fields=>{:raw=>{:type=>:keyword, :ignore_above=>256}}}}}]},
 @type=nil>

Serialized as a ruby hash it looks like:

{:dynamic_templates=>
  [{:strings=>
     {:match_mapping_type=>"string",
      :mapping=>
       {:type=>:text,
        :fields=>{:raw=>{:type=>:keyword, :ignore_above=>256}}}}}],
 :properties=>
  {:authors=>
    {:type=>:nested,
     :properties=>
      {:first_name=>{:type=>:text},
       :id=>{:type=>:keyword},
       :last_name=>{:type=>:text},
       :title=>
        {:type=>:text,
         :fields=>{:typeahead=>{:type=>:text, :analyzer=>:typeahead}}}}},
   :author_ids=>{:type=>:keyword},
   :description=>{:type=>:text, :analyzer=>:html},
   :images=>
    {:type=>:nested,
     :properties=>
      {:size=>{:type=>:keyword},
       :url=>{:type=>:keyword, :index=>false, :doc_values=>false}}},
   :language=>{:type=>:keyword},
   :popularity=>{:type=>:double},
   :producer_ids=>{:type=>:keyword},
   :producers=>
    {:type=>:nested,
     :properties=>
      {:id=>{:type=>:integer},
       :title=>
        {:type=>:text,
         :fields=>{:typeahead=>{:type=>:text, :analyzer=>:typeahead}}}}},
   :sku=>{:type=>:keyword},
   :subtitle=>
    {:type=>:text,
     :fields=>{:typeahead=>{:type=>:text, :analyzer=>:typeahead}}},
   :tags=>{:type=>:text},
   :title=>
    {:type=>:text,
     :fields=>{:typeahead=>{:type=>:text, :analyzer=>:typeahead}}},
   :book_suggest=>
    {:type=>:object, :properties=>{:title=>{:type=>:completion}}}}}

And finally serialized as json:

{
   "dynamic_templates":[
      {
         "strings":{
            "match_mapping_type":"string",
            "mapping":{
               "type":"text",
               "fields":{
                  "raw":{
                     "type":"keyword",
                     "ignore_above":256
                  }
               }
            }
         }
      }
   ],
   "properties":{
      "authors":{
         "type":"nested",
         "properties":{
            "first_name":{
               "type":"text"
            },
            "id":{
               "type":"keyword"
            },
            "last_name":{
               "type":"text"
            },
            "title":{
               "type":"text",
               "fields":{
                  "typeahead":{
                     "type":"text",
                     "analyzer":"typeahead"
                  }
               }
            }
         }
      },
      "author_ids":{
         "type":"keyword"
      },
      "description":{
         "type":"text",
         "analyzer":"html"
      },
      "images":{
         "type":"nested",
         "properties":{
            "size":{
               "type":"keyword"
            },
            "url":{
               "type":"keyword",
               "index":false,
               "doc_values":false
            }
         }
      },
      "language":{
         "type":"keyword"
      },
      "popularity":{
         "type":"double"
      },
      "producer_ids":{
         "type":"keyword"
      },
      "producers":{
         "type":"nested",
         "properties":{
            "id":{
               "type":"integer"
            },
            "title":{
               "type":"text",
               "fields":{
                  "typeahead":{
                     "type":"text",
                     "analyzer":"typeahead"
                  }
               }
            }
         }
      },
      "sku":{
         "type":"keyword"
      },
      "subtitle":{
         "type":"text",
         "fields":{
            "typeahead":{
               "type":"text",
               "analyzer":"typeahead"
            }
         }
      },
      "tags":{
         "type":"text"
      },
      "title":{
         "type":"text",
         "fields":{
            "typeahead":{
               "type":"text",
               "analyzer":"typeahead"
            }
         }
      },
      "book_suggest":{
         "type":"object",
         "properties":{
            "title":{
               "type":"completion"
            }
         }
      }
   }
}

[estolfo]

@skukx Thanks for the details. It's still hard to say exactly what is causing the error so can you also share your Repository file with its attributes? Thanks!

[skukx]

No problem,

##
# Base class for implementing Repository Pattern for elasticsearch
# @see https://github.com/elastic/elasticsearch-rails/tree/master/elasticsearch-persistence#the-repository-pattern
# @see https://github.com/elastic/elasticsearch-rails/tree/master/elasticsearch-persistence#basic-repository-mixin
#
# @example Different ways to initialize
# ApplicationRepository.new
# ApplicationRepository.new client: Elasticsearch::Client.new,
#                           index_name: :my-index,
#                           type: :_doc,
#                           klass: MyClass
#
module RepositorySettings
  extend ActiveSupport::Concern

  TIMESTAMP_FORMAT = '%Y.%m.%d.%H.%M.%S'

  included do
    ##
    # Setup the client, klass, and index name for every repository dynamically
    #
    client Elasticsearch::Client.new(url: Rails.application.credentials.elasticsearch_url)
    klass self.name.gsub(/Repository$/, '').constantize
    index_name "#{klass.to_s.tableize.dasherize}-#{Rails.env.downcase}"
  end # End of included

  class_methods do
    def read_index_name
      @read_index_name ||= "search-#{index_name}"
    end

    def timestamped_index_name
      "#{index_name}-#{Time.current.strftime(TIMESTAMP_FORMAT)}"
    end
  end
end

class ElasticsearchSettings
  ##
  # Pre-built Dynamic Templates (Alpha Order)
  #
  module DynamicTemplates
    ##
    # Automatic generated fields for properties of type :text
    #
    TEXT = {
      strings: {
        match_mapping_type: "string",
        mapping: {
          type: :text,
          fields: {
            raw: {
              type: :keyword,
              ignore_above: 256
            }
          }
        }
      }
    }
  end
end

##
# Repository for Books
#
class BookRepository < ApplicationRepository
  include RepositorySettings

  settings default_settings.to_hash do
    mapping dynamic_templates: [
      ElasticsearchSettings::DynamicTemplates::TEXT
    ]
  end

  mappings {
    indexes :authors, type: :nested do
      indexes :first_name, type: :text
      indexes :id, type: :keyword
      indexes :last_name, type: :text

      indexes :title, type: :text do
        indexes :typeahead, type: :text, analyzer: :typeahead
      end
    end
    indexes :author_ids, type: :keyword

    indexes :description, type: :text, analyzer: :html

    indexes :images, type: :nested do
      indexes :size, type: :keyword
      indexes :url, type: :keyword, index: false, doc_values: false
    end

    indexes :language, type: :keyword
    indexes :popularity, type: :double

    indexes :producer_ids, type: :keyword
    indexes :producers, type: :nested do
      indexes :id, type: :integer
      indexes :title, type: :text do
        indexes :typeahead, type: :text, analyzer: :typeahead
      end
    end

    indexes :sku, type: :keyword

    indexes :subtitle, type: :text do
      indexes :typeahead, type: :text, analyzer: :typeahead
    end

    indexes :tags, type: :text
    indexes :title, type: :text do
      indexes :typeahead, type: :text, analyzer: :typeahead
    end
    indexes :book_suggest, type: :object do
      indexes :title, type: :completion
    end
  }

  ##
  # Serialize a book
  # @see https://github.com/elastic/elasticsearch-rails/tree/master/elasticsearch-persistence#serialization
  # @return [Hash]
  #
  def serialize(book)
    super.tap do |hash|
      suggest = {}
      suggest[:title] = { input: book.title_variants }
      hash.merge!(book_suggest: suggest)
      hash.merge!(author_ids: book.author_ids)
    end
  end
end

[skukx]

Just in case, the settings look like:

{:analysis=>
  {:filter=>{:typeahead=>{:type=>"edge_ngram", :min_gram=>2, :max_gram=>20}},
   :analyzer=>
    {:typeahead=>
      {:tokenizer=>"standard",
       :filter=>["lowercase", "typeahead"],
       :type=>"custom"},
     :path_hierarchy=>
      {:tokenizer=>:path_hierarchy, :filter=>["lowercase"], :type=>:custom},
     :html=>
      {:tokenizer=>:standard,
       :filter=>["lowercase"],
       :char_filter=>["html_strip"]}}}}

[estolfo]

Hi @skukx
I don't know what the details of ApplicationRepository or the default_settings are but my guess from the output you've provided is that older versions of the gem inferred a type (document_type) for the mapping, while 7.0.0.pre doesn't infer a type because types are deprecated. The error you're getting with the 7.0 gem connecting to a 6.8 Elasticsearch server is probably due to there being no type specific in the mapping. I would recommend trying to add document_type on the repository and seeing if that works. Let me know!
Still, I would recommend using 6.x versions of the gem if you are on Elasticsearch v6.8.

[skukx]

The reason I ask is I'd rather not have to bring down our site to upgrade to Elasticsearch 7 but I understand if that isn't possible. I'll keep digging and see if I can find what the incompatibility is.

[estolfo]

ok, I'm pretty sure it's because you don't have a inferred document_type with the 7.x gem. Can you call document_type on the repository when using a 6.x gem version and compare it with calling document_type when using 7.0.0.pre ?

[skukx]

Yeah, I think that did it. Thank you!