Persistence implementation: ActiveRecord pattern

[karmi]

Implement an ActiveRecord-based persistence for model, oriented mainly towards Rails applications, similar to Tire::Persistence.

Using the Elasticsearch::Persistence::Repository from #71, this patch adds support for the ActiveRecord pattern of persistance for Ruby objects in Elasticsearch.

The goal is to have a 1:1 implementation to the ActiveRecord::Base implementation,
allowing to use it as a drop-in replacement for similar OxMs in Rails applications,
with minimal changes to the model definition and application code.

The model implementation uses Virtus for handling the model attributes, and ActiveModel for validations, callbacks, and similar model-related features.

Example

require 'elasticsearch/persistence/model'

class Person
  include Elasticsearch::Persistence::Model

  settings index: { number_of_shards: 1 }

  attribute :name, String,
            mapping: { fields: {
              name: { type: 'string', analyzer: 'snowball' },
              raw:  { type: 'string', analyzer: 'keyword' }
            } }

  attribute :birthday,   Date
  attribute :department, String
  attribute :salary,     Integer
  attribute :admin,      Boolean, default: false

  validates :name, presence: true

  before_save do
    puts "About to save: #{self}"
  end
end

Person.gateway.create_index! force: true

person = Person.create name: 'John Smith', salary: 10_000
# About to save: #<Person:0x007f961e89f010>
# => #<Person:0x007f961e89f010 ...>

person.id
# => "zNf3yxZDQsOTZfNfTX4E5A"

person = Person.find(person.id)
# => #<Person:0x007f961cf1f478 ... >

person.salary
# => 10000

person.increment :salary
# => { ... "_version"=>2}
person.salary
# => 10001

person.update admin: true
# => { ... "_version"=>3}

Person.search('smith').to_a
# => [#<Person:0x007f961ebc5b90 ...>]

TODO

• ActiveRecord (Model) pattern
• Integration tests
• README documentation
• Code annotation (Rubydoc)
• Example Rails application

[2xmc]

Looks great!
Is this an experiment or are you planning to merge it to the master ?

[karmi]

LOL :) No, this is not an experiment, it will be merged into master once done.

[2xmc]

Thanks, looking forward to it.

[ianneub]

Is it or will it be possible to set the elasticsearch client for each model? For example:

class Person
  include Elasticsearch::Persistence::Model

  client Elasticsearch::Client.new url: ENV['ELASTICSEARCH_PERSON_SERVER'], log: true

  settings index: { number_of_shards: 1 }

  attribute :name, String,
            mapping: { fields: {
              name: { type: 'string', analyzer: 'snowball' },
              raw:  { type: 'string', analyzer: 'keyword' }
            } }

  attribute :birthday,   Date
  attribute :department, String
  attribute :salary,     Integer
  attribute :admin,      Boolean, default: false

  validates :name, presence: true

  before_save do
    puts "About to save: #{self}"
  end
end

[karmi]

@ianneub Yeah, that's possible via the gateway:

class MyModel
  # ...
  gateway do
    client Elasticsearch::Client.new url: 'foobar'
  end
end

MyModel.search('f').to_a.first
Faraday::ConnectionFailed: getaddrinfo: nodename nor servname provided, or not known

[ianneub]

Nice! Thanks @karmi

[ianneub]

Is it possible to use Kaminari with Elasticsearch::Persistence::Model?

[karmi]

@ianneub Not yet, but it is planned.

[baronworks]

Playing around with this branch and wondering if it is possible to set the parent with a Elasticsearch::Persistence::Model and if so how?

[karmi]

@baronworks Yeah, I've got an example application in the works, I'll push it, so you can have a look.

[karmi]

I need more time to extract the application into a Rails template. In the meantime, here's how I define the mapping in the model:

class Artist
  include Elasticsearch::Persistence::Model
  # ...
end

class Album
  include Elasticsearch::Persistence::Model

  mapping _parent: { type: 'artist' } do
    indexes :suggest_title, type: 'completion', payloads: true
    indexes :suggest_track, type: 'completion', payloads: true
  end
end

I have an IndexManager class to create the index for both types:

class IndexManager
  def self.create_index(options={})
    client     = Artist.gateway.client
    index_name = Artist.index_name

    client.indices.delete index: index_name rescue nil if options[:force]

    settings = Artist.settings.to_hash.merge(Album.settings.to_hash)
    mappings = Artist.mappings.to_hash.merge(Album.mappings.to_hash)

    client.indices.create index: index_name,
                          body: {
                            settings: settings.to_hash,
                            mappings: mappings.to_hash }
  end

  # ...
end

When an Album is created, parent is passed simply as an argument (all the arguments are passed down the chain):

Album.create { title: "Foo" }, id: 'repeater', parent: 'fugazi'

[baronworks]

Thanks a lot @karmi, very much appreciated. I will be giving this a try later
this afternoon.

[xinuc]

👍 great job @karmi

[xinuc]

Can we get the ability to set the document id please?

User.create id: "karmi", name: "Karel Minarik"

[karmi]

@xinuc This was already added in 4d6aade, I've added an integration test in 783cd1a just now.

[baronworks]

The IndexManager for the mappings was the clue I needed and did manage to get my mappings all set up properly. A couple of mapping questions and\or possible issues.

Scenario 1:

document_type mapping

class Artist
    include Elasticsearch::Persistence::Model
    index_name :my_index
    document_type :my_artist
    ...
end

puts Artist.document_type = :my_artist
puts Artist.mappings.to_hash = {:artist=>{:properties=>{} }

Should the mappings key not be :my_artist and not :artist if setting the document_type?

Scenario 2:

attributes vs indexes and object type mappings

class Artist
    include Elasticsearch::Persistence::Model   

    attribute :some_map
    mapping dynamic: 'true' do
        indexes :some_map, type: 'object', default: {} 
    end 
end
puts Artist.mappings.to_hash = {:artist=>{:dynamic=>"true", :properties=>{:some_map=>{:type=>"object}}}}

Which is the mapping I and behaviour I want, but doing:

class Artist
    include Elasticsearch::Persistence::Model      
    mapping dynamic: 'true' do
        indexes :some_map, type: 'object', default: {} 
    end 
    attribute :some_map
end
puts Artist.mappings.to_hash = {:artist=>{:dynamic=>"true", :properties=>{:some_map=>{:type=>"string"}}}}

Causes the mapping for :some_map to now be :type=>"string", based on the lookup_type in Elasticsearch::Peristence::Model::Utils

So for this scenario I have 2 questions:

• 1. Will attribute: allow for type Object in the future?
• 2. Notice that when setting indexes :field that the field is not an attribute of the model until it is defined using attribute :field. Can defining an indexes: field also define it as a Virtus attribute?

Realize that this is a work in progress and apologies if jumping the gun on any functionality planned but not yet realized. Many thanks!

[xinuc]

@karmi ah, apparently I tried to set _id instead of id

Thanks.

[xinuc]

waiting for association & scoping support 😁

[karmi]

@xinuc There's very little chance there will be any DSL-ish support for associations, because that is handled quite elegantly by Virtus. Scopes can be added in the future, but it's not an immediate plan.

[xinuc]

Sorry, I'm not familiar with Virtus. But do you mean virtus can handle somethings like has_many or belongs_to ?

[karmi]

@xinuc Yes, exactly, please see the Virtus documentation. There is no belongs_to DSL, you just correctly configure the attribute -- which is the right way. There will be a full example application with Artist has many albums type of relationship published soon, as part of this repository.

[xinuc]

Great!

but I really don't understand how this works.

I have

class Conversation
  attribute :messages, Array[Message]
end

class Message
  attribute :body, String
end

my conversation doc

{
  "_index" : "messages",
  "_type" : "conversation",
  "_id" : "wGYKa0uMTKCMWV9rgGq3cw",
  "found" : false
}

and my message doc

{
  "_index" : "messages",
  "_type" : "message",
  "_id" : "BufkHKqHReyzw6eBkl60Ow",
  "_version" : 3,
  "found" : true,
  "_source":{"created_at":"2014-06-17T09:09:18.713+00:00","updated_at":"2014-06-17T09:09:21.255+00:00","body":"hello"}
}

and conversation.messages works great.

where does virtus save the association? there's no foreign key there 😕

[xinuc]

ugh, sorry,

apparently all messages saved as embedded doc inside conversation.

{
  "_index" : "messages-conversations",
  "_type" : "conversation",
  "_id" : "wGYKa0uMTKCMWV9rgGq3cw",
  "_version" : 1,
  "found" : true,
  "_source":{"created_at":"2014-06-17T09:08:55.322+00:00","updated_at":"2014-06-17T09:08:55.322+00:00","user_id":null,"partner_id":null,
  "messages":[
      {"created_at":"2014-06-17T09:09:18.713+00:00","updated_at":"2014-06-17T09:09:21.255+00:00","body":"halo","id":"BufkHKqHReyzw6eBkl60Ow"}]
      }
}

Is there any way to create a relational-like association without embedding?

[karmi]

@xinuc Have a look at Artist.rb and Album.rb, which use parent+child relationship in Elasticsearch.

[xinuc]

w00t

[twmills]

@karmi Is support for the :parent option available in 0.1.4?

I followed the example code, but the parent doesn't save on the child document.