Document best practices for ensure_index #337

Open
kylev opened this Issue Nov 2, 2011 · 19 comments

Comments

Projects
None yet
7 participants
@kylev
Contributor

kylev commented Nov 2, 2011

Right now I'm putting "ensure_index" statements right in my models, but this seems likely to cause problems (or at least a lot of no-ops). How are people using ensure_index? Do they call it from a rake task post-deploy? Put it in an initializer and hope everyone remembers to set ":background => true" or suffer blocking deploy hell?

It seems like a pretty important topic for it to be utterly absent from the docs.

@jnunemaker

This comment has been minimized.

Show comment
Hide comment
@jnunemaker

jnunemaker Nov 2, 2011

Contributor

Indexing should not be tied to your application's boot process. I do them in a rake task and run it separately whenever it is needed.

On Tuesday, November 1, 2011 at 9:06 PM, Kyle VanderBeek wrote:

Right now I'm putting "ensure_index" statements right in my models, but this seems likely to cause problems (or at least a lot of no-ops). How are people using ensure_index? Do they call it from a rake task post-deploy? Put it in an initializer and hope everyone remembers to set ":background => true" or suffer blocking deploy hell?

It seems like a pretty important topic for it to be utterly absent from the docs.

Reply to this email directly or view it on GitHub:
#337

Contributor

jnunemaker commented Nov 2, 2011

Indexing should not be tied to your application's boot process. I do them in a rake task and run it separately whenever it is needed.

On Tuesday, November 1, 2011 at 9:06 PM, Kyle VanderBeek wrote:

Right now I'm putting "ensure_index" statements right in my models, but this seems likely to cause problems (or at least a lot of no-ops). How are people using ensure_index? Do they call it from a rake task post-deploy? Put it in an initializer and hope everyone remembers to set ":background => true" or suffer blocking deploy hell?

It seems like a pretty important topic for it to be utterly absent from the docs.

Reply to this email directly or view it on GitHub:
#337

@kylev

This comment has been minimized.

Show comment
Hide comment
@kylev

kylev Nov 2, 2011

Contributor

Excellent, that's what I thought. I'll just put in this ticket, in case it gets searched for, that I am now writing a db:mongo:index rake task to do the work. It looks something like this:

namespace :db do
  namespace :mongo do

    desc "Create mongo_mapper indexes"
    task :index => :environment do
      Pants.ensure_index :color

      Outfit.ensure_index :nickname, :unique => true
      Outfit.ensure_index [[:created_at, Mongo::DESCENDING]], :background => true
    end

    desc "Remove old MongoDB indexes"
    task :deindex => :environment do
      # If you find out an index is un-needed, you should add an appropriate Model.collection.drop_index here.
      # MongoMapper doesn't yet have an inverse of ensure_index, so you have to do it via the index name and Mongo driver.
    end
  end
end

No: I do not own enough clothes to warrant MongoDB-backed tracking. It's just an example.

Contributor

kylev commented Nov 2, 2011

Excellent, that's what I thought. I'll just put in this ticket, in case it gets searched for, that I am now writing a db:mongo:index rake task to do the work. It looks something like this:

namespace :db do
  namespace :mongo do

    desc "Create mongo_mapper indexes"
    task :index => :environment do
      Pants.ensure_index :color

      Outfit.ensure_index :nickname, :unique => true
      Outfit.ensure_index [[:created_at, Mongo::DESCENDING]], :background => true
    end

    desc "Remove old MongoDB indexes"
    task :deindex => :environment do
      # If you find out an index is un-needed, you should add an appropriate Model.collection.drop_index here.
      # MongoMapper doesn't yet have an inverse of ensure_index, so you have to do it via the index name and Mongo driver.
    end
  end
end

No: I do not own enough clothes to warrant MongoDB-backed tracking. It's just an example.

@bkeepers

This comment has been minimized.

Show comment
Hide comment
@bkeepers

bkeepers Nov 2, 2011

Contributor

@kylev you want to take a crack at documenting it? You just need to check out the gh-pages branch and make the changes there.

Contributor

bkeepers commented Nov 2, 2011

@kylev you want to take a crack at documenting it? You just need to check out the gh-pages branch and make the changes there.

@hamin

This comment has been minimized.

Show comment
Hide comment
@hamin

hamin Nov 2, 2011

Contributor

@jnunemaker @bkeepers @kylev i've actually been looking at ensure_index lately too. One thing that I came across is that mongo mapper is actually NOT doing ensure_index per se the ensure_index method for a collection provided by the mongo ruby driver. The mongomapper method is using create_index under the hood. As far as mongo and the ruby driver are concerned, ensure_index and create_index are different, so for the MM method to be called ensure_index and actually just use create_index is misleading imho. @jnunemaker is there another reason why MM's ensure_index map's to the ruby driver's create_index ?

I'd be happy to send a pull-request to make sure that MM's ensure_index actually uses the ruby driver's ensure_index method under the hood. I was also thinking of mapping, these ruby driver methods to the MM indexes plugin: drop_index, drop_indexes, create_index, and index_information. I know that one can easily just drop to the ruby driver to call those methods, but it would be nice to have them in the indexes plugin. What do you guys think?

Contributor

hamin commented Nov 2, 2011

@jnunemaker @bkeepers @kylev i've actually been looking at ensure_index lately too. One thing that I came across is that mongo mapper is actually NOT doing ensure_index per se the ensure_index method for a collection provided by the mongo ruby driver. The mongomapper method is using create_index under the hood. As far as mongo and the ruby driver are concerned, ensure_index and create_index are different, so for the MM method to be called ensure_index and actually just use create_index is misleading imho. @jnunemaker is there another reason why MM's ensure_index map's to the ruby driver's create_index ?

I'd be happy to send a pull-request to make sure that MM's ensure_index actually uses the ruby driver's ensure_index method under the hood. I was also thinking of mapping, these ruby driver methods to the MM indexes plugin: drop_index, drop_indexes, create_index, and index_information. I know that one can easily just drop to the ruby driver to call those methods, but it would be nice to have them in the indexes plugin. What do you guys think?

@bkeepers

This comment has been minimized.

Show comment
Hide comment
@bkeepers

bkeepers Nov 2, 2011

Contributor

@hamin sounds reasonable. I don't know that I would map the other methods, simply because I think it's really unlikely that they'll be used, but I'm not opposed.

One idea I've been throwing around lately is a way to define syntaxes in a non-ruby syntax. I'd love to have a file like db/indexes.yml (yaml is not a great format for this, I tried) that a built-in rake task can look for and call to load the indexes. I'm thinking non-ruby so the rake task doesn't have to load all the models and everything. But if Ruby makes the most sense, then that's fine to. But basically, it'd be like db/seeds.rb.

thoughts?

Contributor

bkeepers commented Nov 2, 2011

@hamin sounds reasonable. I don't know that I would map the other methods, simply because I think it's really unlikely that they'll be used, but I'm not opposed.

One idea I've been throwing around lately is a way to define syntaxes in a non-ruby syntax. I'd love to have a file like db/indexes.yml (yaml is not a great format for this, I tried) that a built-in rake task can look for and call to load the indexes. I'm thinking non-ruby so the rake task doesn't have to load all the models and everything. But if Ruby makes the most sense, then that's fine to. But basically, it'd be like db/seeds.rb.

thoughts?

@hamin

This comment has been minimized.

Show comment
Hide comment
@hamin

hamin Nov 2, 2011

Contributor

so the reason i though the other methods would be good to have is coz i was thinking of something similiar to what you were saying, something like db/seeds.rb for indexes. I thought that if the other methods were mapped too it would make it convenient to provide rake tasks to reindex and drop_index . But then again it would have to load the models.

and just so i'm clear, you guys think its fine to have MM ensure_index call the ruby driver's ensure_index method and MM create_index method call the ruby driver's create_index method? Just feel like we need to be consistent with the driver
@bkeepers @jnunemaker thoughts?

Contributor

hamin commented Nov 2, 2011

so the reason i though the other methods would be good to have is coz i was thinking of something similiar to what you were saying, something like db/seeds.rb for indexes. I thought that if the other methods were mapped too it would make it convenient to provide rake tasks to reindex and drop_index . But then again it would have to load the models.

and just so i'm clear, you guys think its fine to have MM ensure_index call the ruby driver's ensure_index method and MM create_index method call the ruby driver's create_index method? Just feel like we need to be consistent with the driver
@bkeepers @jnunemaker thoughts?

@bkeepers

This comment has been minimized.

Show comment
Hide comment
@bkeepers

bkeepers Nov 4, 2011

Contributor

@hamin that sounds good to me.

Contributor

bkeepers commented Nov 4, 2011

@hamin that sounds good to me.

@kylev

This comment has been minimized.

Show comment
Hide comment
@kylev

kylev Nov 8, 2011

Contributor

@bkeepers Do you have a methodology for previewing the textile contents? I started on additions to the docs, but have no idea if they'll render correctly. Do you have a little textile web server or a Rakefile to generate them?

Contributor

kylev commented Nov 8, 2011

@bkeepers Do you have a methodology for previewing the textile contents? I started on additions to the docs, but have no idea if they'll render correctly. Do you have a little textile web server or a Rakefile to generate them?

@brianhempel

This comment has been minimized.

Show comment
Hide comment
@brianhempel

brianhempel Nov 8, 2011

Contributor

@kylev check out http://mongomapper.com/documentation/contributing.html I think that has the answer to your question.

Contributor

brianhempel commented Nov 8, 2011

@kylev check out http://mongomapper.com/documentation/contributing.html I think that has the answer to your question.

@bkeepers

This comment has been minimized.

Show comment
Hide comment
@bkeepers

bkeepers Nov 9, 2011

Contributor

I made the changes we talked about, along with adding a rake task. See 4529a72 and 7a8d0dc

Contributor

bkeepers commented Nov 9, 2011

I made the changes we talked about, along with adding a rake task. See 4529a72 and 7a8d0dc

@hamin

This comment has been minimized.

Show comment
Hide comment
@hamin

hamin Nov 9, 2011

Contributor

@bkeepers damn you stole my commits :) nice job, what's Forwadable? I'm assuming it just forwards those named methods to the Ruby driver ?

Contributor

hamin commented Nov 9, 2011

@bkeepers damn you stole my commits :) nice job, what's Forwadable? I'm assuming it just forwards those named methods to the Ruby driver ?

@hamin

This comment has been minimized.

Show comment
Hide comment
@hamin

hamin Nov 9, 2011

Contributor

@bkeepers feel like we should definitely update the docs for indexes to reveal that change

Contributor

hamin commented Nov 9, 2011

@bkeepers feel like we should definitely update the docs for indexes to reveal that change

@bkeepers

This comment has been minimized.

Show comment
Hide comment
@bkeepers

bkeepers Nov 9, 2011

Contributor

@hamin Sorry, I was working on some other stuff and just did it. Forwardable is like delegation in ActiveSupport.

I'll take your docs request as a volunteering to do it. :)

Contributor

bkeepers commented Nov 9, 2011

@hamin Sorry, I was working on some other stuff and just did it. Forwardable is like delegation in ActiveSupport.

I'll take your docs request as a volunteering to do it. :)

@hamin

This comment has been minimized.

Show comment
Hide comment
@hamin

hamin Nov 9, 2011

Contributor

@bkeepers you bastard :)

Contributor

hamin commented Nov 9, 2011

@bkeepers you bastard :)

@bkeepers

This comment has been minimized.

Show comment
Hide comment
@bkeepers

bkeepers Nov 9, 2011

Contributor

@hamin hey man, I don't use the docs, so I don't care about them.

Contributor

bkeepers commented Nov 9, 2011

@hamin hey man, I don't use the docs, so I don't care about them.

@hamin

This comment has been minimized.

Show comment
Hide comment
@hamin

hamin Nov 9, 2011

Contributor

@bkeepers i was just jesting...no i'd be more than happy to contribute to the docs again...i always end up learning something new when i do :)

Contributor

hamin commented Nov 9, 2011

@bkeepers i was just jesting...no i'd be more than happy to contribute to the docs again...i always end up learning something new when i do :)

@SaschaKonietzke

This comment has been minimized.

Show comment
Hide comment
@SaschaKonietzke

SaschaKonietzke Nov 23, 2011

Wouldn't it be nicer to keep the index definitions next to the keys in the models, but use a rake task to actually create the indexes?

Meaning "ensure_index :name" in a model just marks the key for indexing, and the rake task creates the actual index?

Wouldn't it be nicer to keep the index definitions next to the keys in the models, but use a rake task to actually create the indexes?

Meaning "ensure_index :name" in a model just marks the key for indexing, and the rake task creates the actual index?

@bkeepers

This comment has been minimized.

Show comment
Hide comment
@bkeepers

bkeepers Nov 23, 2011

Contributor

I could go either way. Feel free to submit a patch to make it work like you describe.

Contributor

bkeepers commented Nov 23, 2011

I could go either way. Feel free to submit a patch to make it work like you describe.

@JonKernPA

This comment has been minimized.

Show comment
Hide comment
@JonKernPA

JonKernPA Nov 26, 2011

Contributor

Thanks for the inspiration... I documented my own progression through the forest of Indexing... Pretty sure there is lots more bushwhacking left, but I did arrive at a little clearing.

http://technicaldebt.com/?p=1089

Contributor

JonKernPA commented Nov 26, 2011

Thanks for the inspiration... I documented my own progression through the forest of Indexing... Pretty sure there is lots more bushwhacking left, but I did arrive at a little clearing.

http://technicaldebt.com/?p=1089

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment