Method find_each only looping half the records in rails 3.1.0? #2832

Closed
chengguangnan opened this Issue Sep 3, 2011 · 6 comments

Comments

Projects
None yet
4 participants

I don't have this problem on rails 3.0.10, but in 3.1.0. It happens on both MySQL and SQLite.

> i = 0; Word.find(:all).each{|x| i += 1 }; puts i
84037 
> i = 0; Word.all.each{|x| i += 1 }; puts i
84037 
> i = 0; Word.find_each {|x| i += 1 }; puts i
32022
> i = 0; Word.find_each {|x| i += 1 }; puts i
32022
> i = 0; Word.find_each {|x| i += 1 }; puts i
32022

The log is strange too. Why the second batch starts from 940?

Word Load (454.4ms)  SELECT `words`.* FROM `words` WHERE (`words`.`id` >= 0) ORDER BY spell, sound, `words`.`id` ASC LIMIT 1000
Word Load (437.9ms)  SELECT `words`.* FROM `words` WHERE (`words`.`id` > 940) ORDER BY spell, sound, `words`.`id` ASC LIMIT 1000
Word Load (431.2ms)  SELECT `words`.* FROM `words` WHERE (`words`.`id` > 1852) ORDER BY spell, sound, `words`.`id` ASC LIMIT 1000
Word Load (427.6ms)  SELECT `words`.* FROM `words` WHERE (`words`.`id` > 2737) ORDER BY spell, sound, `words`.`id` ASC LIMIT 1000
Word Load (419.0ms)  SELECT `words`.* FROM `words` WHERE (`words`.`id` > 3587) ORDER BY spell, sound, `words`.`id` ASC LIMIT 1000
Member

arunagw commented Sep 3, 2011

@guangnan Can you paste quires which are logging into log file here? it works fine for me

ruby-1.9.3-preview1 :012 > i = 0; Blog.find(:all).each{|x| i += 1 }; puts i
  Blog Load (75.5ms)  SELECT "blogs".* FROM "blogs" 
23000
 => nil 
ruby-1.9.3-preview1 :013 > i = 0; Blog.find_each{|x| i += 1 }; puts i
  Blog Load (2.5ms)  SELECT "blogs".* FROM "blogs" WHERE ("blogs"."id" >= 0) ORDER BY "blogs"."id" ASC LIMIT 1000
  Blog Load (2.3ms)  SELECT "blogs".* FROM "blogs" WHERE ("blogs"."id" > 1000) ORDER BY "blogs"."id" ASC LIMIT 1000
  Blog Load (2.6ms)  SELECT "blogs".* FROM "blogs" WHERE ("blogs"."id" > 2000) ORDER BY "blogs"."id" ASC LIMIT 1000
  Blog Load (2.3ms)  SELECT "blogs".* FROM "blogs" WHERE ("blogs"."id" > 3000) ORDER BY "blogs"."id" ASC LIMIT 1000
  Blog Load (2.3ms)  SELECT "blogs".* FROM "blogs" WHERE ("blogs"."id" > 4000) ORDER BY "blogs"."id" ASC LIMIT 1000
  Blog Load (2.3ms)  SELECT "blogs".* FROM "blogs" WHERE ("blogs"."id" > 5000) ORDER BY "blogs"."id" ASC LIMIT 1000
  Blog Load (2.3ms)  SELECT "blogs".* FROM "blogs" WHERE ("blogs"."id" > 6000) ORDER BY "blogs"."id" ASC LIMIT 1000
  Blog Load (2.2ms)  SELECT "blogs".* FROM "blogs" WHERE ("blogs"."id" > 7000) ORDER BY "blogs"."id" ASC LIMIT 1000
  Blog Load (2.4ms)  SELECT "blogs".* FROM "blogs" WHERE ("blogs"."id" > 8000) ORDER BY "blogs"."id" ASC LIMIT 1000
  Blog Load (2.2ms)  SELECT "blogs".* FROM "blogs" WHERE ("blogs"."id" > 9000) ORDER BY "blogs"."id" ASC LIMIT 1000
  Blog Load (2.4ms)  SELECT "blogs".* FROM "blogs" WHERE ("blogs"."id" > 10000) ORDER BY "blogs"."id" ASC LIMIT 1000
  Blog Load (2.4ms)  SELECT "blogs".* FROM "blogs" WHERE ("blogs"."id" > 11000) ORDER BY "blogs"."id" ASC LIMIT 1000
  Blog Load (2.4ms)  SELECT "blogs".* FROM "blogs" WHERE ("blogs"."id" > 12000) ORDER BY "blogs"."id" ASC LIMIT 1000
  Blog Load (2.7ms)  SELECT "blogs".* FROM "blogs" WHERE ("blogs"."id" > 13000) ORDER BY "blogs"."id" ASC LIMIT 1000
  Blog Load (2.3ms)  SELECT "blogs".* FROM "blogs" WHERE ("blogs"."id" > 14000) ORDER BY "blogs"."id" ASC LIMIT 1000
  Blog Load (2.6ms)  SELECT "blogs".* FROM "blogs" WHERE ("blogs"."id" > 15000) ORDER BY "blogs"."id" ASC LIMIT 1000
  Blog Load (2.5ms)  SELECT "blogs".* FROM "blogs" WHERE ("blogs"."id" > 16000) ORDER BY "blogs"."id" ASC LIMIT 1000
  Blog Load (2.4ms)  SELECT "blogs".* FROM "blogs" WHERE ("blogs"."id" > 17000) ORDER BY "blogs"."id" ASC LIMIT 1000
  Blog Load (2.4ms)  SELECT "blogs".* FROM "blogs" WHERE ("blogs"."id" > 18000) ORDER BY "blogs"."id" ASC LIMIT 1000
  Blog Load (2.3ms)  SELECT "blogs".* FROM "blogs" WHERE ("blogs"."id" > 19000) ORDER BY "blogs"."id" ASC LIMIT 1000
  Blog Load (2.2ms)  SELECT "blogs".* FROM "blogs" WHERE ("blogs"."id" > 20000) ORDER BY "blogs"."id" ASC LIMIT 1000
  Blog Load (2.2ms)  SELECT "blogs".* FROM "blogs" WHERE ("blogs"."id" > 21000) ORDER BY "blogs"."id" ASC LIMIT 1000
  Blog Load (2.8ms)  SELECT "blogs".* FROM "blogs" WHERE ("blogs"."id" > 22000) ORDER BY "blogs"."id" ASC LIMIT 1000
  Blog Load (0.2ms)  SELECT "blogs".* FROM "blogs" WHERE ("blogs"."id" > 23000) ORDER BY "blogs"."id" ASC LIMIT 1000
23000
 => nil 

Contributor

Mik-die commented Sep 3, 2011

@guangnan, do you have some default scope?

There should not be 'ORDER BY spell, sound, words.id', but only 'ORDER BY words.id'

Member

arunagw commented Sep 3, 2011

@guangnan Opps i didn't see that time logs.

@Mik-die, yes,

class Word < ActiveRecord::Base
  default_scope :order => 'spell, sound'
end

In console:

ruby-1.9.2-p290 :007 > i = 0; Word.find_each{|x| i += 1 }; puts i
Scoped order and limit are ignored, it's forced to be batch order and batch size
  Word Load (454.4ms)  SELECT `words`.* FROM `words` WHERE (`words`.`id` >= 0) ORDER BY spell, sound, `words`.`id` ASC LIMIT 1000

So version 3.1.0 seems failed to force the order, right? Because I don't have this issue in 3.0.10.

@arunagw arunagw added a commit to arunagw/rails that referenced this issue Sep 4, 2011

@arunagw arunagw We need to recorder here. Need to drop the order from default scope.
Fixes #2832
9066e34
Member

arunagw commented Sep 4, 2011

@guangnan can you give a try with 3-1-stable branch now?

@arunagw, it seems fixed in 3-1-stable. Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment