Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

ensure_unique_url very slow and memory consuming on large datasets #69

Closed
atomgas opened this Issue Nov 16, 2012 · 6 comments

Comments

Projects
None yet
3 participants

atomgas commented Nov 16, 2012

in https://github.com/rsl/stringex/blob/master/lib/stringex/acts_as_url.rb#L109 it does a find(:all) on the model, which in case of an empty strings or nil in url_attribute, will be all records (> 70k in my case).

I think you need a special strategy to handle empty strings or nil in url_attribute, for the other cases I would also recommend not using find(:all)

Owner

rsl commented Nov 16, 2012

already planning on using batched finds here. open to other ideas as well.

atomgas commented Nov 16, 2012

is would use SQL count on attr_name with "SOME%" this would give a good starting point,
after that I would generate the next say 100 possible results, try to find these all at once using SQL IN (possible_20, possible_21, ...) and then try to find the first not existing result with ruby, this you can repeat without to worry about preformance.

Owner

rsl commented Nov 16, 2012

got a patch idea?

Owner

rsl commented Nov 19, 2012

master uses find_each to go through smaller sets. short of a patch submission i don't have a clear idea here. feel free to submit a pull request or whatever. happy thanksgiving as well.

norbert commented Nov 23, 2012

@rsl we ran into another similar issue. Expect a patch shortly.

Owner

rsl commented Aug 27, 2013

closing this as it seems dead. reopen if needed.

@rsl rsl closed this Aug 27, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment