ensure_unique_url very slow and memory consuming on large datasets

[atomgas]

in https://github.com/rsl/stringex/blob/master/lib/stringex/acts_as_url.rb#L109 it does a find(:all) on the model, which in case of an empty strings or nil in url_attribute, will be all records (> 70k in my case).

I think you need a special strategy to handle empty strings or nil in url_attribute, for the other cases I would also recommend not using find(:all)

[rsl]

already planning on using batched finds here. open to other ideas as well.

[atomgas]

is would use SQL count on attr_name with "SOME%" this would give a good starting point,
after that I would generate the next say 100 possible results, try to find these all at once using SQL IN (possible_20, possible_21, ...) and then try to find the first not existing result with ruby, this you can repeat without to worry about preformance.

[rsl]

got a patch idea?

[rsl]

master uses find_each to go through smaller sets. short of a patch submission i don't have a clear idea here. feel free to submit a pull request or whatever. happy thanksgiving as well.

[norbert]

@rsl we ran into another similar issue. Expect a patch shortly.

[rsl]

closing this as it seems dead. reopen if needed.