Don't tokenize string when counting characters

[iangreenleaf]

The length validation in ActiveModel offers a :tokenizer option, but when none is given it defaults to using // to split the string by character. This adds up to a serious performance hit when validating long strings. AFAICT it's faster, easier, and works just as well to use String#length unless a tokenizer is provided.

A simple wall-time benchmark:

class Foo
  include ActiveModel::Validations
  attr_accessor :bar
  validates_length_of :bar, :maximum => 1
end

LONG_STRING = "a" * 100000

100.times do
  f = Foo.new
  f.bar = LONG_STRING
  f.valid?
end

Before:

real  0m6.196s
user  0m6.113s
sys   0m0.077s

After:

real  0m0.720s
user  0m0.650s
sys   0m0.063s

[tolgap]

@josevalim What happens if your string contains a new line? Rails receives new lines as \r\n and it reads that as two characters. I just ran against this issue. So my choice was to call gsub!("\r\n", \n) on my attribute in the params hash in my controller to get validates_length_of() to work as expected.

What do you think?