Use Ruby's native Encoding functionality to implement tidy_bytes

[burke]

The current implementation is quite slow. This leverages some of the transcoding abilities built into Ruby 1.9 instead. It is roughly 96% faster.

The roundtrip through UTF_8_MAC here is because ruby won't let you transcode from UTF_8 to UTF_8. I chose the closest encoding I could find as an intermediate.

These test cases seem quite thorough and it passes all of them: https://github.com/rails/rails/blob/master/activesupport/test/multibyte_chars_test.rb#L630-L661

Benchmarking

require 'benchmark'

kilobyte = "1" * 1024

u = ActiveSupport::Multibyte::Unicode

res = Benchmark.realtime do
  10_000.times do
    u.tidy_bytes(kilobyte)
  end
end

puts "tidy_bytes took #{res / 10}ms"

Save as ./activesupport/bm.rb, cd to ./activesupport, and run:

ruby -I./lib -ractive_support/multibyte/unicode bm.rb

Output before proposed change:

tidy_bytes took 1.952ms

After proposed change:

tidy_bytes took 0.0668ms

[jeremy]

Nice, @burke. We do this in our apps, but round-trip through UTF-16.

Check out the new String#scrub in Ruby 2.1 (trunk) that does this directly, without an intermediate encode step!

[burke]

I tried UTF-16 at first but I ran into some weirdness with BOMs, so I switched to a different encoding rather than debugging it :). I didn't benchmark thoroughly but I remember finding this way to be a bit faster as well.

I hadn't heard about String#scrub -- good news! Finally, tools to deal with encodings sanely are starting to show up.

[jeremy]

May as well preserve the existing object: return string if string.empty?

[burke]

We just put this in production. It seems to be working well, except that the final line fails if the input string was UTF-16. On further investigation, so does the current implementation :)

[jeremy]

Could you rebase against master and squash to a single commit?

[burke]

Sure. Done.