Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add ActiveSupport::Multibyte. Provides String#chars which lets you de…
…al with strings as a sequence of chars, not of bytes. Closes #6242 [Julian Tarkhanov, Manfred Stienstra & Jan Behrens] git-svn-id: http://svn-commit.rubyonrails.org/rails/trunk@5223 5ecf4fe2-1ee6-0310-87b1-e25e094e27de
- Loading branch information
Showing
15 changed files
with
1,390 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -40,3 +40,5 @@ | |
require 'active_support/values/time_zone' | ||
|
||
require 'active_support/json' | ||
|
||
require 'active_support/multibyte' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
42 changes: 42 additions & 0 deletions
42
activesupport/lib/active_support/core_ext/string/unicode.rb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,42 @@ | ||
module ActiveSupport #:nodoc: | ||
module CoreExtensions #:nodoc: | ||
module String #:nodoc: | ||
# Define methods for handeling unicode data. | ||
module Unicode | ||
# +chars+ is a Unicode safe proxy for string methods. It creates and returns an instance of the | ||
# ActiveSupport::Multibyte::Chars class which encapsulates the original string. A Unicode safe version of all | ||
# the String methods are defined on this proxy class. Undefined methods are forwarded to String, so all of the | ||
# string overrides can also be called through the +chars+ proxy. | ||
# | ||
# name = 'Claus Müller' | ||
# name.reverse #=> "rell??M sualC" | ||
# name.length #=> 13 | ||
# | ||
# name.chars.reverse.to_s #=> "rellüM sualC" | ||
# name.chars.length #=> 12 | ||
# | ||
# | ||
# All the methods on the chars proxy which normally return a string will return a Chars object. This allows | ||
# method chaining on the result of any of these methods. | ||
# | ||
# name.chars.reverse.length #=> 12 | ||
# | ||
# The Char object tries to be as interchangeable with String objects as possible: sorting and comparing between | ||
# String and Char work like expected. The bang! methods change the internal string representation in the Chars | ||
# object. Interoperability problems can be resolved easily with a +to_s+ call. | ||
# | ||
# For more information about the methods defined on the Chars proxy see ActiveSupport::Multibyte::Chars and | ||
# ActiveSupport::Multibyte::Handlers::UTF8Handler | ||
def chars | ||
ActiveSupport::Multibyte::Chars.new(self) | ||
end | ||
|
||
# Returns true if the string has UTF-8 semantics (a String used for purely byte resources is unlikely to have | ||
# them), returns false otherwise. | ||
def is_utf8? | ||
ActiveSupport::Multibyte::Handlers::UTF8Handler.consumes?(self) | ||
end | ||
end | ||
end | ||
end | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
module ActiveSupport::Multibyte | ||
DEFAULT_NORMALIZATION_FORM = :kc | ||
NORMALIZATIONS_FORMS = [:c, :kc, :d, :kd] | ||
UNICODE_VERSION = '5.0.0' | ||
end | ||
|
||
require 'active_support/multibyte/chars' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,129 @@ | ||
require 'active_support/multibyte/handlers/utf8_handler' | ||
require 'active_support/multibyte/handlers/passthru_handler' | ||
|
||
# Encapsulates all the functionality related to the Chars proxy. | ||
module ActiveSupport::Multibyte | ||
# Chars enables you to work transparently with multibyte encodings in the Ruby String class without having extensive | ||
# knowledge about the encoding. A Chars object accepts a string upon initialization and proxies String methods in an | ||
# encoding safe manner. All the normal String methods are also implemented on the proxy. | ||
# | ||
# String methods are proxied through the Chars object, and can be accessed through the +chars+ method. Methods | ||
# which would normally return a String object now return a Chars object so methods can be chained. | ||
# | ||
# "The Perfect String ".chars.downcase.strip.normalize #=> "the perfect string" | ||
# | ||
# Chars objects are perfectly interchangeable with String objects as long as no explicit class checks are made. | ||
# If certain methods do explicitly check the class, call +to_s+ before you pass chars objects to them. | ||
# | ||
# bad.explicit_checking_method "T".chars.downcase.to_s | ||
# | ||
# The actual operations on the string are delegated to handlers. Theoretically handlers can be implemented for | ||
# any encoding, but the default handler handles UTF-8. This handler is set during initialization, if you want to | ||
# use you own handler, you can set it on the Chars class. Look at the UTF8Handler source for an example how to | ||
# implement your own handler. If you your own handler to work on anything but UTF-8 you probably also | ||
# want to override Chars#handler. | ||
# | ||
# ActiveSupport::Multibyte::Chars.handler = MyHandler | ||
# | ||
# Note that a few methods are defined on Chars instead of the handler because they are defined on Object or Kernel | ||
# and method_missing can't catch them. | ||
class Chars | ||
|
||
attr_reader :string # The contained string | ||
alias_method :to_s, :string | ||
|
||
include Comparable | ||
|
||
# The magic method to make String and Chars comparable | ||
def to_str | ||
# Using any other ways of overriding the String itself will lead you all the way from infinite loops to | ||
# core dumps. Don't go there. | ||
@string | ||
end | ||
|
||
# Create a new Chars instance. | ||
def initialize(str) | ||
@string = (str.string rescue str) | ||
end | ||
|
||
# Returns -1, 0 or +1 depending on whether the Chars object is to be sorted before, equal or after the | ||
# object on the right side of the operation. It accepts any object that implements +to_s+. See String.<=> | ||
# for more details. | ||
def <=>(other); @string <=> other.to_s; end | ||
|
||
# Works just like String#split, with the exception that the items in the resulting list are Chars | ||
# instances instead of String. This makes chaining methods easier. | ||
def split(*args) | ||
@string.split(*args).map { |i| i.chars } | ||
end | ||
|
||
# Gsub works exactly the same as gsub on a normal string. | ||
def gsub(*a, &b); @string.gsub(*a, &b).chars; end | ||
|
||
# Like String.=~ only it returns the character offset (in codepoints) instead of the byte offset. | ||
def =~(other) | ||
handler.translate_offset(@string, @string =~ other) | ||
end | ||
|
||
# Try to forward all undefined methods to the handler, when a method is not defined on the handler, send it to | ||
# the contained string. Method_missing is also responsible for making the bang! methods destructive. | ||
def method_missing(m, *a, &b) | ||
begin | ||
# Simulate methods with a ! at the end because we can't touch the enclosed string from the handlers. | ||
if m.to_s =~ /^(.*)\!$/ | ||
result = handler.send($1, @string, *a, &b) | ||
if result == @string | ||
result = nil | ||
else | ||
@string.replace result | ||
end | ||
else | ||
result = handler.send(m, @string, *a, &b) | ||
end | ||
rescue NoMethodError | ||
result = @string.send(m, *a, &b) | ||
rescue Handlers::EncodingError | ||
@string.replace handler.tidy_bytes(@string) | ||
retry | ||
end | ||
|
||
if result.kind_of?(String) | ||
result.chars | ||
else | ||
result | ||
end | ||
end | ||
|
||
# Set the handler class for the Char objects. | ||
def self.handler=(klass) | ||
@@handler = klass | ||
end | ||
|
||
# Returns the proper handler for the contained string depending on $KCODE and the encoding of the string. This | ||
# method is used internally to always redirect messages to the proper classes depending on the context. | ||
def handler | ||
if utf8_pragma? | ||
@@handler | ||
else | ||
ActiveSupport::Multibyte::Handlers::PassthruHandler | ||
end | ||
end | ||
|
||
private | ||
|
||
# +utf8_pragma+ checks if it can send this string to the handlers. It makes sure @string isn't nil and $KCODE is | ||
# set to 'UTF8'. | ||
def utf8_pragma? | ||
!@string.nil? && ($KCODE == 'UTF8') | ||
end | ||
end | ||
end | ||
|
||
# When we can load the utf8proc library, override normalization with the faster methods | ||
begin | ||
require 'utf8proc_native' | ||
require 'active_support/multibyte/handlers/utf8_handler_proc' | ||
ActiveSupport::Multibyte::Chars.handler = ActiveSupport::Multibyte::Handlers::UTF8HandlerProc | ||
rescue LoadError | ||
ActiveSupport::Multibyte::Chars.handler = ActiveSupport::Multibyte::Handlers::UTF8Handler | ||
end |
141 changes: 141 additions & 0 deletions
141
activesupport/lib/active_support/multibyte/generators/generate_tables.rb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,141 @@ | ||
#!/usr/bin/env ruby | ||
begin | ||
require File.dirname(__FILE__) + '/../../../active_support' | ||
rescue IOError | ||
end | ||
require 'open-uri' | ||
require 'tmpdir' | ||
|
||
module ActiveSupport::Multibyte::Handlers #:nodoc: | ||
class UnicodeDatabase #:nodoc: | ||
def self.load | ||
[Hash.new(Codepoint.new),[],{},{}] | ||
end | ||
end | ||
|
||
class UnicodeTableGenerator #:nodoc: | ||
BASE_URI = "http://www.unicode.org/Public/#{ActiveSupport::Multibyte::UNICODE_VERSION}/ucd/" | ||
SOURCES = { | ||
:codepoints => BASE_URI + 'UnicodeData.txt', | ||
:composition_exclusion => BASE_URI + 'CompositionExclusions.txt', | ||
:grapheme_break_property => BASE_URI + 'auxiliary/GraphemeBreakProperty.txt' | ||
} | ||
|
||
def initialize | ||
@ucd = UnicodeDatabase.new | ||
|
||
default = Codepoint.new | ||
default.combining_class = 0 | ||
default.uppercase_mapping = 0 | ||
default.lowercase_mapping = 0 | ||
@ucd.codepoints = Hash.new(default) | ||
|
||
@ucd.composition_exclusion = [] | ||
@ucd.composition_map = {} | ||
@ucd.boundary = {} | ||
end | ||
|
||
def parse_codepoints(line) | ||
codepoint = Codepoint.new | ||
raise "Could not parse input." unless line =~ /^ | ||
([0-9A-F]+); # code | ||
([^;]+); # name | ||
([A-Z]+); # general category | ||
([0-9]+); # canonical combining class | ||
([A-Z]+); # bidi class | ||
(<([A-Z]*)>)? # decomposition type | ||
((\ ?[0-9A-F]+)*); # decompomposition mapping | ||
([0-9]*); # decimal digit | ||
([0-9]*); # digit | ||
([^;]*); # numeric | ||
([YN]*); # bidi mirrored | ||
([^;]*); # unicode 1.0 name | ||
([^;]*); # iso comment | ||
([0-9A-F]*); # simple uppercase mapping | ||
([0-9A-F]*); # simple lowercase mapping | ||
([0-9A-F]*)$/ix # simple titlecase mapping | ||
codepoint.code = $1.hex | ||
#codepoint.name = $2 | ||
#codepoint.category = $3 | ||
codepoint.combining_class = Integer($4) | ||
#codepoint.bidi_class = $5 | ||
codepoint.decomp_type = $7 | ||
codepoint.decomp_mapping = ($8=='') ? nil : $8.split.collect { |element| element.hex } | ||
#codepoint.bidi_mirrored = ($13=='Y') ? true : false | ||
codepoint.uppercase_mapping = ($16=='') ? 0 : $16.hex | ||
codepoint.lowercase_mapping = ($17=='') ? 0 : $17.hex | ||
#codepoint.titlecase_mapping = ($18=='') ? nil : $18.hex | ||
@ucd.codepoints[codepoint.code] = codepoint | ||
end | ||
|
||
def parse_grapheme_break_property(line) | ||
if line =~ /^([0-9A-F\.]+)\s*;\s*([\w]+)\s*#/ | ||
type = $2.downcase.intern | ||
@ucd.boundary[type] ||= [] | ||
if $1.include? '..' | ||
parts = $1.split '..' | ||
@ucd.boundary[type] << (parts[0].hex..parts[1].hex) | ||
else | ||
@ucd.boundary[type] << $1.hex | ||
end | ||
end | ||
end | ||
|
||
def parse_composition_exclusion(line) | ||
if line =~ /^([0-9A-F]+)/i | ||
@ucd.composition_exclusion << $1.hex | ||
end | ||
end | ||
|
||
def create_composition_map | ||
@ucd.codepoints.each do |_, cp| | ||
if !cp.nil? and cp.combining_class == 0 and cp.decomp_type.nil? and !cp.decomp_mapping.nil? and cp.decomp_mapping.length == 2 and @ucd[cp.decomp_mapping[0]].combining_class == 0 and !@ucd.composition_exclusion.include?(cp.code) | ||
@ucd.composition_map[cp.decomp_mapping[0]] ||= {} | ||
@ucd.composition_map[cp.decomp_mapping[0]][cp.decomp_mapping[1]] = cp.code | ||
end | ||
end | ||
end | ||
|
||
def normalize_boundary_map | ||
@ucd.boundary.each do |k,v| | ||
if [:lf, :cr].include? k | ||
@ucd.boundary[k] = v[0] | ||
end | ||
end | ||
end | ||
|
||
def parse | ||
SOURCES.each do |type, url| | ||
filename = File.join(Dir.tmpdir, "#{url.split('/').last}") | ||
unless File.exist?(filename) | ||
$stderr.puts "Downloading #{url.split('/').last}" | ||
File.open(filename, 'wb') do |target| | ||
open(url) do |source| | ||
source.each_line { |line| target.write line } | ||
end | ||
end | ||
end | ||
File.open(filename) do |file| | ||
file.each_line { |line| send "parse_#{type}".intern, line } | ||
end | ||
end | ||
create_composition_map | ||
normalize_boundary_map | ||
end | ||
|
||
def dump_to(filename) | ||
File.open(filename, 'wb') do |f| | ||
f.write Marshal.dump([@ucd.codepoints, @ucd.composition_exclusion, @ucd.composition_map, @ucd.boundary]) | ||
end | ||
end | ||
end | ||
end | ||
|
||
if __FILE__ == $0 | ||
filename = ActiveSupport::Multibyte::Handlers::UnicodeDatabase.filename | ||
generator = ActiveSupport::Multibyte::Handlers::UnicodeTableGenerator.new | ||
generator.parse | ||
print "Writing to: #{filename}" | ||
generator.dump_to filename | ||
puts " (#{File.size(filename)} bytes)" | ||
end |
9 changes: 9 additions & 0 deletions
9
activesupport/lib/active_support/multibyte/handlers/passthru_handler.rb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
# Chars uses this handler when $KCODE is not set to 'UTF8'. Because this handler doesn't define any methods all call | ||
# will be forwarded to String. | ||
class ActiveSupport::Multibyte::Handlers::PassthruHandler | ||
|
||
# Return the original byteoffset | ||
def self.translate_offset(string, byte_offset) #:nodoc: | ||
byte_offset | ||
end | ||
end |
Oops, something went wrong.