Skip to content


Subversion checkout URL

You can clone with
Download ZIP
Browse files

Add alternatives document

  • Loading branch information...
commit 47b0cba0a2047eca0612b4e24a540f011cf2cac3 1 parent 8dbbf3c
@candlerb candlerb authored
Showing with 136 additions and 0 deletions.
  1. +136 −0 alternatives.markdown
136 alternatives.markdown
@@ -0,0 +1,136 @@
+Alternative approaches
+I suppose I can't criticise all this without offering some alternatives
+(other than sticking with 1.8 of course)
+1. Automatic transcoding
+There seems to me to be little benefit in having every single String in your
+program tagged with an encoding, if the main thing it does is introduce
+opportunities for Ruby to raise exceptions when these strings encounter each
+But if Ruby trancoded strings automatically as required, this might actually
+become useful.
+Consider: I'm building up a string of UTF-8 characters. Then I append a
+string of ISO-8859-1. Hey presto, it is converted to UTF-8 and appended, and
+the program continues happily. Ditto when interpolating:
+ "This error message incorporates #{str1} and #{str2}"
+where str1 and str2 could be of different encodings. They would both be
+transcoded to the source encoding of the outer string.
+* Everything is compatible with everything else, by definition.
+* If I combine str1 (encoding A) with str2 (encoding B), then str2 is
+transcoded to encoding A before the operation starts, and the result is of
+encoding A.
+* If I match str (encoding S) with regexp (encoding R), then *regexp* is
+transcoded to encoding S automatically.
+ Consider, for example, that
+ str =~ /abc/
+ would work even if str were in a wide encoding like UTF-16BE, which
+ would contain "\x00a\x00b\x00c", because the regexp has been transcoded
+ to a UTF-16BE regexp behind the scenes.
+ For efficiency, multiple encoding representations of the same regexp
+ could be stored as a cache inside the regexp object itself.
+* Have a binary regexp /./n which matches one *byte* always (whereas /./
+would match one *character* in the source string, in the source's encoding)
+* Transcoding errors could still occur, but I think these should normally
+default to substituting a ? character. If you want to raise an exception
+then use the `encode` or `encode!` methods with appropriate arguments to
+request this behaviour - or there could be a global variable to set the
+default behaviour.
+* Transcoding to or from ASCII-8BIT is a null operation, so if you are
+working with binary all you need to do is ensure one of your arguments is
+As another example:
+ s2 ="as","Aß")
+would first transcode both "as" and "Aß" to the encoding of s1, before
+running the tr method. This would therefore work even if s1 were in a wide
+encoding, and s2 would still be in a wide encoding. It would also work if s1
+were in ISO-8859-1 but the source encoding of the file were UTF-8, since
+both have a representation for "ß".
+I haven't worked this all the way through, but I believe you would end up
+with a much simpler set of rules for combining strings of different
+encodings. Encoding.compatible? would be dropped completely, and
+String#ascii_only? would become purely an optimisation (so that transcoding
+becomes a null operation in common cases)
+It would also be much easier to reason about encodings, because for an
+expression like s3 = s1 + s2 the encoding of s3 will always be the encoding
+of s1.
+This does introduce some asymmetry, but it's still got to be better than
+raising exceptions. It's quite a fundamental change though, and probably
+too late for ruby 1.9.x.
+2. Universally compatible BINARY encoding
+* Make ASCII-8BIT compatible with every other encoding, and the result is
+always ASCII-8BIT
+* A regexp match between an ASCII-8BIT string and a regexp with encoding R
+takes place as if the string had encoding R
+* Transcoding from ASCII-8BIT to any other encoding is a null operation and
+always succeeds
+This would be very easy to implement, but I consider it a band-aid only. It
+only helps people working with binary, and it adds even more rules to
+3. Separate encoding-aware class
+Allow the user to deal in raw strings of bytes that have no encoding at all
+(as opposed to an encoding of ASCII-8BIT, which still has its own rules for
+compatibility etc)
+This could have been done by leaving String as it was in 1.8, and have a new
+class which wraps a String with an Encoding: e.g.
+ str = Chars("hello", "UTF-8")
+* Chars#to_str would return just the String. Therefore any operation between
+String and Chars would succeed, and return a String.
+* String#[] and ?x would be changed to give a one-character string, as in 1.9
+* Chars#[] would then give a Chars object containing one character, and so
+on, duck-typing String but returning Chars instead.
+* File I/O could be modified to return Chars objects, but only if you
+explicitly ask for it (and it should not use the encoding from the
+environment locale unless you ask for that too)
+* Do we need a per-file Source encoding at all? If so, it could be made a
+constant like `__FILE__` and `__LINE__`:
+ str = Chars("hello", __ENCODING__)
+This approach means that encoding behaviour is optional, expandable (e.g. to
+include locale behaviour in future), and code could be made 1.8-compatible
+just by adding a few .to_str calls - whereas at the moment, code has to
+choose between calling .size or .bytesize dependent on whether it's running
+under 1.8.6 or 1.9.
+However I expect something like this was considered when 1.9 was being
+designed, and rejected as being too clumsy.
Please sign in to comment.
Something went wrong with that request. Please try again.