Convert a node.js utf-8 buffer w/ smp characters into a v8 string with surrogate pairs.
Switch branches/tags
Nothing to show
Latest commit bf3f72c Mar 29, 2012 @arlolra node v0.7.7
Failed to load latest commit information.
cesu-8.js use var Jan 30, 2012
license 40L Jan 31, 2012
package.json present an api Jan 30, 2012 node v0.7.7 Mar 29, 2012
test.js present an api Jan 30, 2012

#CESU-8 Encoding for Node.js

Warning: As of Node.js v0.7.7 this is no longer necessary!

###An Explanation

At the moment, Node is punting on non-BMP unicode characters. What to do?

An unsatisfactory workaround was to manually do the unwinding, as proposed here:

Which does in fact work, see:

Though it seems rather brittle. What happens if String.fromCharCode returns a { and garbles the JSON? Clearly not the answer.

So what's the problem? Why doesn't it work to begin with? Well, v8 doesn't support strict UTF-8, meaning that handing it a UTF-8 encoded buffer is never going to work, unless Javascript itself changes. (N.B. There is a strawman proposal)

But clearly something can be done 'cause the browser is working around the problem. From what I've surmised, using CESU-2 encoding, in lieu of UTF-8, will work. At least, that's what the browser appears to be doing.

To do the encoding, I looked at:

This library shows that you can take a UTF-8 encoded buffer, convert it to UTF-16 with node-iconv and then build a string with the surrogate pairs. There are a few issues with this, String.length will be one longer per non-BMP character than expected, but that's the way it should work and does in the browser. But otherwise parsing JSON, the original use case, will be fine.

This can probably be done purely in Javascript, ie. no libiconv, or even pushed down to C++, as a new encoding, CESU-8.


With everyone's favourite package manager,

npm install cesu-8

###Example Usage

var cesu = require('cesu-8')

// emoji 🍨 🍩 🍪
var arr = [
  101, 109, 111, 106, 105, 32, 240, 159, 141, 168,
  32, 240, 159, 141, 169, 32, 240, 159, 141, 170
]  // these are the octets for the above string in utf8

var utf8buffer = new Buffer(arr)

var mystring = cesu.toString(utf8buffer)

// do whatever you want with the string
// have fun
// seriously

// convert it back to a buffer
var backtobuf = cesu.toBuffer(mystring)


  • Find out what the first char is: 0xFFEF
  • Remove iconv dependency.