Unicode 4 byte chars are displayed as ??

[fourplusone]

Paste any Unicode char >0xFFFF into a pad (like https://gist.github.com/3055654). It will be turned into '??'

It seems like contentcollector.jss

function sanitizeUnicode(s)
{
  return s.replace(/[\uffff\ufffe\ufeff\ufdd0-\ufdef\ud800-\udfff]/g, '?');
}

is responsible for this behavior.

However, socket.io seems to have problems too with handling chars > 0xFFFF

Any ideas how to fix this?

[JohnMcLear]

Dupe of #526

[JohnMcLear]

if you comment out the return UNorm.nfc(s).replace... then etherpad will store, accept, recieve and transmit the 𝕄 char. It looks like perhaps our replace is too strict there.

[JohnMcLear]

Updating unorm and moving lib/unorm.js to index.js the same problem persists.