[WIP] proposed new unicode module

[jasnell]

The Node.js enhancement proposal nodejs/node-eps#15 proposes the addition of a new 'unicode' module that introduces a number of ICU backed APIs, including the ability to transcode a Buffer from one encoding to another. This is the prototype implementation of the new module.

It is a work in progress. This is not intended to be landed at this time.

/cc @trevnorris @addaleax @srl295

[mscdex]

Why restrict this to just utf8? What about just having general slice() and length() functions that accept an encoding name? I'm thinking this would be beneficial for other multi-byte encodings as well, such as utf16.

[jasnell]

It's just UTF8 for now, I'm going to expand this out to support each of our main encodings.

[jasnell]

that said, out of the encodings we support out of the box (utf8, utf16le, latin1, and ascii), utf8 is the only one that's difficult here. latin1 and ascii would fallback to Buffer.prototype.split and Buffer.prototype.length, and utf16le is always 2 bytes so length is always Buffer.prototype.length / 2 and split is straightforward.

Where it begins to get complicated is if we decide to support the full range of encodings offered by ICU with full-data.

[addaleax]

Nice, I’m excited! I’ve PRed with some of the changes I’m suggesting here at jasnell#10, feel free to do with those whatever you like!

[jasnell]

Awesome, thanks all. The impl here is still pretty rough and I already have some updates queued up. I'll work in the suggested changes and I'll take a look at your pr @addaleax. Much appreciated!

[targos]

does it mean it returns the same codepoint if you are at the start or in the middle of a UTF-8 character?
If that is the case, it may be good to say that the behavior is different than String.codePointAt

[targos]

The name of this module conflicts with https://www.npmjs.com/package/unicode. How do you plan to handle this ?

[trevnorris]

This doesn't operate like String#slice() or Buffer#slice(). If a negative number is passed then it should slice going backwards. There are some other minor argument coercion differences between this and how Typed Array handles arguments. Buffer should also have its argument parsing reviewed and fixed, but might as well start now. Here's an idea of how they should be parsed:

if (!start) start = 0;
// This works b/c "start < 0" calls valueOf() from start. Evaluate this
// way to be consistent with how Typed Array evaluates. For example:
//   ui8array.subarray(4.6, 5.1): 4.6 -> 4; 5.1 -> 5
//   ui8array.subarray(-3.1, -1.9): -3.1 -> -3; -1.9 -> -1
else start = start < 0 ? Math.ceil(start) : Math.floor(start);

if (end === undefined) end = buf.length;
else if (!end) end = 0;
else end = end < 0 ? Math.ceil(end) : Math.floor(end);

if (end <= start) return Buffer.alloc(0);

There's probably be a simpler way to achieve this, but intentionally wanted this to be verbose to easily see what's going on.

[jasnell]

Yeah, I've had a standing todo to go back and reconcile this. I'll hit that in the next update.

[jasnell]

My plan on this is to back off adding a new module. And instead:

a) Have separate PRs that add transcode and normalize as statics on require('buffer') ... NOT as prototype methods on Buffer but as statics on the module itself; and
b) To introduce getCharacterProperty() and getColumnWidth() in separate PRs.

[jasnell]

Closed this so I can work on breaking it up into individual PRs