Buffer ucs2 and utf16le encoding is not a real alias, buffer size wrong. #4075

Closed
attilah opened this Issue Oct 2, 2012 · 3 comments

Projects

None yet

3 participants

@attilah
attilah commented Oct 2, 2012

The documentation says that ucs2 is the same as utf16le and in the source I saw the same, but it behaves differently. Here is a repro code:

var buffer = require ('buffer');

var testString = '1234567890';

var test1 = new Buffer(testString, 'ucs2');
var test2 = new Buffer(testString, 'utf-16le');

console.log ('len: ' + test1.length + ' data: ' + test1.toString ('hex'));
console.log ('len: ' + test2.length + ' data: ' + test2.toString ('hex'));

the output is this:

len: 20 data: 3100320033003400350036003700380039003000
len: 10 data: 31003200330034003500

I tested it with a node built from the latest source and the issue occurs.

@koichik koichik added a commit to koichik/node that referenced this issue Oct 2, 2012
@koichik koichik buffer: fix byteLength with UTF-16LE
Fixes #4075.
e63c62d
@koichik
koichik commented Oct 2, 2012

@attilah - Thanks for the report.

Can someone review koichik/node@e63c62d (for v0.8 branch) ?

@bnoordhuis
Member

@koichik The patch LGTM. Equating UTF-16 to UCS is perhaps not quite correct but it's irrelevant if we're only concerned with the byte length, not the character length.

@koichik koichik added a commit that closed this issue Oct 4, 2012
@koichik koichik buffer: fix byteLength with UTF-16LE
Fixes #4075.
fbb0ee6
@koichik koichik closed this in fbb0ee6 Oct 4, 2012
@koichik
koichik commented Oct 4, 2012

@bnoordhuis - Thanks, merged in fbb0ee6.

Equating UTF-16 to UCS is perhaps not quite correct

I agree, but V8 does not distinguish them.

but it's irrelevant if we're only concerned with the byte length, not the character length.

In ECMAScript (at least ES5), "character" means code unit, not code point (i.e. '\uD842\uDF9F'.length is 2, not 1). So, the character length is the same in both UCS-2 and UTF-16.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment