feat(node): add string_decoder module #11095
Conversation
|
|
| * - charLength: expected number of bytes for the incomplete character | ||
| * - index: index in the buffer where the incomplete character begins | ||
| * @param {Buffer} _buffer Buffer we are checking to see if it has an incompelte "character" at the end | ||
| * @returns {IncompleteCharObject} |
There was a problem hiding this comment.
⚠️ common/Resources/ti.internal/extensions/node/string_decoder.js line 104 – Unexpected @returns tag; function has no return statement. (valid-jsdoc)
|
Fixes TIMOB-27286
sgtcoolguy
force-pushed
the
node-string-decoder
branch
from
7571c7b
to
daf7826
Compare
There was a problem hiding this comment.
Looks good! Just added two minor notes in the test suite but nothing important.
I guess we have to see if and how the caveats have any impact on third-party modules that use string_decoder. Regarding a fix for this i'd vote for the actual fix in Ti.Buffer on iOS to get the correct behavior natively, assuming that we don't make the switch to UIInt8Array for our buffers anytime soon.
| }); | ||
| }); | ||
|
|
||
| // FIXME: I'm using ES6 classes which doesn't allow non-new constructor! |
There was a problem hiding this comment.
Is this still an issue? Looks like you are just delegating to the ES6 classes so it should work fine?
| should(decoder.write(Buffer.from([]))).eql(''); | ||
| should(decoder.end()).eql(''); | ||
| // now writing an empty buffer doesn't append anything! | ||
| // testEnd('base64', Buffer.of(0x61), Buffer.of(), 'YQ=='); |
There was a problem hiding this comment.
Are these a placeholder for some missing tests?
|
Merged manually to master. |
JIRA: https://jira.appcelerator.org/browse/TIMOB-27286
Description:
Follow on to #11084 which adds Buffer
This adds the
string_decodermodule, which is sort of a safer way to interact with Buffers when dealing with character data.Basically a string decoder will temporarily cache/buffer up incomplete multi-byte characters until they're completed before outputting them. This is handy if you expect multi-byte utf8/utf16/base64 data and want to spit it out as you go.
Caveats
V8 and Node handle invalid UTF-8 multi-byte characters by replacing invalid/incomplete multi-byte sequences with '\ufffd'. This implementation does not - and on a platform like iOS can result in
undefinedbeing returned. I've left comments about this in here for now - but the "fix" here is 2-fold:The only "invalid" case we do handle is when you call
decoder.end()and it has a buffered incomplete multi-byte character.