Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: decode string with specified buffer size #19

Merged
merged 2 commits into from
Jan 6, 2023

Conversation

toyobayashi
Copy link
Owner

Fixes #17

@toyobayashi toyobayashi merged commit 458121b into main Jan 6, 2023
@RReverser
Copy link
Contributor

This won't work correctly with pthreads, because TextDecoder works only on unshared memory. Note how Emscripten has to work around that with custom helper that switches to .slice(...) instead of .subarray(...) when memory is shared: https://github.com/emscripten-core/emscripten/blob/0d7014e1b20ead9739f1fce54b316152413601c3/src/runtime_strings.js#L115

abort('FATAL ERROR: ' +
(location_len === -1 ? UTF8ToString(location) : emnapiRt.utf8Decoder.decode(HEAPU8.subarray(location, location + location_len))) +
' ' +
(message_len === -1 ? UTF8ToString(message) : emnapiRt.utf8Decoder.decode(HEAPU8.subarray(message, message + message_len)))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe better to extract this into a helper emnapiRt.readString? Could be a bit cleaner than duplicating === -1 ? ... everywhere.

export const utf8Decoder: { decode: (input: BufferSource) => string } = typeof TextDecoder === 'function'
? new TextDecoder()
: {
decode (input: BufferSource) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd suggest to either exclude the fallback, or at least guard it under #if TEXTDECODER == 2 etc like Emscripten does.

It's quite a bit of code for something that none of the modern engines needs.

throw new TypeError('The "input" argument must be an instance of ArrayBuffer or ArrayBufferView')
}
const bytes = isArrayBuffer ? new Uint16Array(input) : new Uint16Array(input.buffer, input.byteOffset, input.byteLength / 2)
return String.fromCharCode.apply(String, bytes as any)
Copy link
Contributor

@RReverser RReverser Jan 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is often tempting, but will fail with stack overflow on any large strings (probably worth adding a test for that). It's important to use a loop here instead.

@toyobayashi
Copy link
Owner Author

Thanks. working in progress

@toyobayashi
Copy link
Owner Author

Done. c8f1175

@RReverser
Copy link
Contributor

Looks good. I think it would be still worth adding regression tests for 2 mentioned cases - one to check that strings work when compiled with -pthread, and one for large strings.

@toyobayashi toyobayashi deleted the fix-string-decoding branch April 6, 2023 16:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

napi_create_string_utf8 should preserve null terminator
2 participants