This repository was archived by the owner on Aug 31, 2021. It is now read-only.
🍄 [[ Builder ]] Further work on encoding/decoding module #1754
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Conflicts: engine/src/modules.cpp libscript/libscript.xcodeproj/project.pbxproj libscript/libstdscript-modules.list toolchain/lc-compile/lc-compile.xcodeproj/project.pbxproj toolchain/lc-compile/src/module-helper.cpp
…when encoding (resp. decoding) [[ LCB StdLib ]] Return undefined when replacement is unspecified and encoding or decoding fails
Remove both MCStringGetNativeChars() and MCStringGetNativeCharsWithReplacement() from the public libfoundation API.
* Return true on success/false on failure from MCStringGetNativeChars() and MCStringGetNativeCharsWithReplacement(). * Add an in parameter for size of character buffer provided and an out parameter for number of characters used/needed. * Allow multi-byte replacement sequences for characters that can't be represented in the native encoding, and add an explicit in parameter for the replacement sequence length.
Add a new function that allows more efficient conversion to ASCII, without going via the native encoding.
…acement(). Allow zero-byte and multibyte replacement sequences. Note that it is *not* an error for the replacement array to contain non-native (or non-ASCII) values. **NOTE**: This patch means that "ASCII" encoding **really is** ASCII encoding, not the native encoding discarding bytes > 127.
…ding. The "native" encoding available to script programs is rarely correct for modern systems, e.g.: * Almost all Linux systems use UTF-8, not ISO-8859-1. * All Macs use UTF-8, not MacRoman. Block all Builder programs from using "native" to specify an encoding. If someone's absolutely determined to get "native encoding exactly as in script", they can use "-native" to bypass the check.
Allow multi-character substitutions for non-ASCII chars, including allowing non-ASCII characters in the replacement.
|
Discussion with @runrevmark today (offline). About the unresolved issues: the thing that determines a decoding error is if we encounter something while decoding that we can't represent internally. That means:
A couple of additional things were brought up:
We need to decide how far we want to go with all of the above before the next release. |
peter-b
changed the title
peter-b
changed the title
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request is for continued work on encoding/decoding support for Builder, based on @livecodeali's progress in #1652.
In particular, I have:
Not done yet
Unresolved issues
There are still some unresolved issues when decoding UTF-8, UTF-16 and UTF-32.
When decoding UTF-8, we still use
UTF8ToUnicode(), which detects and ignores errors with helpful comments like:Should there be a
MCStringCreateWithUTF8CharsAndReplacement()function? My current feeling is thatMCStringCreateWithBytesAndReplacement()should be able to detect UTF-8 encoding errors and thus a replacement forUTF8ToUnicode()is needed.When decoding UTF-16 and UTF-32, what should be done with trailing bytes? Should they be ignored, should they cause a replacement string to be inserted, or should they always cause an error? @livecodeali and I have discussed it, and have had difficulty coming to a solid conclusion.
When decoding UTF-16, what should be done about unpaired surrogates? Given that we can safely round-trip an unpaired surrogate through LiveCode's internal representation, I don't think that it makes sense to consider them to be a encoding error.
What should be done about UTF-16 encoded as UTF-8? @runrevmark brought this up in the previous pull request.
Several of these issues could be solved by allowing flags to be passed to the codec functions, at the cost of making the API even more complicated.