RFC Allow optional strict decoding of windows-1251/1252 (new ops) #798
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Not entirely keen on the I think I'd prefer it slightly if the default were to be strict and we had a flag that made it loose. One other thought: it's tiresome to implement every decoder twice, once streaming and once not. I wonder if we can do everything in terms of the streaming (not at the op level, but at the function level). |
We have new versions of `encode`, `decode`, and `encoderep` that allow a new config flag. These ops differ from the previous encode/decode ops in that by default they decode using strict rather than permissive methods. They throw on encountering a codepoint that is not mapped in the standard and optionally will decode permissively. These new ops all end with *config. While all other ops are `config` variants of current ops, `decoderep` never existed before, so `decoderepconfig` has been added to allow decoding with replacements By default with windows-1252 and windows-1251 these new ops: * Throw when they encounter a character which does not map to the other encoding. * When used under replacement mode (*repconfig), this causes replacements to be done with codepoints that may fit into the target encoding but are invalid (i.e. 129 in windows-1252). This adds new ops: * `decoderepconfig`: Strict and replaces decoded characters that don't have official mappings with a supplied replacement string. Currently it is limited to substituting the first grapheme of the supplied replacement string (should be useful in most cases). * `encodeconfig`: like `encode` but strict by default and new config flag * `decodeconfig`: like `decode` but strict by default and new config flag * `encoderepconfig`: like `encoderep` but strict by default and new config flag
samcv
force-pushed
the
encode-decode-options
branch
from
3549ce5
to
9c948d2
Compare
|
@jnthn I've updated it so that all the ops end with |
|
This was added in to master. Closing. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is an RFC for comment on these changes. The op names need to be
changed, so name suggestions are appreciated. Also I would like comment
on whether I should just change the arguments for any function which
is not used in a MoarVM OP, instead of having redundant functions,
would be appreciated.
We have new versions of
encode,decode, andencoderepthat allowthe new config flag.
decoderepnever existed before, so this featurewas added to the decoder functions and a function to decode supporting
replacements and a flag was added (
decode2rep)Flags for these op currently have only one setting:
Config flag 1:
other encoding.
replacements to be done with codepoints that may fit into the target
encoding but are invalid (i.e. 129 in windows-1252).
This adds new ops (names will likely be changed)
decode2rep: we never had adecoderepop. This op replaces decodedcharacters that don't have official mappings with a supplied
replacement string. Currently it is limited to replacement of strings
with only one grapheme.
encode2: likeencodeexcept you can set an additional config flagdecode2: likedecodeexcept you can set an additional config flagencoderep2: likeencoderepexcept you can set an additional configflag