Fix mojibake for non-ASCII text on lines with wiki markup - #20
Merged
Conversation
secondPass's tokenized-line loop wrote unmatched bytes with out.WriteRune(rune(line[beg])), treating each raw UTF-8 byte as its own code point and advancing beg by one byte at a time. Any multibyte character (Greek, Cyrillic, CJK, accented Latin, emoji) on a line containing wiki markup like *bold* or a heading got corrupted into mojibake, since only lines with at least one token go through this byte-wise fallback. Decode a full rune at the current byte offset with utf8.DecodeRuneInString and advance by its byte width instead.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Jira wiki-to-markdown parser corrupted non-ASCII characters (Greek, Cyrillic, CJK, accented Latin, emoji, etc.) whenever they appeared on a line that also contained wiki markup such as bold, a heading, or a list marker.
In secondPass's tokenized-line loop, the fallback branch for bytes that don't start a recognized token wrote
out.WriteRune(rune(line[beg]))while advancing one byte at a time. This treats every raw UTF-8 byte as its own Unicode code point and re-encodes it, mangling any multibyte sequence. Lines without any markup skip this loop entirely (they're written verbatim), which is why the bug only showed up on lines mixing wiki syntax with non-ASCII text.The fix decodes a full rune at the current byte offset with
utf8.DecodeRuneInStringand advances by its actual byte width instead of a fixed one byte, so multibyte characters pass through intact regardless of what markup surrounds them on the line.Added a test case mixing bold markup with Greek, Cyrillic, and Japanese text to lock in the fix.
Addresses ankitpokhrel#1003.