Incorrect handling of emphasis for Japanese language

[mik-menshikov]

Emphasis markup is parsed incorrectly for Japanese language

Your environment

• OS: macOS 11.0.1
• Packages: micromark@2.10.1
• Env: yarn 1.22.10

Steps to reproduce

Input (Japanese) :

console.log(micromark("1.  **新規アプリの追加（NEW APP）**を選択します。"));

Output - incorrect:

<ol>
<li>**新規アプリの追加（NEW APP）**を選択します。</li>
</ol>

Expected behavior

Emphasis should be parsed correctly

<ol>
<li><strong>新規アプリの追加（NEW APP）</strong>を選択します。</li>
</ol>

Additional:

The same text in English and Chinese

Input (English):

console.log(micromark("1.  Select **NEW APP** (top-left corner)"));

Output - correct:

<ol>
<li>Select <strong>NEW APP</strong> (top-left corner)</li>
</ol>

Input (Chinese):

console.log(micromark("1.  选择**添加应用**（左上角）"));

Output - correct:

<ol>
<li>选择<strong>添加应用</strong>（左上角）</li>
</ol>

This bug appeared when I switched to remark-parse@9.0.0 from remark-parse@8.0.3.

The next code works correctly:

import unified from "unified";
import markdown from "remark-parse"; // 8.0.3
import rehype from "remark-rehype"; // 8.0.0
import stringify from "rehype-stringify"; // 8.0.0

unified()
    .use(markdown)
    .use(rehype)
    .use(stringify)
    .process("1.  __新規アプリの追加（NEW APP）__を選択します。", function(err, file) {
        console.log(String(file));
    });

[wooorm]

Hi @mik-menshikov!

“Output - incorrect” is what CommonMark does too: https://spec.commonmark.org/dingus/?text=1.%20%20**新規アプリの追加（NEW%20APP）**を選択します%E3%80%82. We follow CM, so this is not an issue in micromark but one in CM.

commonmark/commonmark-spec#650 seems to be the issue, and also commonmark/commonmark-spec#488 might be related.

[mik-menshikov]

@wooorm that's sad. It seems that the spec was developed without taking into account East-Asian languages. I think it should be mentioned somewhere, so people know the limitations and choose alternative solutions.

I read you story and think you have done a great work. Thanks!

[zamfofex]

@mik-menshikov, in case you are interested, there has been a variety of conversations about this on the CommonMark Discourse:

• “Single asterisks in subsequent words should not lead to emphasis” (Check the replies.)
• “Emphasis and East Asian text”
• “测试**foo**可以 works but 测试**foo()**可以 fails”
• “Bold failure in Chinese sentence, because of the usage of Chinese punctuation is different from English”