-
-
Notifications
You must be signed in to change notification settings - Fork 313
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Emphasis with CJK punctuation #650
Comments
This and the above issues are caused by the change in #618. It is mixed in only v0.30 spec. https://spec.commonmark.org/0.30/changes
The definition of left- and -right-franking emphasis for * and ** must use ASCII punctuation characters instead of Unicode ones. does not cause such problem, so remark depended by MDX v2+ is affected. |
Again, there is no change in 618. That PR is just about words, terminology. MDX 1 did not follow CM correctly and had other bugs. Can you please read what I say, and please stop spamming, and actually contribute? |
The extension by MDX is not the culprit. https://codesandbox.io/s/remark-playground-wmfor?file=/package.json As of Not reproduced in The latest Prettier (uses
This means that the credit for the change goes to the fact that it turns to be clear that this specification is a terrible one that should be revised. Old |
https://spec.commonmark.org/0.29/
You are right. I'm sorry. I will look for another version. |
I finally found that the current broken definition sentences were introduced in 0.14. https://spec.commonmark.org/0.14/changes https://spec.commonmark.org/0.13/ I will investigate why these are introduced. |
https://github.com/commonmark/commonmark-spec/blob/0.14/changelog.spec.txt
http://talk.commonmark.org/t/903/6
Note I replaced the link with a cache by the Wayback machine. I conclude that this problem was caused by a lack of consideration for Chinese and Japanese by |
I would like to ask them why they included non-ASCII punctuation characters and why only ASCII punctuation characters are not sufficient. |
I will blame https://github.com/vfmd/vfmd-spec/blob/gh-pages/specification.md later. The test cases in vfmd considered only ASCII punctuation. |
I found the commit containing the initial definition in the spec of vfmd:
|
@tats-u dude, here and in your comments on #618 you come off as arrogant and very disrespectful. You make absolutist claims and then frequently correcting yourself because it turns out you didn't do your homework. You need to have the humility to realize that your perception that "something broke or is broken" might have to do with you not understanding one or more of the following (I don't have the time to figure out which ones, the responsibility is on you):
A more reasoned, respectful and helpful approach would be to have a discussion with other people who are affected by what you claim is broken, including the makers and other users of the downstream tool that you claim is now broken. Diagnose the problem with them, assuming they agree with you that there is a problem, before making a claim that the source of the problem is upstream in CommonMark. If it turns out that you are alone in this, that should tell you something. |
@tats-u This issue is still open, so indeed it is looking for a solution. It is also something I have heard from others. However, it is not easy to solve. There are also legitimate cases where you do want to use an asterisk or underscore but don’t want it to result in emphasis/strong. Also in East-Asian languages. One idea I have, that could potentially help emphasis/strong, is the Unicode line breaking algorithm: https://unicode.org/reports/tr14/. |
@vassudanagunta I had got too much angry at that time. I do think it was over the limit now.
Let me say there are never in each framework. This problem can be reproduced in the most major JS Markdown frameworks, remark (unified) and markdown-it. Remark-related issues that I have raised are closed immediately with the reason that they are on spec.
I never have. This is why I have looked into the background and the impact of my proposed changes now.
It looks like a lot of work to study the impact of breaking changes and decide whether or not to apply them.
Due to this problem, it became necessary for me (us) to tell all Japanese (and some Chinese) Markdown writers to refrain from surrounding whole sentences with <!-- What would you feel if Markdown would not recognize ** here as <strong> if you remove 4 or 5 spaces? -->
**Don't surround the whole sentence with the double-asterisk without adding extra spaces!** The Foobar language which is spoken by most CommonMark maintainers use as many as 6 spaces to split sentences.
This is what I have looked into by digging through rummaging through the Git history, change logs, and test cases now.
It is not surprising that maintainers and you lower the priority of this problem, since it does not affect any European language family, which puts space next to punctuation or parentheses.
I clearly doubt this. @wooorm I apologize again at this time for my anger and for being too militant in my remarks. My humble suggestions and comments on them:
I know. It is the background of this problem.
I have looked for ones and their frequency. Escaping them does not modify the rendered content itself, but I have been disgusted of having to modify the content by adding extra space or to depend on the inline raw JSX tag (
I will look into it later. (I do not expect it either) |
Checking the general Unicode categories Pc, Pd, Pe, Pf, Pi, Po and Ps, U+3001 Ideographic Comma and U+3002 Ideographic Full Stop are of course included in what Commonmark considers punctuation marks, which are all treated alike. For its definitions of flanking, CM could start to handle Open/Start Possibly affected Examples are, for instance: 363, 367+368, 371+372, 376 and 392–394. |
I checked the raised test cases. 367 is most affected in them. However, there are some ones not raised but more important. I am not convinced in the test case 378 (
Does it not mean that FYI, as of https://hypestat.com/info/github.com, one in six visitors in GitHub live in China or Japan. This percentage would not be able to be ignored or underestimated. |
The “Permitted content: Phrasing content” bit allow it for both.
I don’t think anybody is underestimating that. Practically, this is also open source, which implies that somebody has to do the work for free here, probably because they think it’s fun or important to do. And then folks working on markdown parsers need to do it too. To illustrate, GitHub hasn’t really done anything in the last 3 years (just security vulnerabilities / new fancy footnote footnotes feature). |
Getting emphasis right in markdown (especially nested emphasis) is very difficult. Changing the existing rules without messing up cases that currently work is highly nontrivial. For what it's worth, my rationalized syntax djot has simpler rules for emphasis, gives you what you want in the above Japanese example, and allows you to use braces to clarify nesting in cases where it's unclear, e.g. |
This is technically possible but not practical or necessary. It is much easier and faster to type "「" & "」" from the keyboard directly, and you cannot copy these brackets in
Almost all description on Markdown for newbies including the following say that
I do not know of SaaSes in Japan that customize the style of The current behavior of CommonMark forces newbies in China or Japan to try to decipher its spec. It is for developers of Markdown parsers, not for users except for experts. CommonMark has now grown to the point where it can manipulate the largest Markdown implementations (remark, markdonw-it, goldmark (used by Hugo), commonmarker (possibly used by GitHub), and so on) from behind the scenes. We may well lobby to revise its specification. (unenforceable of course though!) It would not be difficult to create a new specification of Markdown, but is difficult to give sufficient power to it. These are why I had tried to stop the left- and right-flanking, but I have found a convincing plan to recently. We have only to change by my plan:
We do not have to change the other. I hope most Chinese and Japanese can be convinced by it. Also, you can continue to nest
I am a little relieved to hear that. I apologize for the misunderstanding.
It would affect too many documents if the left- & right-flanking rule were abolished. However, the new plan will not affect on most existing documents except for ones that abuse the details of the spec. Do you mean that they are also included in "all existing" ones? I suggest new terms "punctuation run preceded by space" & "puncuation run followed by space".
(2a) and (2b) is going to be changed like the following:
This change treats punctuation characters that are not adjacent to space as normal letters. To see if the " **これは太字になりません。**ご注意ください。
カッコに注意**(太字にならない)**文が続く場合に要警戒。
**[リンク](https://example.com)**も注意。(画像も同様)
先頭の**`コード`も注意。**
**末尾の`コード`**も注意。 Also, we can parse even the following English as intended: You should write “John**'s**” instead. We do not concatenate too many punctuation characters, so we do not have to search more than ten and some (e.g. 16) punctuation characters for space from the previous or next of the target delimiter run. To check if the delimiter run is "the last characters in punctuation run preceded by space" (without using cache): flowchart TD
Next{"Is the<br>next character<br>an Unicode punctuation<br>chracter?"}
Next--> |YES| F["<code>return false</code>"]
Next--> |NO| Init["<code>current =</code><br>(previous character)<br><code>n =</code><br>(Length of delimiter run)"]
Init--> Exceed{"<code>n >= 16</code>?"}
Exceed--> |YES| F
Exceed --> |NO| Previous{"What type is <code>current</code>?"}
Previous --> |Not punctuation or space| F
Previous --> |Space| T["<code>return true</code>"]
Previous --> |Unicode punctuation| Iter["<code>n++<br>current =</code><br>(previous character)"]
Iter --> Exceed
In the current spec, to non-advanced users especially in China or Japan, " |
0.31 changes the wording slightly, but as far as I can tell this does not change flanking behavior at all.
|
The change made the situation even worse.
The few improvements are only that it is easier to explain the condition to beginners (we can now use the single word “symbols”) and more consistent with ASCII punctuation characters. |
This particular change was not intended to address this issue; it was just intended to make things more consistent. @tats-u I am sorry, I have not yet had time to give your proposal proper consideration. |
I guess it, but as a result it did cause a breaking change and break some documents (much less than ones affected by 0.14 though), which is a kind of regressions you have mostly feared and cared about. For the first place, we cannot easily access to convincing and practical examples that describe how legitimate controversial parts of specifications and changes are; we can easily find only ones that are designed only for testing and do not have meaning (e.g. What is needed is like: Price: **€**10 per month (note: you cannot pay in US$!)
FYI you do not have evaluate how optimize the algorithm in the above flowchart; it is too naive and can be optimized. All I want you to do first is to evaluate how acceptable breaking changes brought by my revision are. It might be better for me to make a PoC to make it easy to do it. |
To be honest, I didn't anticipate these breaking changes, and I would have thought twice about the change if I had. Having a parser to play with that implements your idea would make it easier to see what its consequences would be. (Ideally, a minimally altered cmark or commonmark.js.) It's also important to have a plan that can be implemented without significantly degrading the parser's performance. But my guess is that if it's just a check that has to be run once for each delimiter + punctuation run, it should be okay. |
@ArcticLampyrid You shouldn't confuse |
Take a look at Unicode spec, Full-width symbols are not exclusive to CJK at all. They can be used in any language for typesetting purposes. |
Not general. |
This Halfwidth and Fullwidth Forms was defined to provided so that older encodings containing both halfwidth and fullwidth characters can have lossless translation to/from Unicode. (words after "to" was quoted from Wikipedia) |
I think we should skip a standard variation selector in determining the "character before." There remains the question how to identify CJK characters. There are two main proposals: (a) script-based: use the Unicode Script property and check for one of the following scripts: Han, Hangul, Hiragana, Katakana, Bopomofo, Yiii. In cmark we could substitute code point ranges derived from these. (b) unicode block based: e.g., the following Unicode blocks from @woorm's comment above:
I am still not sure about how to decide between these. The script-based approach seems simpler. @tats-u pointed out some characters that were missing from the script-based regex in #650 (comment) Does the revision in #650 (comment) fix this problem? The Ideographic Variation Selectors (U+E0100–U+E01EF) should also be included (if they're not already captured by (a) or (b)). Wide parentheses: these do seem to be used only in CJK and may appear in some of the examples we want to handle; on the other hand it seems unprincipled if they're not in the scripts/ranges above. One alternative as @ArcticLampyrid suggests would be to make the flankingness determination sensitive to Unicode markings as open vs closed punctuation; this might make it unnecessary to include these in the CJK range, and might have other good effects elsewhere. It would be helpful if someone could go through this thread and compile a large list of the examples that have been proposed, which we can use for tests. |
All example [
"これは**私のやりたかったこと。**だからするの。",
"**[製品ほげ](./product-foo)**と**[製品ふが](./product-bar)**をお試しください",
"単語と**[単語と](word-and)**単語",
"**これは太字になりません。**ご注意ください。",
"カッコに注意**(太字にならない)**文が続く場合に要警戒。",
"**[リンク](https://example.com)**も注意。(画像も同様)",
"先頭の**`コード`も注意。**",
"**末尾の`コード`**も注意。",
"税込**¥10,000**で入手できます。",
"正解は**④**です。",
"太郎は\\ **「こんにちわ」**\\ といった",
"太郎は​**「こんにちわ」**​といった",
"太郎は**「こんにちわ」**といった",
"太郎は **「こんにちわ」** といった",
"太郎は**「こんにちわ」**といった",
"太郎は**\"こんにちわ\"**といった",
"太郎は**こんにちわ**といった",
"太郎は**「Hello」**といった",
"太郎は**\"Hello\"**といった",
"太郎は**Hello**といった",
"太郎は**「Oh my god」**といった",
"太郎は**\"Oh my god\"**といった",
"太郎は**Oh my god**といった",
"**C#**や**F#**は**「.NET」**というプラットフォーム上で動作します。",
"IDが**001号**になります。",
"IDが**001号**になります。",
"Go**「初心者」**を対象とした記事です。",
"Go**\"初心者\"**を対象とした記事です。",
"**[リンク](https://example.com)**も注意。",
"先頭の**",
"も注意。**",
"**⻲田太郎**と申します",
"・**㋐**:選択肢1つ目",
"**真,**她",
"**真。**她",
"**真、**她",
"**真;**她",
"**真:**她",
"**真?**她",
"**真!**她",
"**真“**她",
"**真”**她",
"**真‘**她",
"**真’**她",
"**真(**她",
"**真)**她",
"**真【**她",
"**真】**她",
"**真《**她",
"**真》**她",
"**真—**她",
"**真~**她",
"**真…**她",
"**真·**她",
"**真〃**她",
"**真-**她",
"**真々**她",
"**真**她",
"**真,** 她",
"**真**,她",
"**真,**​她",
"私は**⻲田太郎**と申します",
"選択肢**㋐**: 1つ目の選択肢",
"**さようなら︙**と太郎はいった。",
".NET**(.NET Frameworkは不可)**では、",
"「禰󠄀」の偏は示ではなく**礻**です。",
"Git**(注:不是GitHub)**",
"太郎は**「こんにちわ」**といった。",
"𰻞𰻞**(ビャンビャン)**麺",
"ハイパーテキストコーヒーポット制御プロトコル**(HTCPCP)**",
"﨑**(崎)**",
"国際規格**[ECMA-262](https://tc39.es/ecma262/)**",
"㐧**(第の俗字)**",
"𠮟**(こちらが正式表記)**",
"𪜈**(トモの合略仮名)**",
"𫠉**(馬の俗字)**",
"谺𬤲**(こだま)**石神社",
"石𮧟**(いしただら)**",
"ハイパーテキストコーヒーポット制御プロトコル**(HTCPCP)**",
"﨑**(崎)**",
"㐧**(第の俗字)** (Unofficial form)",
"𠮟**(こちらが正式表記)**",
"𪜈**(トモの合略仮名)** (Mixed-in uncommon joined katakana)",
"𫠉**(馬の俗字)** (Unofficial form)",
"谺𬤲**(こだま)**石神社 (shrine)",
"石𮧟**(いしただら)** (address)",
"**推荐几个框架:**React、Vue等前端框架。",
"葛󠄀**(こちらが正式表記)**城市",
"禰󠄀**(こちらが正式表記)**豆子"
] |
Removed some unintended duplicated ones (originally proposed by me) and added some. [
"これは**私のやりたかったこと。**だからするの。",
"**[製品ほげ](./product-foo)**と**[製品ふが](./product-bar)**をお試しください",
"単語と**[単語と](word-and)**単語",
"**これは太字になりません。**ご注意ください。",
"カッコに注意**(太字にならない)**文が続く場合に要警戒。",
"**[リンク](https://example.com)**も注意。(画像も同様)",
"先頭の**`コード`も注意。**",
"**末尾の`コード`**も注意。",
"税込**¥10,000**で入手できます。",
"正解は**④**です。",
"太郎は\\ **「こんにちわ」**\\ といった",
"太郎は​**「こんにちわ」**​といった",
"太郎は**「こんにちわ」**といった",
"太郎は **「こんにちわ」** といった",
"太郎は**「こんにちわ」**といった",
"太郎は**\"こんにちわ\"**といった",
"太郎は**こんにちわ**といった",
"太郎は**「Hello」**といった",
"太郎は**\"Hello\"**といった",
"太郎は**Hello**といった",
"太郎は**「Oh my god」**といった",
"太郎は**\"Oh my god\"**といった",
"太郎は**Oh my god**といった",
"**C#**や**F#**は**「.NET」**というプラットフォーム上で動作します。",
"IDが**001号**になります。",
"IDが**001号**になります。",
"Go**「初心者」**を対象とした記事です。",
"**[リンク](https://example.com)**も注意。",
"先頭の**",
"も注意。**",
"**⻲田太郎**と申します",
"・**㋐**:選択肢1つ目",
"**真,**她",
"**真。**她",
"**真、**她",
"**真;**她",
"**真:**她",
"**真?**她",
"**真!**她",
"**真“**她",
"**真”**她",
"**真‘**她",
"**真’**她",
"**真(**她",
"真**(她**",
"**真)**她",
"**真【**她",
"真**【她**",
"**真】**她",
"**真《**她",
"真**《她**",
"**真》**她",
"**真—**她",
"**真~**她",
"**真…**她",
"**真·**她",
"**真〃**她",
"**真-**她",
"**真々**她",
"**真**她",
"**真,** 她",
"**真**,她",
"**真,**​她",
"私は**⻲田太郎**と申します",
"選択肢**㋐**: 1つ目の選択肢",
"**さようなら︙**と太郎はいった。",
".NET**(.NET Frameworkは不可)**では、",
"「禰󠄀」の偏は示ではなく**礻**です。",
"Git**(注:不是GitHub)**",
"太郎は**「こんにちわ」**といった。",
"𰻞𰻞**(ビャンビャン)**麺",
"𰻞𰻞**(ビャンビャン)**麺",
"ハイパーテキストコーヒーポット制御プロトコル**(HTCPCP)**",
"﨑**(崎)**",
"国際規格**[ECMA-262](https://tc39.es/ecma262/)**",
"㐧**(第の俗字)**",
"𠮟**(こちらが正式表記)**",
"𪜈**(トモの合略仮名)**",
"𫠉**(馬の俗字)**",
"谺𬤲**(こだま)**石神社",
"石𮧟**(いしただら)**",
"**推荐几个框架:**React、Vue等前端框架。",
"葛󠄀**(こちらが正式表記)**城市",
"禰󠄀**(こちらが正式表記)**豆子",
"**(U+317DB)**",
"阿寒湖アイヌシアターイコㇿ**(Akanko Ainu Theater Ikor)**",
"あ𛀙**(か)**よろし",
"**(simplified form of 龘 in China)**",
"大塚︀**(or 大塚 / 大塚)**"
] |
Great! Do these include any with a Standard Variation Selector? Would be good to have one of those, too, with a CJK character with Standard Variation Selector before the |
The first and last ones have the same form but only the last one is converted to the middle one if normalized. Only the first one contains a SVS. https://light.fusic.co.jp/2021/05/06/20210506-sakai/ (Japanese) |
Thanks, I added it to the list. |
We have to note the current plan can't cover this case because neither |
@tats-u This may be another issue and should not be addressed in this issue 🤔 |
I've wanted to split this issue into more than one. |
This just means |
This comment was marked as outdated.
This comment was marked as outdated.
"太郎は**\"こんにちわ\"**といった",
"**真-**她" They should be passed even in the current plan because the outer characters are han. |
@tats-u Sorry, my bad, hidden. |
I'm still not sure about the question I asked above regarding unicode scripts vs code point blocks. Is the script-based solution viable? |
I appreciate all the discussion; thank you. Reading through the issue, it's not clear to me whether half-width and full-width punctuation and numbers are going to be treated the same (for example, is
|
These should be fine because the
These cases should be fine, too, as long as there is whitespace around the
But if you had something like
it would not be handled, unless we adopt the suggestion above of making flankingness detection sensitive to the open- and close-punctuation classes. Even that suggestion would not help with
The first of those cases could be handled if we decided to treat full-width 5 and % as CJK characters. As for the second, I don't think it's handled by any current proposal. |
If we were going to adopt the former as the main, we would have to make up for the lack of characters by the latter because scripts don't seem to cover some essential CJK symbols. Scripts has the pros the spec doesn't have to be revised even if new Unicode blocks for CJK languages are appended in the future.
All correct. |
Can you be more specific about what, exactly, is missing? |
@jgm #650 (comment) at least I doubt |
I'll look into https://www.unicode.org/Public/15.0.0/ucd/ScriptExtensions.txt or try |
@jgm @tats-u I believe that even addressing just these cases would greatly improve the current CJK support; it’s not necessary to solve all the problems at once.
I will pull some Chinese content from the internet and analyze the frequency of these cases. |
Agree with you and it's why I've shelved other more complex solutions.
Could you share it with us how? (Especially how to scrape and filter) |
It worked only on
In the first place, fullwidth ASCII-compatible symbols are not written in |
As expected. For the first one, it's related to SVS, which should be handled with additional rules. |
This is IVS, not SVS, but can be handled in the same way as SVS (or just treated as CJK as I've said).
Definitely we have to do. This is why I assert we should use Unicode blocks as primary or at least secondary. |
Hi, I encountered some strange behavior when using CJK full-width punctuation and trying to add emphasis.
Original issue here
Example punctuation that causes this issue:
。!?、
To my mind, all of these should work as emphasis, but some do and some don't:
I'm not sure if this is the spec as intended, but in Japanese, as a general rule there are no spaces in sentences, which leads to the following kind of problem when parsing emphasis.
In English, this is emphasized as expected:
This is **what I wanted to do.** So I am going to do it.
But the same sentence emphasized in the same way in Japanese fails:
これは**私のやりたかったこと。**だからするの。
The text was updated successfully, but these errors were encountered: