[For v3+ / breaking change] Should we change how to handle line breaking surrounded by han/kana when `proseWrap` is `always` #14936

tats-u · 2023-06-11T14:19:56Z

# Options (if any):
--print-width=40 --prose-wrap=always

Input:

English 日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語 English

Output:

English 日本語日本語日本語日本語日本語日
本語日本語日本語日本語日本語日本語日本語
日本語日本語日本語日本語日本語日本語
English

Expected behavior:

English
日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語日本語
English

Almost all Markdown renderers always treat the line breaking as space even if han/kana is included in characters surrounding it.

English 日本語日本語日本語日本語日本語日
本語日本語日本語日本語日本語日本語日本語
日本語日本語日本語日本語日本語日本語
English

↓

English 日本語日本語日本語日本語日本語日 本語日本語日本語日本語日本語日本語日本語 日本語日本語日本語日本語日本語日本 English

Which should Prettier consider is more important, the appearance of formatted documents, or the compatibility with Markdown renderers?

~~It is OK for me to keep the current behavior and ask authors of renderers to change their behavior.~~

Memo:

#3026 (the beginning of the nightmare)
↓
#5040 (mitigation for Korean)
↓
#11597 (mitigation)
↓
#14936 (fix for this issue)

The text was updated successfully, but these errors were encountered:

kachkaev · 2023-06-13T11:08:14Z

👋 @tats-u, Is #11597 solve your issue? See blog post draft in the diff. This is change planned for 3.0.

tats-u · 2023-06-13T12:55:59Z

@kachkaev I'm its author. I didn't change the current behavior in the PR intentionally.

https://deploy-preview-11597--prettier.netlify.app/playground/#N4Igxg9gdgLgprEAuEBRKBzANgSwM4AWABIKemgNOaB5UeVZdXbQzU-c4y9UetvgSADQgIABxg5oeZKACGAJxkQA7gAVZCCSilYFUgJ4SBAIxlSwAazgwAylIC2cADI4ocZDBkBXOIeNmLloSZOGK4eXiBwNgZwACbRMfZSmO5SGHAAYhAyNlIwopjIIFLuMBD8IAQwNlgA6gQ48HgBYHCWavU4AG71OgVgePogTnhwMjBKxhjZyABmmsMCAFZ4AB6WQVhwAIruEPAzc2EBMsMyBdkyptGKUGVCMk4w1TjRMLxIACwADAJ3EMPVxiEBU02j0ZQAjjt4ONhOpCngALTOGIxMoyOCQnDo8YpKZIWZYeYgYY2HAhTwCPDrLZQlz4g4CGBSAxPF5vABMjOMOFwmAAwhAbHjClgsGV3MMACrM9QEokdTwASSgcVgljA9xEAEEVZYYDoNvtCXAAL4moA

This is because there was no agreement to change the behavior.

I fixed just the problem Prettier inserts spaces between han/kana and alphanumeric not separated by a space in that PR.

tats-u · 2023-06-14T15:14:34Z

A conforming parser may render a soft line break in HTML either as a line ending or as a space.

A renderer may also provide an option to render soft line breaks as hard line breaks.

https://spec.commonmark.org/0.30/#softbreak

tats-u · 2023-06-14T15:21:16Z

I tried in CodePen.

HTML:

あ
い
う
a
b
c
あ
。
a。
あ。
あ！
あ！

↓

Looks like in Firefox:

あいう a b c あ。 a。 あ。あ！あ！

Copy & paste / Looks like in Chrome:

あ い う a b c あ 。 a。 あ。 あ！ あ！

I also previewed this HTML in both browser and got the same result.

<!DOCTYPE html>
<html lang="ja">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Document</title>
</head>
<body>
あ
い
う
a
b
c
あ
。
a。
あ。
あ！
あ！ 
</body>
</html>

https://infra.spec.whatwg.org/#strip-and-collapse-ascii-whitespace

To strip and collapse ASCII whitespace in a string, replace any sequence of one or more consecutive code points that are ASCII whitespace in the string with a single U+0020 SPACE code point, and then remove any leading and trailing ASCII whitespace from that string.

↑ I couldn't find where it's used.

ASCII whitespace is U+0009 TAB, U+000A LF, U+000C FF, U+000D CR, or U+0020 SPACE.

tats-u · 2023-07-04T14:21:45Z

I (or we) want to have a PR in time for the 3.0 release, or it'll be much more difficult to reach agreement on a fix.

tats-u · 2023-07-05T12:54:12Z

Omg v3 was suddenly released without any betas. Should we push this back to v4?
I think proseWrap = always is not practical In Chinese and Japanese because there's no soft line breaks stipulated in HTML & Markdown in these languages.
Line breaks are never converted to spaces there.

fisker · 2023-07-07T05:21:49Z

I agree it's a bug. We can only change the space to a new line.

tats-u · 2023-07-07T08:49:24Z

@fisker Can't we do the opposite like HTML?
I think compatibility with HTML should be observed rather than the number of characters per line.

fisker · 2023-07-07T09:05:26Z

I think so.

tats-u · 2023-07-07T09:57:05Z

@fisker
I couldn't get which your "so" refers to, but:

Newline and space will be able to be interchanged with each other unconditionally
Continuous Chinese and Japanese won't be broken (like Korean)

This is my plan. Can we go with it?

fisker · 2023-07-07T11:36:28Z

I mean

Newline and space will be able to be interchanged with each other unconditionally.

Continuous Chinese and Japanese won't be broken (like Korean)

Make sense.

tats-u · 2023-07-07T11:54:06Z

Thanks. Now that we have the specs, someone or I can try for a PR.

fisker · 2023-07-07T11:56:24Z

Let's wait a bit longer for more maintainers to express their opinions before start.

tats-u · 2023-07-07T11:57:57Z

Okey dokey.

tats-u · 2023-07-07T12:01:43Z

By the way, if this is a bug, can we force a patch for it into v3.0.1 or 3.1, without waiting for v4?

fisker · 2023-07-07T12:05:19Z

I forgot how it works in v2. Do we insert new lines?

tats-u · 2023-07-07T13:48:07Z

https://deploy-preview-5040--prettier.netlify.app/playground/#N4Igxg9gdgLgprEAuEhpRnRzXs93vAOlEYBMqZ5FlV1NtVRRqjzUTrDrb6jbHJpAqP2JDhZUsKKBJYIA2EGdLkKoUokulr5HKYHxXQBIMgfQZAQAzT9xzct2GTKs0SMgANCAgAHGAEtoAZ2SgAhgBOgRAA7gAKQQi+KP4AbhAeACZOIABGgf5gANZwMADKrlkeUADmyDCBAK5wziXecIEw4ZmlALb+yABm-jINzgBW3gAeAEKZOXn5-m1wADIlcN29-SBFgQ2ByCAdgdlJYVCproElMADqyTAAFsgALAAMzicQDeeZrtu9of4Anr7OQJwACOVQ8QJa-nanSQPT6tRADTaHgq1QR3hKpRkcAAilUIPBlvDnDB-GlLkkbsgAEwkzIeGSYgDCEDaHW2UGgS2cVQaABUyTE4Q0AL4ioA

As of #5040, all of CJK used to be able to be split by hard-line.
This behavior is not HTML-compliant. I don't like it now.
We're going to ban it for han/kana (I've already banned for hangul)

tats-u · 2023-07-09T02:51:47Z

@fisker Should we still remove trailing or leading full-width spaces (U+3000)?
I agree with the trailing without any conditions and the leading only if recommends users to use the following CSS:

/* or some classes */
p {
    text-indent: -1em;
    margin-left: 1em;
}

fisker · 2023-07-11T15:01:13Z

Should we still remove trailing or leading full-width spaces (U+3000)?

I think we need keep it.

fisker · 2023-07-11T15:06:25Z

Newline and space will be able to be interchanged with each other unconditionally.

This should be HTML whitespace. They can change.

tats-u · 2023-07-16T14:55:18Z

@fisker

This should be HTML whitespace. They can change.

I continue with the current specs.

I think we need keep it.

You say we should remove this line, right?

prettier/src/language-markdown/utils.js

Line 151 in 01d93c6

![lastNode.value, node.value].some((value) => /\u3000/.test(value))

thorn0 · 2023-07-21T11:24:48Z

Which should Prettier consider is more important, the appearance of formatted documents, or the compatibility with Markdown renderers?

The CommonMark spec is Prettier's main reference point, so I agree this needs to be changed.

tats-u · 2023-07-22T11:02:30Z

@thorn0 Thanks for your approval.

prettier/src/language-markdown/utils.js

Line 151 in 01d93c6

![lastNode.value, node.value].some((value) => /\u3000/.test(value))

I found this is innocent.
The real culprits are String.prototype.trim{Start,End}. This seems to remove all type of spaces (Zs in Unicode).

tats-u · 2023-07-22T11:32:30Z

I have started to use the hack since several weeks ago:

((U+FEFF))<SomeComponent />I *can* use Markdown with leading JSX like `<SomeComponent />` in MDX v1!

((U+FEFF)) is the ZERO WIDTH NO-BREAK SPACE.

But Prettier removes the U+FEFF and the text after <SomeComponent /> will be rendered as is (not interpreted as Markdown).

tats-u · 2023-07-22T11:59:43Z

#15081 is now ready for battle tests.

chalin · 2024-06-07T13:54:09Z

Any planned progress here and/or workarounds, other than disabling Prettier over zh-* and ja pages? Thanks. /cc @windsonsea

tats-u · 2024-06-08T03:42:56Z

Unfortunately suspended recently

tats-u changed the title ~~[For v3 / breaking changes] when proseWrap is always~~ [For v3 / breaking changes] Should we change how to handle line breaking surrounded by han/kana when proseWrap is always Jun 11, 2023

tats-u changed the title ~~[For v3 / breaking changes] Should we change how to handle line breaking surrounded by han/kana when proseWrap is always~~ [For v3 / breaking change] Should we change how to handle line breaking surrounded by han/kana when proseWrap is always Jun 11, 2023

tats-u changed the title ~~[For v3 / breaking change] Should we change how to handle line breaking surrounded by han/kana when proseWrap is always~~ [For v3+ / breaking change] Should we change how to handle line breaking surrounded by han/kana when proseWrap is always Jun 11, 2023

kachkaev added the status:awaiting response Issues that require answers to questions from maintainers before action can be taken label Jun 13, 2023

github-actions bot removed the status:awaiting response Issues that require answers to questions from maintainers before action can be taken label Jun 13, 2023

kachkaev added status:needs discussion Issues needing discussion and a decision to be made before action can be taken lang:markdown Issues affecting Markdown labels Jun 13, 2023

tats-u mentioned this issue Jul 6, 2023

Markdown: Add an option to re-enable Prettier 2.x's automatic space insertion in CJK #15015

Closed

fisker added type:bug Issues identifying ugly output, or a defect in the program and removed status:needs discussion Issues needing discussion and a decision to be made before action can be taken labels Jul 7, 2023

tats-u linked a pull request Jul 11, 2023 that will close this issue

Make Markdown's newline & space handling conform to web-platform-tests & Firefox's behavior #15081

Open

4 tasks

windsonsea mentioned this issue Apr 23, 2024

[zh] Add zh version of homepage open-telemetry/opentelemetry.io#4341

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[For v3+ / breaking change] Should we change how to handle line breaking surrounded by han/kana when `proseWrap` is `always` #14936

[For v3+ / breaking change] Should we change how to handle line breaking surrounded by han/kana when `proseWrap` is `always` #14936

tats-u commented Jun 11, 2023 •

edited

kachkaev commented Jun 13, 2023

tats-u commented Jun 13, 2023 •

edited

tats-u commented Jun 14, 2023

tats-u commented Jun 14, 2023 •

edited

tats-u commented Jul 4, 2023 •

edited

tats-u commented Jul 5, 2023 •

edited

fisker commented Jul 7, 2023

tats-u commented Jul 7, 2023 •

edited

fisker commented Jul 7, 2023

tats-u commented Jul 7, 2023 •

edited

fisker commented Jul 7, 2023

tats-u commented Jul 7, 2023 •

edited

fisker commented Jul 7, 2023

tats-u commented Jul 7, 2023

tats-u commented Jul 7, 2023 •

edited

fisker commented Jul 7, 2023

tats-u commented Jul 7, 2023 •

edited

tats-u commented Jul 9, 2023

fisker commented Jul 11, 2023

fisker commented Jul 11, 2023

tats-u commented Jul 16, 2023

thorn0 commented Jul 21, 2023 •

edited

tats-u commented Jul 22, 2023

tats-u commented Jul 22, 2023 •

edited

tats-u commented Jul 22, 2023

chalin commented Jun 7, 2024

tats-u commented Jun 8, 2024 •

edited

[For v3+ / breaking change] Should we change how to handle line breaking surrounded by han/kana when proseWrap is always #14936

[For v3+ / breaking change] Should we change how to handle line breaking surrounded by han/kana when proseWrap is always #14936

Comments

tats-u commented Jun 11, 2023 • edited

kachkaev commented Jun 13, 2023

tats-u commented Jun 13, 2023 • edited

tats-u commented Jun 14, 2023

tats-u commented Jun 14, 2023 • edited

tats-u commented Jul 4, 2023 • edited

tats-u commented Jul 5, 2023 • edited

fisker commented Jul 7, 2023

tats-u commented Jul 7, 2023 • edited

fisker commented Jul 7, 2023

tats-u commented Jul 7, 2023 • edited

fisker commented Jul 7, 2023

tats-u commented Jul 7, 2023 • edited

fisker commented Jul 7, 2023

tats-u commented Jul 7, 2023

tats-u commented Jul 7, 2023 • edited

fisker commented Jul 7, 2023

tats-u commented Jul 7, 2023 • edited

tats-u commented Jul 9, 2023

fisker commented Jul 11, 2023

fisker commented Jul 11, 2023

tats-u commented Jul 16, 2023

thorn0 commented Jul 21, 2023 • edited

tats-u commented Jul 22, 2023

tats-u commented Jul 22, 2023 • edited

tats-u commented Jul 22, 2023

chalin commented Jun 7, 2024

tats-u commented Jun 8, 2024 • edited

[For v3+ / breaking change] Should we change how to handle line breaking surrounded by han/kana when `proseWrap` is `always` #14936

[For v3+ / breaking change] Should we change how to handle line breaking surrounded by han/kana when `proseWrap` is `always` #14936

tats-u commented Jun 11, 2023 •

edited

tats-u commented Jun 13, 2023 •

edited

tats-u commented Jun 14, 2023 •

edited

tats-u commented Jul 4, 2023 •

edited

tats-u commented Jul 5, 2023 •

edited

tats-u commented Jul 7, 2023 •

edited

tats-u commented Jul 7, 2023 •

edited

tats-u commented Jul 7, 2023 •

edited

tats-u commented Jul 7, 2023 •

edited

tats-u commented Jul 7, 2023 •

edited

thorn0 commented Jul 21, 2023 •

edited

tats-u commented Jul 22, 2023 •

edited

tats-u commented Jun 8, 2024 •

edited