Make Markdown's newline & space handling conform to web-platform-tests & Firefox's behavior #15081

tats-u · 2023-07-11T14:56:31Z

Description

Fixes #14936

~~Don't wrap even Chinese & Japanese when proseWrap is always~~
Make newline and whitespace interchangeable even between (Chinese or Japanese) and another character when proseWrap is always or never
Don't trim spaces other than ASCII whitespace at the beginning or end of paragraphs

Checklist

I’ve added tests to confirm my change works.
(If changing the API or CLI) I’ve documented the changes I’ve made (in the docs/ directory).
(If the change is user-facing) I’ve added my changes to changelog_unreleased/*/XXXX.md file following changelog_unreleased/TEMPLATE.md.
I’ve read the contributing guidelines.

✨Try the playground for this PR✨

tats-u · 2023-07-22T15:00:31Z

Shoud I add changelog_unreleased/markdown/15081.md?

fisker · 2023-07-22T15:13:19Z

Can you send a separate PR for this first?

Don't trim spaces other than ASCII whitespace at the beginning or end of paragraphs

And we can use this module to do the html trim.

tats-u · 2023-07-22T15:27:11Z

@fisker In CommonMark spec, the form feed \f (U+000C) is in the same category as the full-width space (and other spaces in Zs).

https://spec.commonmark.org/0.30/#unicode-whitespace-character

Either way, it is destined to be removed at the stage where the converted HTML is processed, so it may as well be removed at the stage of conversion in Prettier.

https://codepen.io/tats-u/pen/bGQjrvM

~~I'll reuse that code.~~

Trailing form feeds don't seem always to be removed.

fisker · 2023-07-22T16:48:39Z

Interesting, https://bugs.webkit.org/show_bug.cgi?id=13159 seems we already made mistake in HTML/Handlebars printer.

fisker · 2023-07-22T17:06:40Z

Anyway, I think we should reuse htmlWhitespaceUtils, but consider remove \f in future if it really matters.

tats-u · 2023-07-23T08:01:31Z

@fisker Your change has just been applied.

tats-u · 2023-07-24T08:56:54Z

I found an interesting description in https://drafts.csswg.org/css-text-3/#line-break-transform

Then any remaining segment break (\n) is either transformed into a space (U+0020) or removed depending on the context before and after the break. The rules for this operation are UA-defined in this level.

NOTE: Historically, HTML and CSS have unconditionally converted segment breaks to spaces, which has prevented content authored in languages such as Chinese from being able to break lines within the source. Thus UA heuristics need to be conservative about where they discard segment breaks even as they strive to improve support for such languages.

What sucks is not the HTML or CSS specs but browsers' implementations.
I have to modify the change log and comments in the source code.

fisker · 2023-07-29T03:03:25Z

src/language-markdown/print-whitespace.js

-  if (isLink) {
-    return true;
-  }
+function lineBreakCanBeConvertedToSpace() {


Do you think we should remove this function?

It depends on whether we'll have to add it again.
I can't predict when all browsers will change their behavior.

fisker

The result looks good, thank you!

tats-u · 2023-07-29T14:18:25Z

@fisker I'll change the behavior.

I looked into the detailed behavior of Firefox.

Table: Is a newline between the followings converted to a space in Firefox?

\	Latin & Korean	Chinese & Japanese & CJK punctuations
Latin & Korean	Yes	Yes
Chinese & Japanese & CJK punctuations	Yes	No

A newline between [Chinese & Japanese & CJK punctuations] should not be converted to a space.
Once it is removed, these

漢字
だよ
。

↓ OK

漢字だよ。

↓ STOP!

漢字
だよ。

This will improve the compatibility with the conventional behavior to some extend and reduce browser-defined behaviors.

A space around Chinese (& Japanese) is going not to be converted to a newline. It's not compatible with Firefox.

fisker · 2023-07-29T14:22:38Z

A newline between [Chinese & Japanese & CJK punctuations] should not be converted to a space.

Didn't get it.

tats-u · 2023-07-29T14:24:05Z

[Chinese & Japanese & CJK punctuations]

These seem to be treated as a single group in Firefox.
A newline surrounded only by that group is simply removed (in a view).
Of course, a space is added once copied and pasted on somewhere else.

tats-u · 2023-07-29T14:27:19Z

https://codepen.io/tats-u/pen/poQQVyR

fisker · 2023-07-29T14:28:13Z

Should't we only need make sure the output equivalence in html?

Ah, maybe you are right, in markdown one single new line doesn't generate a space. But should not related to languages.

tats-u · 2023-07-29T14:47:06Z

@fisker

But should not related to languages.

Does it mean it's better to always treat a newline in Markdown as a space even if surrounded only by han or kana?
Indeed Markdown viewers other than browsers use Blink, but the behavior doesn't look natural as a Chinese or Japanese paragraph for me.
For the first place few people break lines in Chinese or Japanese sentences in Markdown documents, though.

tats-u · 2023-08-06T09:59:39Z

I found the latest version of CSS is 4 but the specification related to this issue has not been changed.

https://drafts.csswg.org/css-text-4/#white-space-trim

fisker · 2023-08-06T11:01:45Z

I'm glad you seem to understand it.

For the record, I don't.

I still think we only need make sure we doesn't change the output "HTML".

a b.

equals to

<p>a b.</p>

and

a
b.

equals to

<p>a
b.</p>

And you are saying they are not equal when we replace a or b with other characters.

This means we also handle it incorrectly in HTML printer. Playground link I think we should deal with the HTML printer first then.

fisker · 2023-08-06T11:11:34Z

@thorn0 You are more familiar with the markdown text wrapping. Do you have a comment about the problem here?

tats-u · 2023-08-06T23:07:34Z

@fisker

And you are saying they are not equal when we replace a or b with other characters.

I'm not saying that for the Markdown-to-HTML conversion. Your interruption for Markdown-to-HTML conversion is correct. All Newline is kept as is every when converted to HTML.
I've been discussing how the HTML is interpreted using CSS by browsers.

This means we also handle it incorrectly in HTML printer.

The current one breaks documents that explain how browsers handle a newline in HTML. I have to add prettier-ignore. I'll add concrete examples later.

The CommonMark specification misunderstands how newline in HTML should be interpreted by browsers. Its author mustn't have read the CSS specification.

A soft line break may be rendered in HTML either as a line ending or as a space. The result will be the same in browsers.

https://spec.commonmark.org/0.30/#soft-line-breaks

The last sentence isn't applicable to Chinese and Japanese documents.

tats-u · 2023-08-07T12:57:40Z

<p>
  In the CSS specification, it depends on browsers how newlines in HTML are treated. When viewed in Firefox, no extra spaces appear in the following sentences.
</p>
<blockquote>
  <p>
    これは日本語
    の文です。这
    是一个中文句
    子。
  </p>
</blockquote>

tats-u · 2023-09-09T06:51:59Z

@fisker

https://wpt.fyi/results/css/css-text/line-breaking?label=master&label=experimental&aligned
Test cases beginning with segment-break-transformation- are helpful.
Chrome treats the current behavior as a bug.

tats-u · 2023-09-09T12:58:23Z

Shall we follow all its tests?

tats-u · 2023-09-16T12:12:57Z

I wonder why KATAKANA MIDDLE DOT U+30FB (・) has been treated as non-CJK in Prettier.
It is treated as a kind of Japanese letter in Firefox.

<p>中点
・
中点</p>

This is rendered as "中点・中点" in Firefox but Prettier can format the following Markdown as "中点・中点".

中点
・
中点

tats-u · 2023-09-16T12:28:01Z

Katakana-Hiragana Double Hyphen U+30A0 (゠) is also excluded from CJK in Prettier even though it's in the Kataka Unicode block.

チンギス゠カン (Genghis Khan)

tats-u · 2023-09-25T13:26:19Z

@fisker I finished to change. You can start a review or merge.

…platform-tests

tats-u · 2023-10-02T14:17:32Z

Is this planned to be included in v3.1?

sosukesuzuki · 2023-10-04T14:57:11Z

I think this has too much impact to release as a minor update...

tats-u · 2023-10-08T13:43:18Z

I'll have to split changes in this PR into about 3 PRs:

Don't trim spaces other than ASCII whitespace at the beginning or end of paragraphs
Change for this change but don't be a breaking change
This change (maybe breaking)

Of course, I want the spec of Prettier to the current changes ultimately. I hope you agree with this plan.

tats-u · 2023-12-04T09:04:38Z

@fisker @sosukesuzuki Recently I have noticed the plan for the release of v4.
How do you think about this change as a content in v4?

fisker · 2023-12-05T06:17:53Z

Current v4 is published for the new CLI test.

tats-u force-pushed the markdown-newline-html branch 3 times, most recently from 92276e4 to 01d93c6 Compare July 13, 2023 13:57

tats-u marked this pull request as ready for review July 16, 2023 14:55

tats-u force-pushed the markdown-newline-html branch 2 times, most recently from 894c31f to 01eb21b Compare July 22, 2023 11:54

tats-u mentioned this pull request Jul 22, 2023

[For v3+ / breaking change] Should we change how to handle line breaking surrounded by han/kana when proseWrap is always #14936

Open

tats-u changed the title ~~Make Makdown's newline handling conform to HTML~~ Make Makdown's newline & space handling conform to HTML Jul 22, 2023

tats-u force-pushed the markdown-newline-html branch from 24e63a7 to 65182d7 Compare July 22, 2023 14:50

tats-u changed the title ~~Make Makdown's newline & space handling conform to HTML~~ Make Markdown's newline & space handling conform to HTML Jul 23, 2023

tats-u force-pushed the markdown-newline-html branch 2 times, most recently from 90de55f to 304b1de Compare July 23, 2023 10:58

tats-u changed the title ~~Make Markdown's newline & space handling conform to HTML~~ Make Markdown's newline & space handling conform to browsers Jul 27, 2023

fisker reviewed Jul 29, 2023

View reviewed changes

tats-u marked this pull request as draft July 29, 2023 03:12

fisker approved these changes Jul 29, 2023

View reviewed changes

github-actions bot mentioned this pull request Jul 29, 2023

Misc tests prettier/prettier-regression-testing#342

Open

tats-u closed this Aug 6, 2023

tats-u reopened this Aug 6, 2023

tats-u force-pushed the markdown-newline-html branch 2 times, most recently from 6032cc7 to e9c58f8 Compare September 16, 2023 12:08

tats-u marked this pull request as ready for review September 16, 2023 13:27

tats-u requested a review from fisker September 16, 2023 13:27

tats-u force-pushed the markdown-newline-html branch 2 times, most recently from f21da19 to 78d7874 Compare September 23, 2023 13:15

tats-u changed the title ~~Make Markdown's newline & space handling conform to browsers~~ Make Markdown's newline & space handling conform to web-platform-tests & Firefox Sep 24, 2023

tats-u changed the title ~~Make Markdown's newline & space handling conform to web-platform-tests & Firefox~~ Make Markdown's newline & space handling conform to web-platform-tests & Firefox's behavior Sep 24, 2023

tats-u added 4 commits October 1, 2023 12:56

Make Markdown's newline handling conform to Firefox's behavior & web-…

2db2fee

…platform-tests

Add test cases for trimming spaces

458f076

Add changelog

7ef3eb5

Treat U+30A0 & U+30FB in Kataka Block as CJK

21bff02

tats-u force-pushed the markdown-newline-html branch from 78d7874 to 21bff02 Compare October 1, 2023 03:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make Markdown's newline & space handling conform to web-platform-tests & Firefox's behavior #15081

Make Markdown's newline & space handling conform to web-platform-tests & Firefox's behavior #15081

tats-u commented Jul 11, 2023 •

edited

Loading

tats-u commented Jul 22, 2023

fisker commented Jul 22, 2023

tats-u commented Jul 22, 2023 •

edited

Loading

fisker commented Jul 22, 2023

fisker commented Jul 22, 2023

tats-u commented Jul 23, 2023 •

edited

Loading

tats-u commented Jul 24, 2023 •

edited

Loading

fisker Jul 29, 2023

tats-u Jul 29, 2023

fisker left a comment

tats-u commented Jul 29, 2023 •

edited

Loading

fisker commented Jul 29, 2023

tats-u commented Jul 29, 2023 •

edited

Loading

tats-u commented Jul 29, 2023

fisker commented Jul 29, 2023 •

edited

Loading

tats-u commented Jul 29, 2023 •

edited

Loading

tats-u commented Aug 6, 2023 •

edited

Loading

fisker commented Aug 6, 2023

fisker commented Aug 6, 2023

tats-u commented Aug 6, 2023 •

edited

Loading

tats-u commented Aug 7, 2023 •

edited

Loading

tats-u commented Sep 9, 2023

tats-u commented Sep 9, 2023

tats-u commented Sep 16, 2023 •

edited

Loading

tats-u commented Sep 16, 2023

tats-u commented Sep 25, 2023

tats-u commented Oct 2, 2023

sosukesuzuki commented Oct 4, 2023

tats-u commented Oct 8, 2023

tats-u commented Dec 4, 2023

fisker commented Dec 5, 2023

Make Markdown's newline & space handling conform to web-platform-tests & Firefox's behavior #15081

Are you sure you want to change the base?

Make Markdown's newline & space handling conform to web-platform-tests & Firefox's behavior #15081

Conversation

tats-u commented Jul 11, 2023 • edited Loading

Description

Checklist

tats-u commented Jul 22, 2023

fisker commented Jul 22, 2023

tats-u commented Jul 22, 2023 • edited Loading

fisker commented Jul 22, 2023

fisker commented Jul 22, 2023

tats-u commented Jul 23, 2023 • edited Loading

tats-u commented Jul 24, 2023 • edited Loading

fisker Jul 29, 2023

Choose a reason for hiding this comment

tats-u Jul 29, 2023

Choose a reason for hiding this comment

fisker left a comment

Choose a reason for hiding this comment

tats-u commented Jul 29, 2023 • edited Loading

fisker commented Jul 29, 2023

tats-u commented Jul 29, 2023 • edited Loading

tats-u commented Jul 29, 2023

fisker commented Jul 29, 2023 • edited Loading

tats-u commented Jul 29, 2023 • edited Loading

tats-u commented Aug 6, 2023 • edited Loading

fisker commented Aug 6, 2023

fisker commented Aug 6, 2023

tats-u commented Aug 6, 2023 • edited Loading

tats-u commented Aug 7, 2023 • edited Loading

tats-u commented Sep 9, 2023

tats-u commented Sep 9, 2023

tats-u commented Sep 16, 2023 • edited Loading

tats-u commented Sep 16, 2023

tats-u commented Sep 25, 2023

tats-u commented Oct 2, 2023

sosukesuzuki commented Oct 4, 2023

tats-u commented Oct 8, 2023

tats-u commented Dec 4, 2023

fisker commented Dec 5, 2023

tats-u commented Jul 11, 2023 •

edited

Loading

tats-u commented Jul 22, 2023 •

edited

Loading

tats-u commented Jul 23, 2023 •

edited

Loading

tats-u commented Jul 24, 2023 •

edited

Loading

tats-u commented Jul 29, 2023 •

edited

Loading

tats-u commented Jul 29, 2023 •

edited

Loading

fisker commented Jul 29, 2023 •

edited

Loading

tats-u commented Jul 29, 2023 •

edited

Loading

tats-u commented Aug 6, 2023 •

edited

Loading

tats-u commented Aug 6, 2023 •

edited

Loading

tats-u commented Aug 7, 2023 •

edited

Loading

tats-u commented Sep 16, 2023 •

edited

Loading