Desktop: Fix #5626: When web clipper clipping code blocks, keep code in multiline and delete code number lines #10126

wljince007 · 2024-03-15T08:08:39Z

Desktop: Fix #5626: when clipper, keep code in multiline and delete code number lines

See an example, clipper from https://blog.csdn.net/qq_42025798/article/details/121727781,

html look:

befor this modify, web clipper clipping code blocks, code blocks are in one line:

after this modify, web clipper clipping code blocks, code blocks are in multiline, readability is good:

…lines

laurent22 · 2024-03-15T10:15:46Z

We would need test units for this, but even then I don't understand what is being fixed at the moment

personalizedrefrigerator · 2024-03-15T21:34:01Z

what is being fixed at the moment

From the description, this seems to fix #5626 (closed due to inactivity).

Edit: For reference, tests for existing Webclipper HTML->MD logic can be found here.

wljince007 · 2024-03-15T23:10:27Z

Yes it is. this PR try to fix #5626, I well trying to add test units.

github-actions · 2024-03-16T08:51:01Z

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

wljince007 · 2024-03-16T09:08:53Z

@laurent22 Add test units packages/app-cli/tests/html_to_md/{code_multiline_1.html,code_multiline_2.html} run ok.

… delete code number lines

laurent22 · 2024-03-20T11:06:33Z

packages/turndown/src/commonmark-rules.js

+    // delete code number lines
+    code = code.replace(/(^-\s+\d+\s*$\r?\n)+/gm, '');
+


I'm concerned that this may also remove information that shouldn't be removed from certain code blocks. In the comment above could clarify exactly what strings are being removed? (as I'm unable to read that regex)

This regular expression removes consecutive lines of code that start with a - followed by a number. The regular expression marker /gm means that each line break is separated by a line, and can be perfectly matched with a line using(^ - \ s+\ d+\ s * $\ r? \ n)

Ref: RegExp.prototype.multiline

@laurent22 Match the lines:

Continuous lines of code starting with - followed by numbers are basically useless information.

It might make more sense to ignore code-block children of type ul with class pre-numbering. This should also match line numbers in the second example provided.

This would allow the web clipper (and perhaps also the Rich Text Editor) to save blocks that contain lists of numbers, while also removing line numbers.

Thank you for your patient guidance. I will try to modify the code using the method you mentioned.

@personalizedrefrigerator Already edit rules.list as your guidance. Test units run ok. Thanks a lot.

I'm concerned that this may also remove information that shouldn't be removed from certain code blocks.

@laurent22 This regular replacement code has been removed.

laurent22 · 2024-03-23T15:02:35Z

packages/turndown/src/commonmark-rules.js

+
+    content = content.replace(/\r?\n|\r/g, '\n')
+    // if code is multiline and in codeBlock, just return it, codeBlock well add fence(default is ```)
+    if (content.indexOf('\n') != -1 && node.parentNode && isCodeBlock(node.parentNode)){


Please always use strict inequality !==

Thanks, fixed, submitted.

personalizedrefrigerator · 2024-03-23T15:20:48Z

I've noticed that the web clipper has trouble with code blocks on Joplin's website (e.g. the table of contents plugin tutorial page). Do you know if this pull request would also fix processing of those code blocks?

(Thank you for taking the time to work on this!)

…st case

wljince007 · 2024-03-25T22:52:28Z

I've noticed that the web clipper has trouble with code blocks on Joplin's website

Add packages/turndown/src/commonmark-rules.js/rules.joplinOrgTokenLineSpan to fix this.

Copy first code blocks from: toc_plugin

html looks:

befor `rules.joplinOrgTokenLineSpan` added, on Joplin 2.13.15, save md looks:

after `rules.joplinOrgTokenLineSpan` added, save md looks:

laurent22 · 2024-03-27T18:52:12Z

packages/turndown/src/commonmark-rules.js

+// Fix: Web clipper has trouble with code blocks on Joplin's website.
+// See https://github.com/laurent22/joplin/pull/10126#issuecomment-2016523281 .
+// Web clipper clippering contents is all in oneline. The format features are: <pre ...><code ...><span class="token-line">. Span with class of "token-line" represent one line. 
+// Test case: packages/app-cli/tests/html_to_md/code_multiline_3.html .
+rules.joplinOrgTokenLineSpan = {
+  filter: function (node) {
+    const grandfather = node.parentNode.parentNode?? null;
+    return node.nodeName === 'SPAN' && node.getAttribute('class') === 'token-line' && grandfather && isCodeBlock(grandfather); 
+  },
+
+  replacement: function (content, node, options) {
+    content = content.replace(/\r?\n|\r/g, '');
+    // If content replaced '\r\n' is empty, it indicates that the line is empty line; if not, keep the leading before and after content.
+    return content === '' ? '\n\n' :  node.flankingWhitespace.leading + content + node.flankingWhitespace.trailing + '\n';
+  }
+}
+


I'm struggling to review this, I don't really understand what it does

@personalizedrefrigerator, was this for the bug you mentioned, and does the fix make sense to you?

@personalizedrefrigerator, was this for the bug you mentioned, and does the fix make sense to you?

This was the bug I mentioned, however I'm not sure that the fix makes sense to me. I think the issue here is that the  s at the end of Docusaurus code blocks aren't being converted to linebreaks by Turndown.

For context, I've attached simplified HTML for a Docusaurus code block (taken from here):

<code class="codeBlockLines_FJaf"> --- id : [...] [...] --- </code>

Observe that each (inline) token-line span ends with a   that previously wasn't detected by Turndown.

Remove packages/turndown/src/commonmark-rules.js/rules.joplinOrgTokenLineSpan. Analyze the current code, find the reasons for incorrect formatting in the code block, modify code for preserve the line break formatting for code block. Web clipper clipping code blocks on Joplin's website run ok. Test case run ok.

wljince007 · 2024-04-01T01:38:42Z

One of the main goals of Joplin's 3.0 version development is to add mobile plugin functionality. Can we take this opportunity to add a user script type for ContentScriptType.TurndownPlugin? So that users can write plugins to affect the process of 'HtmlToMd'. I think this feature is missing in the current Joplin plugin system.

…ecial handling of code blocks

…ecial handling of code blocks, keep space

…ecial handling of code blocks, keep space, test case

personalizedrefrigerator

I've left a few comments based on a quick read through the code.

Thank you for working on this!

packages/app-cli/tests/html_to_md/code_multiline_1.html

packages/turndown/src/commonmark-rules.js

packages/turndown/src/turndown.js

packages/turndown/src/commonmark-rules.js

Coding style fix Co-authored-by: Henry Heino <46334387+personalizedrefrigerator@users.noreply.github.com>

fix typo Co-authored-by: Henry Heino <46334387+personalizedrefrigerator@users.noreply.github.com>

Optimize regular expressions: .replace(/( *\t*\n)+( *\t*)*$/, '') equivalent to .trimEnd() Co-authored-by: Henry Heino <46334387+personalizedrefrigerator@users.noreply.github.com>

wljince007 · 2024-04-09T13:46:40Z

We could discuss it but you might want to create a thread on the forum about it. If you could mention example plugins that could benefit from it that could help decide if we add this or not

Add Feature request: Add content script type of ContentScriptType.TurndownPlugin

personalizedrefrigerator

I've tested this locally with Joplin's website and it works!!!

I've left a few minor stylistic/commenting suggestions. For the most part, I think this is ready to merge!

packages/app-cli/tests/html_to_md/code_multiline_5.html

packages/turndown/src/commonmark-rules.js

packages/turndown/src/turndown.js

fix typo Co-authored-by: Henry Heino <46334387+personalizedrefrigerator@users.noreply.github.com>

better comments Co-authored-by: Henry Heino <46334387+personalizedrefrigerator@users.noreply.github.com>

add space Co-authored-by: Henry Heino <46334387+personalizedrefrigerator@users.noreply.github.com>

remove space Co-authored-by: Henry Heino <46334387+personalizedrefrigerator@users.noreply.github.com>

wljince007 · 2024-04-18T06:24:25Z

Thank you for the revision. My English proficiency is relatively weak and annotations are not well expressed. Thank you for your guidance.

laurent22 · 2024-04-20T12:49:12Z

Thanks for implementing this @wljince007 and thanks for the review @personalizedrefrigerator!

Desktop: when clipper, keep code in multiline and delete code number …

c9c324a

…lines

wljince007 changed the title ~~Desktop: when clipper, keep code in multiline and delete code number lines~~ Desktop: Fix #5626: when clipper, keep code in multiline and delete code number lines Mar 15, 2024

wljince007 force-pushed the csdn_clipper branch from c5cfa1a to c9c324a Compare March 16, 2024 08:53

add code multiline test units

4d618f1

wljince007 added 2 commits March 18, 2024 16:09

add code multiline test units

e727504

Desktop: Fix laurent22#5626: when clipper, keep code in multiline and…

031c040

… delete code number lines

laurent22 reviewed Mar 20, 2024

View reviewed changes

ignore code-block children of type ul with class pre-numbering

68b0963

wljince007 force-pushed the csdn_clipper branch from f7913ab to 68b0963 Compare March 21, 2024 03:16

add code blank

64ece4f

laurent22 reviewed Mar 23, 2024

View reviewed changes

wljince007 force-pushed the csdn_clipper branch from 18ded33 to 64ece4f Compare March 24, 2024 03:57

wljince007 added 2 commits March 24, 2024 11:57

always use strict inequality !==

57ab1b6

fix: web clipper has trouble with code blocks on Joplin's website, te…

1e9da84

…st case

wljince007 force-pushed the csdn_clipper branch from a52a6ab to 1e9da84 Compare March 25, 2024 13:38

add comments

88c703d

laurent22 reviewed Mar 27, 2024

View reviewed changes

wljince007 changed the title ~~Desktop: Fix #5626: when clipper, keep code in multiline and delete code number lines~~ Desktop: Fix #5626: When web clipper clipping code blocks, keep code in multiline and delete code number lines Mar 28, 2024

wljince007 added 4 commits April 2, 2024 15:47

fix: web clipper has trouble with code blocks on Joplin's website, sp…

c294b79

…ecial handling of code blocks

fix: web clipper has trouble with code blocks on Joplin's website, sp…

d5d1c59

…ecial handling of code blocks, keep space

fix: web clipper has trouble with code blocks on Joplin's website, sp…

8decaf5

…ecial handling of code blocks, keep space, test case

Keep empty lines in code blocks

f8803d2

wljince007 added 10 commits April 8, 2024 14:50

add test case

84e6ccc

add test case

f301a60

add test case

f5a016a

add test case

73c23f2

remove code block leading and trailing newlines

6cf3f72

add comments

9c91b54

add comments

1588572

remove

d6c8c90

add comments

071b472

Merge branch 'laurent22:dev' into csdn_clipper

15ef55b

personalizedrefrigerator reviewed Apr 9, 2024

View reviewed changes

wljince007 and others added 4 commits April 9, 2024 17:06

Update packages/turndown/src/commonmark-rules.js

c34faeb

Coding style fix Co-authored-by: Henry Heino <46334387+personalizedrefrigerator@users.noreply.github.com>

Update packages/turndown/src/turndown.js

67e0cb2

fix typo Co-authored-by: Henry Heino <46334387+personalizedrefrigerator@users.noreply.github.com>

Update packages/turndown/src/commonmark-rules.js

c814129

Optimize regular expressions: .replace(/( *\t*\n)+( *\t*)*$/, '') equivalent to .trimEnd() Co-authored-by: Henry Heino <46334387+personalizedrefrigerator@users.noreply.github.com>

Optimize regular expressions

3aaef30

personalizedrefrigerator self-requested a review April 12, 2024 17:02

wljince007 added 2 commits April 13, 2024 22:00

Merge branch 'laurent22:dev' into csdn_clipper

7c9f9cd

Merge branch 'laurent22:dev' into csdn_clipper

1b9a255

personalizedrefrigerator reviewed Apr 18, 2024

View reviewed changes

wljince007 and others added 5 commits April 18, 2024 14:16

Update packages/app-cli/tests/html_to_md/code_multiline_5.html

099d2e2

fix typo Co-authored-by: Henry Heino <46334387+personalizedrefrigerator@users.noreply.github.com>

Update packages/turndown/src/commonmark-rules.js

145ff6c

better comments Co-authored-by: Henry Heino <46334387+personalizedrefrigerator@users.noreply.github.com>

Update packages/turndown/src/commonmark-rules.js

aec121d

better comments Co-authored-by: Henry Heino <46334387+personalizedrefrigerator@users.noreply.github.com>

Update packages/turndown/src/turndown.js

948c49f

add space Co-authored-by: Henry Heino <46334387+personalizedrefrigerator@users.noreply.github.com>

Update packages/turndown/src/turndown.js

4be5dde

remove space Co-authored-by: Henry Heino <46334387+personalizedrefrigerator@users.noreply.github.com>

fix typo

5ce49a8

personalizedrefrigerator approved these changes Apr 19, 2024

View reviewed changes

laurent22 merged commit 294cc4a into laurent22:dev Apr 20, 2024
10 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Desktop: Fix #5626: When web clipper clipping code blocks, keep code in multiline and delete code number lines #10126

Desktop: Fix #5626: When web clipper clipping code blocks, keep code in multiline and delete code number lines #10126

wljince007 commented Mar 15, 2024 •

edited

laurent22 commented Mar 15, 2024

personalizedrefrigerator commented Mar 15, 2024 •

edited

wljince007 commented Mar 15, 2024

github-actions bot commented Mar 16, 2024 •

edited

wljince007 commented Mar 16, 2024

laurent22 Mar 20, 2024

wljince007 Mar 20, 2024 •

edited

wljince007 Mar 20, 2024

wljince007 Mar 20, 2024

wljince007 Mar 20, 2024

personalizedrefrigerator Mar 20, 2024 •

edited

wljince007 Mar 21, 2024

wljince007 Mar 21, 2024

wljince007 Mar 21, 2024

laurent22 Mar 23, 2024

wljince007 Mar 24, 2024

personalizedrefrigerator commented Mar 23, 2024

wljince007 commented Mar 25, 2024 •

edited

laurent22 Mar 27, 2024

laurent22 Mar 27, 2024

personalizedrefrigerator Mar 30, 2024 •

edited

wljince007 Apr 2, 2024 •

edited

wljince007 commented Apr 1, 2024

personalizedrefrigerator left a comment

wljince007 commented Apr 9, 2024

personalizedrefrigerator left a comment

wljince007 commented Apr 18, 2024

laurent22 commented Apr 20, 2024

		// delete code number lines
		code = code.replace(/(^-\s+\d+\s*$\r?\n)+/gm, '');

Desktop: Fix #5626: When web clipper clipping code blocks, keep code in multiline and delete code number lines #10126

Desktop: Fix #5626: When web clipper clipping code blocks, keep code in multiline and delete code number lines #10126

Conversation

wljince007 commented Mar 15, 2024 • edited

See an example, clipper from https://blog.csdn.net/qq_42025798/article/details/121727781,

html look:

befor this modify, web clipper clipping code blocks, code blocks are in one line:

after this modify, web clipper clipping code blocks, code blocks are in multiline, readability is good:

laurent22 commented Mar 15, 2024

personalizedrefrigerator commented Mar 15, 2024 • edited

wljince007 commented Mar 15, 2024

github-actions bot commented Mar 16, 2024 • edited

wljince007 commented Mar 16, 2024

Choose a reason for hiding this comment

wljince007 Mar 20, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

personalizedrefrigerator Mar 20, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

personalizedrefrigerator commented Mar 23, 2024

wljince007 commented Mar 25, 2024 • edited

html looks:

befor rules.joplinOrgTokenLineSpan added, on Joplin 2.13.15, save md looks:

after rules.joplinOrgTokenLineSpan added, save md looks:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

personalizedrefrigerator Mar 30, 2024 • edited

Choose a reason for hiding this comment

wljince007 Apr 2, 2024 • edited

Choose a reason for hiding this comment

wljince007 commented Apr 1, 2024

personalizedrefrigerator left a comment

Choose a reason for hiding this comment

wljince007 commented Apr 9, 2024

personalizedrefrigerator left a comment

Choose a reason for hiding this comment

wljince007 commented Apr 18, 2024

laurent22 commented Apr 20, 2024

wljince007 commented Mar 15, 2024 •

edited

personalizedrefrigerator commented Mar 15, 2024 •

edited

github-actions bot commented Mar 16, 2024 •

edited

wljince007 Mar 20, 2024 •

edited

personalizedrefrigerator Mar 20, 2024 •

edited

wljince007 commented Mar 25, 2024 •

edited

befor `rules.joplinOrgTokenLineSpan` added, on Joplin 2.13.15, save md looks:

after `rules.joplinOrgTokenLineSpan` added, save md looks:

personalizedrefrigerator Mar 30, 2024 •

edited

wljince007 Apr 2, 2024 •

edited