Split formatted markdown strings with unicode support. #4470

sidharthv96 · 2023-06-09T11:22:12Z

📑 Summary

Continuation of #4416

📏 Design Decisions

This will handle graphemes using Intl.Segmenter if available. (Not available in FF)
It also handles lines word by word instead of per character. So performance should be better as we are doing a lot less DOM operations.

📋 Tasks

Make sure you

📖 have read the contribution guidelines
💻 have added unit/e2e tests (if appropriate)
📓 have added documentation (if appropriate)
🔖 targeted develop branch

…maid into sidv/splitUnicode * 'sidv/splitUnicode' of https://github.com/mermaid-js/mermaid: Update docs

* develop: Update docs Rename info to note Rename "info" to "note" Update all patch dependencies Fix Directives Documentation Update tutorial link Run build Fix link to Tutorials from n00b-overview page UPdated version to 10.2.3 Remove old changelog Remove old changelog Setting version to 10.2.2 #4446 Improved regular expression #4446 Updating the cleanup criteria #4438 Reverted to the changes from #4285 Fix download Fix compile error in docs. Fix Contributor link in homepage Update docs Add hint on "flowchart" and "graph" (and some more styling)

* develop: Update coveralls Ignore bundlephobia Run docs:build Correct timeline spelling

* develop: Disable coveralls

* develop: (293 commits) chore: Remove lint warnings in example-diagram chore: Reduce codecov pushes Turn off codecov project status check build(docs): fix links to `config.schema.json` ci(lint): check if MermaidConfig types are in sync docs: add link to mermaid config docs in sidebar test(config): add temp test for defaultConfig build(types): create types from config JSON Schema build(types): add script to generate Config types build(docs): build JSON Schema docs build: use vite to get default mermaid config feat: add Mermaid Config in JSON Schema format docs: add support for ```regexp``` code blocks test: test partial QuadrantChartConfig options test: fix types in `config.spec.ts` style: fix lint issues in src/config.spec.ts test: rename src/config.spec.js to config.spec.ts fix(quadrant): make quadrant options optional fix lint update homepage community link ...

codecov · 2023-07-06T10:36:14Z

Codecov Report

Merging #4470 (651bc98) into develop (509a580) will increase coverage by 3.10%.
The diff coverage is 69.23%.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #4470      +/-   ##
===========================================
+ Coverage    73.27%   76.37%   +3.10%     
===========================================
  Files          143      135       -8     
  Lines        14535    13764     -771     
  Branches       516      550      +34     
===========================================
- Hits         10650    10512     -138     
+ Misses        3774     3143     -631     
+ Partials       111      109       -2

Flag	Coverage Δ
e2e	`83.59% <69.23%> (+4.76%)`	⬆️
unit	`45.32% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
packages/mermaid/src/rendering-util/splitText.ts	`62.74% <62.74%> (ø)`
packages/mermaid/src/rendering-util/createText.ts	`87.34% <91.66%> (ø)`
...mermaid/src/rendering-util/handle-markdown-text.ts	`97.22% <100.00%> (ø)`

... and 34 files with indirect coverage changes

sidharthv96 · 2023-07-06T16:09:11Z

Review request: @MikeJeffers :)

nirname

I do not fully understand what is happening here, I'm relatively new to the code base, so I suppose I cannot submit reasonable review for it

packages/mermaid/src/rendering-util/createText.ts

aloisklink

This isn't a full review right now, since I don't have much time, but from what I've seen so far, this PR looks great 👍.

However, there's one major flaw: there's no tests for environments where Intl.Segmenter is not available (very important since Firefox doesn't support it).

In order to fix this, can we somehow mock that Intl.Segmenter doesn't exist in Vitest and/or Cypress?

Or, we could just make Cypress run in the Firefox browser too, but that will make our E2E tests very very slow.

packages/mermaid/src/rendering-util/createText.ts

packages/mermaid/src/rendering-util/splitText.ts

MikeJeffers

Speaking to the improvements over the original fix - I approve and appreciate your work improving the initial fix! But I don't want to overstep in prematurely approving the whole PR as a lot here has changed that others might need to speak to first.

aloisklink

Thanks for adding some tests for when Intl.Segmenter isn't available! I agree, the Cypress tests would be nice to have, but having the tests in Vitest is good enough to get some code-coverage!

One last thing, it might be worth adding some info in the comments in functions that use Intl.Segmenter, since they use the default locale, which may return slightly different results on some locales (e.g. Bengali).

packages/mermaid/src/rendering-util/splitText.spec.ts

packages/mermaid/src/rendering-util/splitText.ts

packages/mermaid/src/rendering-util/types.d.ts

packages/mermaid/src/rendering-util/splitText.ts

Co-authored-by: Alois Klink <alois@aloisklink.com>

Yokozuna59 · 2023-07-23T21:33:04Z

Shouldn't this also resolve #1900 and #1925?

sidharthv96 · 2023-07-25T13:34:12Z

Shouldn't this also resolve #1900 and #1925?

These seems to be fixed already.

Already checked in lint.

Co-authored-by: Alois Klink <alois@aloisklink.com>

…maid into sidv/splitUnicode * 'sidv/splitUnicode' of https://github.com/mermaid-js/mermaid: Update packages/mermaid/src/rendering-util/types.d.ts

[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [mermaid](https://togithub.com/mermaid-js/mermaid) | [`10.2.4` -> `10.3.0`](https://renovatebot.com/diffs/npm/mermaid/10.2.4/10.3.0) | [![age](https://developer.mend.io/api/mc/badges/age/npm/mermaid/10.3.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/npm/mermaid/10.3.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/npm/mermaid/10.2.4/10.3.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/npm/mermaid/10.2.4/10.3.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes <details> <summary>mermaid-js/mermaid (mermaid)</summary> ### [`v10.3.0`](https://togithub.com/mermaid-js/mermaid/releases/tag/v10.3.0): 10.3.0 [Compare Source](https://togithub.com/mermaid-js/mermaid/compare/v10.2.4...v10.3.0) #### What's Changed ##### Features - Sankey diagrams by [@nirname](https://togithub.com/nirname) in [mermaid-js/mermaid#4502 - Feature/1838 actor creation destruction by [@Valentine14th](https://togithub.com/Valentine14th) in [mermaid-js/mermaid#4466 - Vertical branches in Git Diagram by [@mastersibin](https://togithub.com/mastersibin) in [mermaid-js/mermaid#4639 - Use JSON Schema to define and document `MermaidConfig` by [@aloisklink](https://togithub.com/aloisklink) in [mermaid-js/mermaid#4112 - Remove the test checking whether the JSON Schema default config matched the old default config by [@aloisklink](https://togithub.com/aloisklink) in [mermaid-js/mermaid#4610 - Fixes support of the macro `ContainerQueue_Ext` for C4 diagrams definition. by [@kislerdm](https://togithub.com/kislerdm) in [mermaid-js/mermaid#4577 ##### Bugfixes - Make quadrant chart options TypeScript types optional by [@aloisklink](https://togithub.com/aloisklink) in [mermaid-js/mermaid#4602 - Remove double parsing by [@nirname](https://togithub.com/nirname) in [mermaid-js/mermaid#4587 - Fix flowchart tooltip typing bug by [@lishid](https://togithub.com/lishid) in [mermaid-js/mermaid#4562 - Bug/4590 allow notes identical to keywords by [@ibrahimWassouf](https://togithub.com/ibrahimWassouf) in [mermaid-js/mermaid#4597 - feat: allow specifying on which weekday a tickInterval should start by [@leinelissen](https://togithub.com/leinelissen) in [mermaid-js/mermaid#4634 - Split formatted markdown strings with unicode support. by [@sidharthv96](https://togithub.com/sidharthv96) in [mermaid-js/mermaid#4470 - fix: Mind maps handles `-` signs in node ids/text by [@knsv](https://togithub.com/knsv) ##### Chores - Remove all TypeScript enums and forbid them in ESLint by [@aloisklink](https://togithub.com/aloisklink) in [mermaid-js/mermaid#4580 - refactor accessibility by [@Yokozuna59](https://togithub.com/Yokozuna59) in [mermaid-js/mermaid#4551 - chore: Reduce codecov pushes by [@sidharthv96](https://togithub.com/sidharthv96) in [mermaid-js/mermaid#4604 - Run PR-labeler-config-validator only if config changes by [@sidharthv96](https://togithub.com/sidharthv96) in [mermaid-js/mermaid#4607 - chore(deps): update all minor dependencies (minor) by [@renovate](https://togithub.com/renovate) in [mermaid-js/mermaid#4624 - Update all patch dependencies (patch) by [@renovate](https://togithub.com/renovate) in [mermaid-js/mermaid#4566 - Update all patch dependencies (patch) by [@renovate](https://togithub.com/renovate) in [mermaid-js/mermaid#4581 - Rename workflow jobs by [@sidharthv96](https://togithub.com/sidharthv96) in [mermaid-js/mermaid#4574 - Removed unused code in state diagrams by [@nirname](https://togithub.com/nirname) in [mermaid-js/mermaid#4631 - chore(deps): update all patch dependencies (patch) by [@renovate](https://togithub.com/renovate) in [mermaid-js/mermaid#4623 - chore: remove unused `devDependency` on coveralls by [@aloisklink](https://togithub.com/aloisklink) in [mermaid-js/mermaid#4641 - Allow entity diagram attribute names to start with asterisk by [@ibrahimWassouf](https://togithub.com/ibrahimWassouf) in [mermaid-js/mermaid#4588 - Bug/4592 fix new line padding class diagram by [@ibrahimWassouf](https://togithub.com/ibrahimWassouf) in [mermaid-js/mermaid#4633 - Fix graph not loading when the img loads too fast or fail to load by [@pierrickouw](https://togithub.com/pierrickouw) in [mermaid-js/mermaid#4496 - convert `cypress/helpers/util.js` to ts by [@Yokozuna59](https://togithub.com/Yokozuna59) in [mermaid-js/mermaid#4552 - build(deps-dev): bump word-wrap from 1.2.3 to 1.2.4 by [@dependabot](https://togithub.com/dependabot) in [mermaid-js/mermaid#4652 - chore(deps): update all minor dependencies (minor) by [@renovate](https://togithub.com/renovate) in [mermaid-js/mermaid#4663 - chore(deps): update all patch dependencies (patch) by [@renovate](https://togithub.com/renovate) in [mermaid-js/mermaid#4662 ##### Documentation - Sankey: Remove duplicated examples by [@nirname](https://togithub.com/nirname) in [mermaid-js/mermaid#4595 - Release docs by [@sidharthv96](https://togithub.com/sidharthv96) in [mermaid-js/mermaid#4493 - Update latest news section by [@huynhicode](https://togithub.com/huynhicode) in [mermaid-js/mermaid#4495 - Fix Typo by [@ryru](https://togithub.com/ryru) in [mermaid-js/mermaid#4567 - Docs: add ChatGPT plugin blog post by [@huynhicode](https://togithub.com/huynhicode) in [mermaid-js/mermaid#4570 - Fix relative link to theme variables list by [@ibrahimWassouf](https://togithub.com/ibrahimWassouf) in [mermaid-js/mermaid#4573 - Fix docs:dev by [@sidharthv96](https://togithub.com/sidharthv96) in [mermaid-js/mermaid#4598 - Docs: update link - "Join the Community" by [@huynhicode](https://togithub.com/huynhicode) in [mermaid-js/mermaid#4601 - Support docs:dev in docker by [@nirname](https://togithub.com/nirname) in [mermaid-js/mermaid#4599 - docs(flowchart): add documentation on multiple nodes style by [@tomperr](https://togithub.com/tomperr) in [mermaid-js/mermaid#4600 - Avoid downloading avtars everytime on docs:dev by [@sidharthv96](https://togithub.com/sidharthv96) in [mermaid-js/mermaid#4603 - docs: Fix checkbox syntax by [@guilhermgonzaga](https://togithub.com/guilhermgonzaga) in [mermaid-js/mermaid#4646 - Fix the "Edit this page on GitHub" link in Vitepress documentation for the Mermaid Config pages by [@aloisklink](https://togithub.com/aloisklink) in [mermaid-js/mermaid#4640 - Support MERMAID_RELEASE_VERSION in docs. by [@sidharthv96](https://togithub.com/sidharthv96) in [mermaid-js/mermaid#4612 - Docs: update Latest News section by [@huynhicode](https://togithub.com/huynhicode) in [mermaid-js/mermaid#4655 - added Typora to integrations list by [@kgilbert78](https://togithub.com/kgilbert78) in [mermaid-js/mermaid#4666 - Docs: Corrects name of C4 link by [@Incognito](https://togithub.com/Incognito) in [mermaid-js/mermaid#4660 - Fix a typo by [@gjtorikian](https://togithub.com/gjtorikian) in [mermaid-js/mermaid#4396 #### New Contributors - [@ryru](https://togithub.com/ryru) made their first contribution in [mermaid-js/mermaid#4567 - [@ibrahimWassouf](https://togithub.com/ibrahimWassouf) made their first contribution in [mermaid-js/mermaid#4573 - [@kislerdm](https://togithub.com/kislerdm) made their first contribution in [mermaid-js/mermaid#4577 - [@leinelissen](https://togithub.com/leinelissen) made their first contribution in [mermaid-js/mermaid#4634 - [@pierrickouw](https://togithub.com/pierrickouw) made their first contribution in [mermaid-js/mermaid#4496 - [@mastersibin](https://togithub.com/mastersibin) made their first contribution in [mermaid-js/mermaid#4639 - [@kgilbert78](https://togithub.com/kgilbert78) made their first contribution in [mermaid-js/mermaid#4666 - [@Incognito](https://togithub.com/Incognito) made their first contribution in [mermaid-js/mermaid#4660 - [@gjtorikian](https://togithub.com/gjtorikian) made their first contribution in [mermaid-js/mermaid#4396 **Full Changelog**: mermaid-js/mermaid@v10.2.4...v10.3.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/levaintech/contented).  Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

sidharthv96 and others added 8 commits June 9, 2023 11:06

Add splitText

c41df42

feat: split unicode properly

0a437f5

Update docs

d4edd98

Use splitLineToFitWidth function

7b46017

Merge branch 'sidv/splitUnicode' of https://github.com/mermaid-js/mer…

3f080e5

…maid into sidv/splitUnicode * 'sidv/splitUnicode' of https://github.com/mermaid-js/mermaid: Update docs

Cleanup

b3ce56c

Add logs

a379cd0

This was linked to issues Jun 13, 2023

Properly handle unicode when splitting words #4465

Closed

timeling bug for chinese #4469

Closed

sidharthv96 added 5 commits June 13, 2023 10:34

rename handle-markdown-text

5903792

Add types

dd4e146

Use joiner to split unicode

b36a017

Merge branch 'develop' into sidv/splitUnicode

25a85ee

* develop: Update coveralls Ignore bundlephobia Run docs:build Correct timeline spelling

Merge branch 'develop' into sidv/splitUnicode

445da58

* develop: Disable coveralls

sidharthv96 marked this pull request as draft July 6, 2023 04:11

sidharthv96 added 3 commits July 6, 2023 16:12

createText to TS

f548463

Handle proper formatting for markdown strings

60a93f7

Fix flowchart failure

5ac70bb

sidharthv96 marked this pull request as ready for review July 6, 2023 16:07

sidharthv96 requested review from aloisklink, jgreywolf, nirname, knsv and Yokozuna59 July 6, 2023 16:07

sidharthv96 added 2 commits July 6, 2023 21:48

Cleanup

7d996c3

Merge branch 'develop' into sidv/splitUnicode

b14bcda

nirname reviewed Jul 6, 2023

View reviewed changes

packages/mermaid/src/rendering-util/createText.ts Show resolved Hide resolved

packages/mermaid/src/rendering-util/createText.ts Outdated Show resolved Hide resolved

aloisklink requested changes Jul 6, 2023

View reviewed changes

packages/mermaid/src/rendering-util/createText.ts Show resolved Hide resolved

packages/mermaid/src/rendering-util/createText.ts Outdated Show resolved Hide resolved

packages/mermaid/src/rendering-util/splitText.ts Show resolved Hide resolved

sidharthv96 changed the title ~~Split unicode~~ Split formatted markdown strings with unicode support. Jul 6, 2023

MikeJeffers reviewed Jul 7, 2023

View reviewed changes

sidharthv96 added 2 commits July 7, 2023 10:02

Add tests without Intl.Segmenter

d58c41d

Add comments

28406fc

sidharthv96 requested a review from aloisklink July 7, 2023 04:38

aloisklink approved these changes Jul 8, 2023

View reviewed changes

Update packages/mermaid/src/rendering-util/splitText.spec.ts

eca2efa

Co-authored-by: Alois Klink <alois@aloisklink.com>

sidharthv96 and others added 3 commits July 25, 2023 18:55

Merge branch 'develop' into sidv/splitUnicode

ea192cc

Fix lint

68305fe

Update docs

4ea1227

sidharthv96 and others added 2 commits July 25, 2023 19:08

Remove redundant test.

409dedb

Already checked in lint.

Update packages/mermaid/src/rendering-util/types.d.ts

a7d4072

Co-authored-by: Alois Klink <alois@aloisklink.com>

knsv approved these changes Jul 25, 2023

View reviewed changes

sidharthv96 added 2 commits July 25, 2023 20:10

Add comments

841ae15

Merge branch 'sidv/splitUnicode' of https://github.com/mermaid-js/mer…

651bc98

…maid into sidv/splitUnicode * 'sidv/splitUnicode' of https://github.com/mermaid-js/mermaid: Update packages/mermaid/src/rendering-util/types.d.ts

knsv added this pull request to the merge queue Jul 25, 2023

Merged via the queue into develop with commit c99e1c6 Jul 25, 2023
20 of 21 checks passed

Yokozuna59 deleted the sidv/splitUnicode branch August 11, 2023 20:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Split formatted markdown strings with unicode support. #4470

Split formatted markdown strings with unicode support. #4470

sidharthv96 commented Jun 9, 2023 •

edited

codecov bot commented Jul 6, 2023 •

edited

sidharthv96 commented Jul 6, 2023

nirname left a comment

aloisklink left a comment

MikeJeffers left a comment

aloisklink left a comment

Yokozuna59 commented Jul 23, 2023 •

edited

sidharthv96 commented Jul 25, 2023

Split formatted markdown strings with unicode support. #4470

Split formatted markdown strings with unicode support. #4470

Conversation

sidharthv96 commented Jun 9, 2023 • edited

📑 Summary

📏 Design Decisions

📋 Tasks

codecov bot commented Jul 6, 2023 • edited

Codecov Report

sidharthv96 commented Jul 6, 2023

nirname left a comment

Choose a reason for hiding this comment

aloisklink left a comment

Choose a reason for hiding this comment

MikeJeffers left a comment

Choose a reason for hiding this comment

aloisklink left a comment

Choose a reason for hiding this comment

Yokozuna59 commented Jul 23, 2023 • edited

sidharthv96 commented Jul 25, 2023

sidharthv96 commented Jun 9, 2023 •

edited

codecov bot commented Jul 6, 2023 •

edited

Yokozuna59 commented Jul 23, 2023 •

edited