From 0306b106ea8c3b9e91ff4889bc7f3e1c672455cc Mon Sep 17 00:00:00 2001 From: Nicholas Bollweg Date: Tue, 14 Mar 2023 18:37:34 -0500 Subject: [PATCH] Adopting a text-based diagram syntax in Jupyter Markdown --- .gitignore | 3 + .../example.dot.svg | 38 ++ .../example.mermaid.svg | 2 + .../example.plantuml.svg | 1 + .../text-based-diagrams.md | 513 ++++++++++++++++-- 5 files changed, 509 insertions(+), 48 deletions(-) create mode 100644 xx-text-based-diagrams-in-markdown/example.dot.svg create mode 100644 xx-text-based-diagrams-in-markdown/example.mermaid.svg create mode 100644 xx-text-based-diagrams-in-markdown/example.plantuml.svg diff --git a/.gitignore b/.gitignore index 54ef78ed..9776b088 100644 --- a/.gitignore +++ b/.gitignore @@ -31,3 +31,6 @@ _build # Python files __pycache__ + +# Jupyter Server +.ipynb_checkpoints/ diff --git a/xx-text-based-diagrams-in-markdown/example.dot.svg b/xx-text-based-diagrams-in-markdown/example.dot.svg new file mode 100644 index 00000000..f46e1a15 --- /dev/null +++ b/xx-text-based-diagrams-in-markdown/example.dot.svg @@ -0,0 +1,38 @@ + + + + + + +LR + + + +chicken + +chicken + + + +egg + +egg + + + +chicken->egg + + + + + +egg->chicken + + + + + + diff --git a/xx-text-based-diagrams-in-markdown/example.mermaid.svg b/xx-text-based-diagrams-in-markdown/example.mermaid.svg new file mode 100644 index 00000000..366dd670 --- /dev/null +++ b/xx-text-based-diagrams-in-markdown/example.mermaid.svg @@ -0,0 +1,2 @@ + +
chicken
egg
 diff --git a/xx-text-based-diagrams-in-markdown/example.plantuml.svg b/xx-text-based-diagrams-in-markdown/example.plantuml.svg new file mode 100644 index 00000000..2a88ddd1 --- /dev/null +++ b/xx-text-based-diagrams-in-markdown/example.plantuml.svg @@ -0,0 +1 @@ +ChickenEggmany1 diff --git a/xx-text-based-diagrams-in-markdown/text-based-diagrams.md b/xx-text-based-diagrams-in-markdown/text-based-diagrams.md index 5ce71610..25cba114 100644 --- a/xx-text-based-diagrams-in-markdown/text-based-diagrams.md +++ b/xx-text-based-diagrams-in-markdown/text-based-diagrams.md @@ -1,84 +1,501 @@ --- -title: -authors: -issue-number: -pr-number: -date-started: +title: Adopting a text-based diagram syntax in Jupyter Markdown +authors: (alphabetically) Nick Bollweg (@bollwyvl) +issue-number: 100 +pr-number: 101 +date-started: 2021-03-14 --- # Summary -One paragraph explanation of the proposal. +This proposal encouraged _Project Jupyter_ to adopt a _text-based diagram_ format in +_Jupyter Markdown_. Easy-to-write diagrams should render consistently, accessibly, and +securely across flagship _Jupyter tools_ like _JupyterLab 4_, _Notebook 7_, _nbconvert_ +and _nbviewer_, in addition to the platforms _Project Jupyter_ uses to collaborate, like +_GitHub_, _HackMD_, and _Discourse_. This proposal recommends adoption of +[_MermaidJS_][mermaidjs], with the _fenced code block_ syntax inherited from +_GitHub-flavored Markdown_. + +[mermaidjs]: https://github.com/mermaid-js/mermaid # Motivation -Why are we doing this? What use cases does it support? What is the expected outcome? +Embedded _text-based diagrams_ within _Jupyter Markdown_ will provide a way to augment +plain- and rich-text narrative when working **in** _Jupyter tools_, as well as **on** +_Jupyter tools_, such as _GitHub_, _Discourse_, _HackMD_, and other common collaboration +platforms. + +Beyond Jupyter tools, selecting a widely-adopted implementation and syntax will require +less re-learning of syntax and skills. # Guide-level explanation -Explain the proposal as if it was already implemented and you were -explaining it to another community member. That generally means: +With the addition of [_MermaidJS_][mermaidjs] to _JupyterLab 4_, _Jupyter Notebook 7_, +_Jupyter _, and _Jupyter NBViewer_, _Jupyter_ joins +[GitLab](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/3711), +[HackMD](https://github.com/hedgedoc/react-client/issues/257), +[GitHub](https://github.blog/2022-02-14-include-diagrams-markdown-files-mermaid/) and +[many other platforms](https://mermaid.js.org/ecosystem/integrations.html) in natively +supporting a portable, interoperable _GitHub-Flavored Markdown_ (GFM) syntax for +_text-based diagrams_ to visually describe complex topics. + +## What can I draw? + +Today, you can start creating the following diagrams in _Jupyter Notebooks_ or _Jupyter +Markdown_ files: + +- _Class Diagrams_ that show the relationships between software components +- _Entity Relationship Diagram_ taht show the relationships between data models +- _Flowchart_ that show decision processes +- _Gantt Charts_ that show the relationship between tasks over time +- _Gitgraph (Git) Diagram_ that show the sometimes-bewildering history of complex + _distributed version control_ operations +- _Mindmaps_ that show the relationship between different ideas +- _Pie Charts_ that show the relative size of different values +- _Requirement Diagrams_ that show the chain of relationships from _what a user needs_ + to _what we should build_ +- _Sequence Diagrams_ that show the exchange of information between different people, + systems, or processes +- _State Diagram_ that show the possible states a system can be in, and how they might + change +- _Timeline Diagrams_ that show events over time +- _User Journey Diagrams_ that show the cycle of events over time + +...and they should _mostly_ render the same way everywhere. + +## How do I draw it? + +The [MermaidJS Documentation](https://mermaid.js.org/intro/) provides full examples of +all the supported diagram formats. But, mostly, try it out inside your _Jupyter tools_! -- Introducing new named concepts. -- Adding examples for how this proposal affects people's experience. -- Explaining how others should *think* about the feature, and how it should impact the experience using Jupyter tools. It should explain the impact as concretely as possible. -- If applicable, provide sample error messages, deprecation warnings, or migration guidance. -- If applicable, describe the differences between teaching this to existing Jupyter members and new Jupyter members. +## Why _text-based diagrams_? -For implementation-oriented JEPs, this section should focus on how other Jupyter -developers should think about the change, and give examples of its concrete impact. For policy JEPs, this section should provide an example-driven introduction to the policy, and explain its impact in concrete terms. +Unlike diagrams authored in heavyweight drawing programs, or as the output of code, +_text-based diagrams_ are easy to create by hand, and are usually readable when they +_can't_ be rendered fully. This helps _everyone_ understand visual concepts, including +those that make use of _assistive devices_ like _screen readers_. By doing this +rendering right in your browser, your ideas become visual as quickly as possible: +combined with the _Jupyter Collaboration_ features introduced in _JupyterLab 3.1_, and +refined in _JupyterLab 4_, this reactivity can be shared with your whole time. + +By choosing this widely-implemented, portable syntax, we hope the all-important _muscle +memory_ mastery developed while working in _Jupyter tools_ are applicable to other +digital platforms, and hopefully will improve the communication of _Project Jupyter_ +itself. + +## Why _MermaidJS_? + +While many other [_text-based diagram_](https://text-to-diagram.com) formats exist, +_MermaidJS_ enjoys: + +- broad adoption across many tools and platforms with a consistent syntax +- a relatively simple and readable format +- a wide number of software-oriented and general-purpose diagrams +- runs entirely within the browser # Reference-level explanation -This is the technical portion of the JEP. Explain the design in -sufficient detail that: +## _MermaidJS_ Details + +_MermaidJS_ is _free software_, distributed under the permissive +[MIT License](https://github.com/mermaid-js/mermaid/blob/develop/LICENSE) and adheres to +a +[code of conduct](https://github.com/mermaid-js/mermaid/blob/develop/CODE_OF_CONDUCT.md). + +The codebase has been actively publishing releases since 2014, with over 400 historical +contributors, with a relatively smaller core of active contributors. + +The `mermaid` package itself is distributed on +[`npmjs.com`](https://www.npmjs.com/package/mermaid) as well as _content delivery +networks_ (CDN). + +At install/run time, aside from its own assets, it entails a net addition of `dagre-d3` +and the full `d3` metapackage. Some advanced features use the `cytoscape` and `elkjs` +_rendering engines_, but are only loaded when needed. + +## _JupyterLab_ PR #1402 + +[jupyterlab/jupyterlab#1402](https://github.com/jupyterlab/jupyterlab/pull/14102) +proposes adding `mermaidjs 10` to the default _Markdown_ rendering engine, `marked`. As +_JupyterLab 4_ is the upstream of _Jupyter Notebook 7_, landing this PR would mean only +a few configuration files would have to change. + +### Implementation + +Implementing `mermaid` with _fenced code blocks_ means _every_ code block is passed +through an extra check for the `mermaid` syntax marker. When found for the first time, +the core `mermaid` bundle (and its transient dependencies) are lazily loaded, and each +extra kind of diagram loads an additional grammar. + +This increases the complexity of the implementation a fair amount, and uses +_asynchronous rendering_ more aggressively than was previously used. + +### Size + +While lazily-loaded, and having no appreciable impact on the _time-to-first +interaction_, these _MermaidJS_ assets are **not small** when installed, or when served +to the browser. The following table compares some relative sizes of the full +distribution of all possible files that _could_ be loaded by a _JupyterLab_ user: + +| | uncompressed (mb) | gzip (mb) | +| :----------- | ----------------: | --------: | +| `4.0.0a46` | 7.02 | 1.98 | +| `08f4d6e` | 9.76 | 2.78 | +| only `elkjs` | 1.43 | 0.41 | + +> - the _uncompressed_ column is what a user _typically_ sees when loading the +> application. +> - the _gzip_ column _roughly_ can be seen as the impact on the as-distributed wheel on +> PyPI + +`elkjs` is called out explictly, due to its size. Likely the state-of-the-art in terms +of _auto layout_ tools, it has potential _well_ beyond many other engines for complex +diagrams, but is currently hidden behind an experimental feature flag, almost certainly +not enabled on other platforms. It may be sensible to omit it from the initial +distribution. + +### Security + +To avoid _cross-site-scripting attacks_, the `marked` API is used with all available +security features, and further applies sanitiziation. Unfortunately, this forces the +delivery of diagrams as base64-encoded +[Data URIs](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs) +as the `src` attribute of `` tags. This means that all interactive _MermaidJS_ +features, like hypyerlinks, tooltips, and cross-diagram configurable styling, are +unavailable. + +An alternate approach could render these inside restrictive `