-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement initial LaTeX support with Prosemirror-Math #36
Conversation
Great! I'll mostly let @mrtcode review this, but some initial notes:
A larger problem is figuring out what to do for HTML export, rich-text output (which is HTML copied to the clipboard as rich text), and word processor integration ("Add Note"). If we don't do anything, you'd get the raw equation input. HTMLFor HTML export, we could just document/provide the prosemirror-math/KaTeX JS and CSS that you'd need to add to your document to render these properly. I know it's from prosemirror-math, but I'm not sure if using Word processor outputThe bigger problem, I think, is word processor integration. For this to be useful, you really should be able to use Add Note and see the rendered equations in your Word document, but it seems like KaTeX doesn't support SVG output (which we could convert to PNG after rendering to Canvas with something like canvg). MathJax does support SVG output, but prosemirror-math can't use MathJax: "In the future, prosemirror-math will also accept a custom callback that can be used to invoke alternative renderers like MathJax." It's possible we could render in a hidden browser and take screenshots of each element, but that's getting pretty dicey. We should probably see if anyone has gotten prosemirror-math to work using MathJax, or look into what it would take to add support for that ourselves. If all fails, we could look into the hidden browser + screenshot option… Rich textUnclear. Could just leave it as the HTML export, maybe with the $ delimiters so that it's easier to render with something later. But ultimately I suspect many people would prefer the option to have inline images, same as with the word processor output. |
So before moving further, I think the main question is if it's acceptable for us to load KaTeX libraries everywhere to show them rendered, or not. Because that determines if we have to render and store images and how much we need to modify prosemirror-math. Notes can be exported/displayed in:
So it looks that rendered images are necessary for importing/copying into word processor, but there is one potential issue — rendered images might not be inserted into word processor in a way we expect. I'm mostly talking about inline math, while math blocks are probably fine. The result can be very chaotic. Which makes me question if we want rendered images at all. What people really want in word processors is probably active LaTeX objects. And I'm not sure if we can do that.
Just fetch and checkout this branch into Zotero client note-editor submodule, and build Zotero client as usually. # Assuming you are in note-editor dir
npm run build
rsync -a build/zotero/ .../zotero-client/build/resource/note-editor/ It builds and copies note-editor directly into Zotero client build directory and then you can just rebuild and run Zotero. More observations:
|
Thank you @mrtcode and @dstillman for your detailed feedback. I have started tackling a few of these issues today and will detail my findings, as well as some questions, tomorrow. Thanks, Sawyer |
Now when I think more about this, there isn't any good way to export inline math into word processors. Many small little images in separate lines won't be useful. I'm wondering what are people expectations here. Should we just do block level math? But that feels too limited. @dstillman maybe we should ask in Zotero forums about this? |
What's the issue? Just that the vertical alignment and line spacing are likely to be thrown off by inline images? It looks like Word supports LaTeX math input, so I imagine we could just create those using the plugin when using Add Note. LibreOffice and Google Docs also support equations, but it doesn't seem like they support LaTeX input natively. If we could actually create real equation objects in Word, and you can use external libraries for raw HTML export, I'd be fairly comfortable just outputting the raw LaTeX with delimiters the rest of the time. You'd either render it using some external tool or you'd manually recreate those in your document. Add Note is already a one-time operation, so if you have to go through and manually rewrite those using your word processor's preferred math syntax, so what? |
@mrtcode @dstillman I will definitely do some more research into how Word supports LaTeX. I also want to see if exporting LaTeX objects as SVGs would work or if PNGs are better. It seems that there will be times when people want to export the LaTeX code itself but other times when they want rendered images. Although, I think exporting images is a more common use case. Does "Add Note" in Word use the DOMSerializer from buildClipboardSerializer() or buildToHTML() or neither? I've made some changes so far...
If the build system were changed, the static fonts could also be transfered into the build from node_modules/ using Webpack. Additionally, when you type |
"Add Note" uses saved note HTML directly from note item. note-editor isn't involved here. Anyway, good job, I'll review this soon. |
Are we sure that we don't want
I doubt there exist a way to preserve images inline when copying HTML into word processors. |
The difference here is that there are regular sentences that would trigger this for people who don't know anything about math input, whereas people are by now fairly used to automatic Markdown in various programs (e.g., Slack). This probably wouldn't come up all that often, but it seems like it would be fairly confusing when it happened. MathJax doesn't define these by default for this reason:
(GitHub has a more technical audience that would be used to unexpected conversions when switching modes.) But my suggestion of using So maybe we only support
You mean for raw HTML output? (I suggested we would include them for Rich Text.) Yeah, I guess that makes sense. A little redundant with defined classes, but it's easier to use the auto-render extensions without needing to programmatically render specific elements.
The ones here seem to work, copied to Word: https://en.wikipedia.org/wiki/Wikipedia:Manual_of_Style/Images#MOS:BORDERIMAGE |
src/core/schema/nodes.js
Outdated
content: "text*", | ||
atom: true, | ||
code: true, | ||
toDOM: () => ["pre", 0], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should use an additional class for math_display
i.e. class="math"
. We should also move math_display
before codeBlock
, to avoid codeBlock
grabbing all <pre>
nodes.
src/core/schema/nodes.js
Outdated
inline: true, | ||
atom: true, | ||
toDOM: () => ["span", 0], | ||
parseDOM: [{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
class="math"
is necessary here as well, because all <span>
will be interpreted as math otherwise.
src/core/plugins/menu.js
Outdated
@@ -217,6 +217,7 @@ class Menu { | |||
let insideBlockquote = nodeIsActive(state, schema.nodes.blockquote); | |||
this.paragraph = this.buildBlock(schema.nodes.paragraph, {}, insideList || insideBlockquote); | |||
this.codeBlock = this.buildBlock(schema.nodes.codeBlock); | |||
this.math_display = this.buildBlock(schema.nodes.math_display); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Creating math_display
doesn't properly focus the node and you have to click it again. Unclear why. Maybe we need to programmatically move selection position into the math node. I'll try to find later an example.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe insertMathCmd could help here.
@@ -14,11 +14,12 @@ | |||
"ff >= 60.9.0" | |||
], | |||
"dependencies": { | |||
"@benrbray/prosemirror-math": "^0.2.2", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add KaTeX package.
Well, we could trigger input rule only if
You are right, and that works in other word processors as well. |
@sawyerpollard I think |
I would say neither, for now. We look into creating equations via the Word plugin for Add Note, and beyond that we add delimiters to output and people can post-process or manually replace in their word processor. |
I guess that's fine. And then if your post-processor converts other things, that's a problem for you to fix in your post-processor. |
As we decided to serialize math nodes with surrounding The code below can be added to math_display: {
...
toDOM: node => {
return ['pre', {
class: 'math'
}, '$$' + node.textContent + '$$']
},
parseDOM: [{
tag: "pre.math",
getContent(dom) {
let text = dom.textContent;
text = text.trim();
if (text.slice(0, 2) === '$$' && text.slice(-2) === '$$') {
text = text.slice(2, -2);
}
return Fragment.from(schema.text(text));
}
}]
...
} Now when we'll do the above, we no longer need But we need to include else if (node.type === schema.nodes.math_display) {
text += blockSeparator + '$$' + node.textContent + '$$' + blockSeparator;
return false;
} And then implement all the same code for Also do we want to use the additional paste rules that |
Thank you for your detailed comments on the current implementation.
I changed the inline math input rule to I spent a lot of time doing a deeper dive into the ProseMirror Selection, ResolvedPosition, Transaction, etc classes. Although the insertMathCmd function wasn't integrating how I'd like, I wrote some code to activate the math block on insertion. It's a little weird if one inserts a math block in the middle of a line, but it's not breaking. Also, if one selects text and then inserts a math block, the text disappears. I believe that with the setBlockType function, the text was actually being included in the LaTeX block. This is something I can investigate further if necessary. Finally, I think it would be nice to include the provided Wikipedia parse rules. I created a PR on prosemirror-math to see if the author wants to export them, which would be nice. Alternatively, I can just copy the code over directly. I have been playing with options in regards to "Add Note" integration but haven't finalized any of those implementations but will keep you updated. Thanks, |
src/core/schema/nodes.js
Outdated
}, '$$' + node.textContent + '$$'], | ||
parseDOM: [{ | ||
tag: 'pre.math', | ||
getContent(dom) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I forgot to write it the last time, but it should be getContent(dom, schema) {
as described here.
We should also add import { Fragment } from 'prosemirror-model';
at the top.
src/core/schema/nodes.js
Outdated
if (text.slice(0, 2) === '$' && text.slice(-2) === '$') { | ||
text = text.slice(2, -2); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some tweaks here are necessary.
src/core/commands.js
Outdated
else if (node.type === schema.nodes.math_inline) { | ||
text += '$' + node.textContent + '$'; | ||
return false; | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If I try to copy two paragraphs (one with inline math, and another with just text) and paste into a plain text editor, they appear in a single line.
src/stylesheets/darwin.scss
Outdated
|
||
.math-node.empty-math .math-render::before { | ||
content: "Click to insert LaTex ..."; | ||
color: black; | ||
} | ||
|
||
.math-node .math-render.parse-error::before { | ||
content: "Cannot parse LaTeX."; | ||
cursor: help; | ||
} | ||
|
||
.math-node .ProseMirror-focused { | ||
outline: none; | ||
} | ||
|
||
math-display { | ||
position: relative; | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess we can just move all the math related SCSS into components/core/math.scss
. From both darwin.scss
and generic.scss
.
Yes, we should do that.
ProseMirror learning curve is really steep. That's unrealistic to learning it in a short time span. If you can't find a solution on something in ProseMirror reference manual, their forums or other PM based projects in Github, just ask it here.
I think we can just copy the code and keep a reference to it. It's already a plugin. Any idea why after a math node there is a new line that can’t be deleted? This happens only on Chrome for me. Doesn't happen in their demo. |
Is this after an inline math node made with a single Or is this a separate issue with math blocks made with |
Yes, it seems this is the same issue. I'll look into that. Edit: This is fixed now. |
3e6d1ab
to
75130ba
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We need to increment note-editor version here and later we'll need to do the same in Zotero client code, when we merge this PR. This is necessary to force older Zotero clients to open newer schema notes in read-only mode, to prevent removing unknown math nodes.
Next, @dstillman said he wants Shift-Enter to escape from math node, but prosemirror-math
already has Ctrl-Enter for that, and even has Ctrl-Backspace to delete the whole math node, although Ctrl- possibly feels less common for macOS auditory. Let's see what he'll say.
src/core/input-rules.js
Outdated
@@ -57,7 +57,7 @@ export function buildInputRules({ enableSmartQuotes }) { | |||
}), | |||
linkInputRule(), | |||
makeBlockMathInputRule(/^\$\$\s+$/, schema.nodes.math_display), | |||
makeInlineMathInputRule(/\$(.+)\$/, schema.nodes.math_inline), | |||
makeInlineMathInputRule(/\$\s((?:[^$\s]|[\s](?!\$))+)\s?\$$/, schema.nodes.math_inline), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it should work with $f(x)=5$
— a space after the closing $
and non obligatory spaces in between $
. Well, not only a space after $
but also other symbols used in sentences like comma, dot, etc. Github is doing this. I still think this is a good idea, because unlike in Github you can easily revert this with Cmd-z. We are doing a similar thing for 1.
input rule to start a list, and I don't see why $
can be more annoying than that.
It seems some code disappeared after the rebase. |
@mrtcode Hi there, I have been testing the Wikipedia math paste rules and so far it's not working that smoothly. It seems that Wikipedia is sometimes able to paste images of their math, however. I don't think this is a critical feature though. There is a question of whether it makes sense to have inline math and block math. You should technically be able to do anything with inline math that you could with block math, because block math ignores new lines. Excited to receive your feedback! |
I get an error when typing Probably
|
Aside from the issue above, this works great.
Ok, we can ignore this feature.
Yes, in theory you can do that, but:
Of course, if you are inserting a note into a word processor, the result is identical. |
@mrtcode Thanks for the feedback. I've corrected that error and refactored the code in the process. I hope it's not overstepping to create a single |
Ok, I think now @dstillman can try this out for the final review. And once we are ready, we should pick an appropriate time to release this, because we're increasing schema version number and therefore non-beta users will start seeing read-only notes. |
And there's no way to only increase the schema version when using a feature that requires it, or something like that? |
That would introduce too much complexity. And we already need to be very careful with all schema changes because that can result in data loss. |
Same problem with not allowing math in group notes in the beta, and using the earlier schema? I just don't think we can leave a beta up for more than a day or two if it's creating notes that say to upgrade, when there won't be upgrades available. |
I'm still afraid of the complexity in this sensitive part. One option would be to bundle and use an older version of note-editor, but, again, doesn't feel worth to do this. We don't do schema updates often, so probably better to test this properly between us. Another possibility is to silently introduce only the schema change. Push to production. And then introduce the feature for the beta. |
That sounds good. |
All right, so I guess @sawyerpollard should extract |
@dstillman, as @sawyerpollard suggested, do you think there would be any sense to only leave inline math?
|
I think we want block. It seems to be what everyone does. |
78761fa
to
a6129c9
Compare
I'm feeling good about the state of the PR now. The only thing that slightly bugs me is that I need to append |
a6129c9
to
0376fa7
Compare
0376fa7
to
0c6126f
Compare
Ok, merged, this is going to be great. |
Hello!
This is a draft pull request for potentially adding math support to the Zotero Note Editor. This implementation uses Benjamin Bray's excellent prosemirror-math package.
prosemirror-math
ConsiderationsProgress
This package provides support for both inline math nodes and block math nodes, although I have only gotten block math nodes fully working.
Here are block math nodes:
![Screen Shot 2022-06-28 at 12 46 05 AM](https://user-images.githubusercontent.com/19192015/176094940-a3167019-cccf-4b5a-9a0b-8a0433508e63.png)
![Screen Shot 2022-06-28 at 12 46 20 AM](https://user-images.githubusercontent.com/19192015/176094986-2e4eb1af-a979-4f12-a341-cca2456210df.png)
The KaTeX and prosemirror-math CSS files are imported from node_modules/ into SCSS with the @import and tilde (~) syntax.
prosemirror-gapcursor was updated to meet prosemirror-math's dependency requirements.
The editor Schema was updated to accept math_display nodes.
Issues
Inline math nodes work great, except, after insertion, the cursor appears to move to the next line, although it continues in the correct place after the user types. This issue, not present in the prosemirror-math demo, is critical to fix. I have tried adjusting many options, tweaking CSS, and inspecting the DOM, but I am still working on a fix.
KaTeX requires 60 fonts, totaling 1.1 MB, to support the full LaTeX feature set. In order to prevent the initial load delay of these fonts, it seems that Webpack's url-loader can inline font URLs into CSS to lazily load them.
<math-display>
HTML tags are used for the math nodes. "math-display" is considered an "unknown HTML tag" by the spec and is widely supported. The browser falls back to rendering it as an inline element.Ultimately, I believe a more elegant solution is to use inline math nodes within paragraph nodes.
I think that I need some clarification on the different ways that people can export notes. How can they be exported as HTML in Zotero, for example? And can they be exported in other formats?
Additionally, rich text is correctly copied from the note editor into other word processors. A clipboardTextSerializer can be included to copy LaTeX code delimited by dollar signs for example.
Additional Thoughts
I would love to try out this current implementation in the Zotero client. Is there a workflow for compiling Zotero with a modified note-editor included? Is it just a matter of pointing the submodule in Zotero to a different commit.
Thank you for your work on this project!