How to run milkdown parser and schema in Nodejs

[navanshu]

I am using the next version of milkdown the new api and trying to convert markdown file in nodejs to yDoc. Is this possible?

[Saul-Mirone]

I think you can give it a try. IMO the @milkdown/transformer doesn't rely on any browser API so it should be possible. If you meet any problem please come back and share.

[navanshu]

Hi, @Saul-Mirone I looked into @milkdown/transformer, and as far as I have understood to create a parser for markdown

import { ParserState, SerializerState } from '@milkdown/transformer';
import { schema } from '@milkdown/preset-gfm';
import remarkGFM from 'remark-gfm';

const remark = remarkGFM();

if (remark) {
 const parser = ParserState.create(schema, remark);
 const serializer = SerializerState.create(schema, remark);
}

But typescript throughs error that MilkdownPlugin[] is incomplatible with Schema<any, any>.

Also I checked the Schema type is quite complex while MilkdownPlugin are functions. Can you guide me on how I can use a function probably to generate schema that is same as the one that is generated by the combination of plugins in the browser.

[navanshu]

Hi @Saul-Mirone, I was working on running the milkdown-gfm parser and schema in nodejs. I was able to come till here.

import remarkGFM from 'remark-gfm';
import { ParserState, SerializerState } from '@milkdown/transformer';
import { schema, gfm } from '@milkdown/preset-gfm';
import * as Y from 'yjs';
import { prosemirrorToYDoc, yDocToProsemirror } from 'y-prosemirror';
import { Schema } from '@milkdown/prose/model';
import {} from '@milkdown/transformer/src/serializer/state.spec';

/** Build Schema?? */
const S = new Schema({
 nodes: {},
 marks: {}
});

const remark = remarkGFM() as import('unified').Transformer<
 import('mdast').Root,
 import('mdast').Root
>;
/** Build a processor?? */
const parser = ParserState.create(S, remark);
const serializer = SerializerState.create(S, remark);

export function CommonMarkToString(doc: Y.Doc) {
 const str = serializer(yDocToProsemirror(S, doc));
 return str;
}

export function CommonMarkToDoc(value: string) {
 return prosemirrorToYDoc(parser(value));
}

I still haven't figured out as to how do it provide it with the schema and remark.

[navanshu]

hi @Saul-Mirone I updated the above code to create a parser from the milkdowns code

import {
 emphasisSchema,
 inlineCodeSchema,
 linkSchema,
 strongSchema
} from '@milkdown/preset-commonmark/src/mark';
import {
 blockquoteSchema,
 bulletListSchema,
 codeBlockSchema,
 docSchema,
 hardbreakSchema,
 headingSchema,
 hrSchema,
 imageSchema,
 orderedListSchema,
 paragraphSchema,
 textSchema,
 listItemSchema
} from '@milkdown/preset-commonmark/src/node';
import {
 extendListItemSchemaForTask,
 tableCellSchema,
 tableHeaderSchema,
 tableRowSchema,
 tableSchema
} from '@milkdown/preset-gfm';
import { Schema } from '@milkdown/prose/model';
import { ParserState, SerializerState } from '@milkdown/transformer';
import {} from '@milkdown/transformer/src/serializer/state.spec';
import { remark } from 'remark';
import { prosemirrorToYDoc, yDocToProsemirror } from 'y-prosemirror';
import * as Y from 'yjs';

/** Build Schema?? */
const S = new Schema({
 nodes: {
  listItem: extendListItemSchemaForTask,
  // listItem: listItemSchema,
  table: tableSchema,
  table_row: tableRowSchema,
  table_cell: tableCellSchema,
  table_header: tableHeaderSchema,
  hr: hrSchema,
  doc: docSchema,
  text: textSchema,
  image: imageSchema,
  heading: headingSchema,
  code_block: codeBlockSchema
 },
 marks: {
  emphasis: emphasisSchema,
  link: linkSchema,
  strong: strongSchema,
  inline: inlineCodeSchema,
  hardbreak: hardbreakSchema,
  paragraph: paragraphSchema,
  blockquote: blockquoteSchema,
  bullet_list: bulletListSchema,
  ordered_list: orderedListSchema
 }
});

const R = remark();

/** Build a processor?? */
const parser = ParserState.create(S, R);
const serializer = SerializerState.create(S, R);

export function CommonMarkToString(doc: Y.Doc) {
 const str = serializer(yDocToProsemirror(S, doc));
 return str;
}

export function CommonMarkToDoc(value: string) {
 return prosemirrorToYDoc(parser(value));
}

But it is not able to parse the string.

[Saul-Mirone]

If all you want is just run milkdown in nodejs, have you considered to use Puppeteer?

Anyway, the code of creating the schema of milkdown is in https://github.com/Milkdown/milkdown/blob/9978182c9828d557c92bb387a5ab3745ed39b642/packages/core/src/internal-plugin/schema.ts, you can copy it try it out. But you need to understand how ctx working.

[navanshu]

• I don't want to run the editor itself, I want its parser and serialiser, so it can seamlessly work with the yjs and the files strings can be converted to YDoc in nodejs. That way I don't have to use yjs-websocket snappy storage.
• The ctx api is difficult to understand, I was not able to locate its source.

[navanshu]

Hi @Saul-Mirone, I manage to get till here, I think this is what you meant

import { remarkCtx, schema, schemaCtx } from '@milkdown/core';
import { Clock, Container, Ctx } from '@milkdown/ctx';
import { gfm } from '@milkdown/preset-gfm';
import { ParserState, SerializerState } from '@milkdown/transformer';
import {} from '@milkdown/transformer/src/serializer/state.spec';
import { prosemirrorToYDoc, yDocToProsemirror } from 'y-prosemirror';
import * as Y from 'yjs';

const clock = new Clock();
const container = new Container();
const ctx = new Ctx(container, clock);
// ctx.use(commonmark);

gfm.forEach((x) => x(ctx));

schema(ctx);
const S = ctx.get(schemaCtx);
const R = ctx.get(remarkCtx);

/** Build a processor?? */
const parser = ParserState.create(S, R);
const serializer = SerializerState.create(S, R);

export function CommonMarkToString(doc: Y.Doc) {
 const str = serializer(yDocToProsemirror(S, doc));
 return str;
}

export function CommonMarkToDoc(value: string) {
 return prosemirrorToYDoc(parser(value));
}

I am getting the error [MilkdownError]: Context "remark" not found, do you forget to inject it?

[navanshu]

Hi @Saul-Mirone, what do you think of this code

import {
 parser,
 prosePluginsCtx,
 remarkCtx,
 remarkPluginsCtx,
 remarkStringifyOptionsCtx,
 schema,
 schemaCtx
} from '@milkdown/core';
import { Clock, Container, Ctx } from '@milkdown/ctx';
import { commonmark } from '@milkdown/preset-commonmark';
import { gfm } from '@milkdown/preset-gfm';
import { ParserState, SerializerState } from '@milkdown/transformer';
import {} from '@milkdown/transformer/src/serializer/state.spec';
import remarkParse from 'remark-parse';
import remarkStringify from 'remark-stringify';
import { unified } from 'unified';
import { prosemirrorToYDoc, yDocToProsemirror } from 'y-prosemirror';
import * as Y from 'yjs';

const clock = new Clock();
const container = new Container();
const ctx = new Ctx(container, clock);

ctx
 .inject(prosePluginsCtx)
 .inject(remarkPluginsCtx)
 .inject(remarkStringifyOptionsCtx)
 .inject(remarkCtx, unified().use(remarkParse).use(remarkStringify));

commonmark.forEach((x) => x(ctx));
gfm.forEach((x) => x(ctx));

schema(ctx);
parser(ctx);

const S = ctx.get(schemaCtx);
const R = ctx.get(remarkCtx);

/** Build a processor?? */
const PR = ParserState.create(S, R);
const SR = SerializerState.create(S, R);

export function CommonMarkToString(doc: Y.Doc) {
 const str = SR(yDocToProsemirror(S, doc));
 return str;
}

export function CommonMarkToDoc(value: string) {
 return prosemirrorToYDoc(PR(value));
}

[Saul-Mirone]

https://stackblitz.com/edit/node-ltifxb?file=index.js

[navanshu]

Thanks that was great. It would have taken me days to figure that out.
I was finally able to run it with gfm and all the yjs schema errors disappeared.
I finally got to try the new Version.

Some plugins like the checkbox are having problems because of the \ escape character.
Is it because of the new Composable API or this is a issue of Schema itself?

[Saul-Mirone]

Can you provide further details about the problem? Like what the markdown is, how it looks in view and what do you get from the serializer.

[navanshu]

Milkdown editors multiple features like

• Checkbox
• Table
• Code

Are not working or working incorrectly, in Microsoft's Edge browser

See the escape character being added before special characters in edge

While

• Table

Is not working in the chrome browser. I tried the Playground on the Milkdowns website and locally.
Same escape character

[Saul-Mirone]

It's not design to work like this, you should use inputrules such as |3x2|<space> to create tables, or using - [ ]<space> to create task lists.

[navanshu]

@Saul-Mirone I checked the table works as you told. But the checkbox is not working in microsoft edge browser.

HTML and Markdown in Edge browser

### Code generated in Edge
*   \[ ] Apple

<ul data-spread="false">
    <div data-node-view-root="true">
        <li class="flex flex-column items-start gap-2 ">
            <span class="flex h-6 items-center">
                   <span class="bg-nord8 dark:bg-nord9 h-2 w-2 rounded-full"></span>
             </span>
            <div class="min-w-0">
                <div data-node-view-content="true" style="white-space: inherit;">
                    <p>[ ] Apple</p>
                </div>
            </div>
        </li>
    </div>
</ul>

HTML and Markdown in Chrome browser

### Code generated in Chrome
*   [ ] Apple

<ul data-spread="false">
    <div data-node-view-root="true">
        <li class="flex flex-column items-start gap-2 ">
            <span class="flex h-6 items-center">
                <input class="form-checkbox rounded" type="checkbox">
            </span>
            <div class="min-w-0">
                <div data-node-view-content="true" style="white-space: inherit;">
                    <p>Apple</p>
                </div>
            </div>
        </li>
    </div>
</ul>

I tried checking the preset-gfm for possible issues, but it goes over my head.