Is there some kind of optimized wire format for the syntax tree?

[skirsten]

Currently on my server-side I am doing something like this:

const processor = unified()
  .use(remarkParse)
  .use(remarkRehype, { allowDangerousHtml: true })
  .use(rehypeSlug);

return await processor.run(processor.parse(markdown));

then transfering it via JSON to the client side where I hydrate it to react:

const Content = useMemo(
  () =>
    unified()
      .use(rehypeReact, {
        createElement: React.createElement,
        Fragment: React.Fragment,
      })
      .stringify(props.content),
  [props.content]
);

Now the JSON is quite large:

HTML in JSON: 46162 bytes (12031 bytes brotli)
AST in JSON (the example above): 110108 bytes (16261 bytes brotli)

Is there any "wire format" or other way to better compress the AST before the JSON stringify?

[ChristianMurphy]

Unified and ASTs aren't really a wire protocol.
It is an AST parsing system.
Due to it's highly pluggable and dynamic design, its usage to process text documents (which usually don't have extremely complex trees, it is mostly text), and being JavaScript based; JSON is a good fit for its internal format.

Given all that marshaling data into a wire format like: ProtoBuff, Cap'n Proto, FlatBuffer, etc aren't likely to have a significant packet size or performance improvement over say gzip or brotli compressing JSON.
Because:

• Most of the content is text (which compression can shrink, but a wire protocol generally won't)
• Of the wrapping content, the AST metadata, it is dynamic enough (with new content types being addable by plugins) that it would be difficult to squeeze much out of the potentially compressible part (it would also likely end up plain text anyway)
• And marshaling in and out of another format over to JSON and back, is likely going to incur a performance cost vs reading and writing JSON as JSON.

Depending on your constraints around performance and network, any of:

• Fully server side rendering
• Sending the AST to the client
• Fully client side rendering

could work, and in all cases, applying compression may help.
But which option you go for, and the level of compression applied depend more on your use case.

[wooorm]

• I don’t think you’ll have problems unless you’re doing this at a giant scale
• What do you mean with the “HTML is JSON” datum? My guess would be that that instead is “HTML”?
• I’d say that compression does enough. And well, you can also send HTML down the wire ;)
• Lastly, React is working on improving streaming and splitting work between servers and clients (one part of which is a “wire protocol”), so, you can also wait a year or so and the React situation will magically improve :)

[skirsten]

Sorry, I meant HTML in JSON (basically this: {"content":"<h1>Hi...</h1>"}) vs AST as JSON ("content":{"type":"root","children":[...)

Thanks you two for the detailed responses.
Some background:

• On the client-side I want to use some custom components. That's why I switched from sending "HTML in JSON" to "AST as JSON".
• I am also server-side rendering this so I am not concerned about the initial page loading performance.
• What I am talking about here mostly affects the client side navigation and preloads.

Basically, I was just a bit shocked by the > 2x raw payload size and was wondering if there is some way to pack or optimize the AST on the server side and unpack it on the client side.

Most of the content is text (which compression can shrink, but a wire protocol generally won't)

Because of the > 2x raw increase it is obvious that (in this case) the AST is mostly non-text in comparison to the HTML which is mostly text (because the text stayed the same).

But I get it, premature optimization is the root of all evil. The current approach is working fine, I was just curious if there is a better way.