Syntactic synchronization, automatic parentheses, metadata in Ast

[kazcw]

Pull Request Description

• Synchronize Y.Js clients by AST (implements AST-based synchronization #8237).
• Before committing an edit, insert any parentheses-nodes needed for the concrete syntax to reflect tree structure (fixes Set component doesn't work in GUI2 #8884).
• Move externalId and all node metadata into a Y.Map owned by each Ast. This allows including metadata changes in an edit, enables Y.Js merging of changes to different metadata fields, and will enable the use of Y.Js objects in metadata. (Implements Keeping metadata as part of AST. #8804.)

Important Notes

• Metadata is now set and retrieved through accessors on the Ast objects.
• Since some metadata edits need to take effect in real time (e.g. node dragging), new lower-overhead APIs (commitDirect, skipTreeRepair) are provided for careful use in certain cases.
• The client is now bundled as ESM.
• The build script cleans up git-untracked generated files in an outdated location, which fixes lint errors related to src/generated that may occur when switching branches.

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

• The documentation has been updated, if necessary.
• Screenshots/screencasts have been attached, if there are any visual changes. For interactive or animated visual changes, a screencast is preferred.
• All code follows the
  Scala,
  Java,
  and
  Rust
  style guides. In case you are using a language not listed above, follow the Rust style guide.
• All code has been tested:
  • Unit tests have been written where possible.
  • If GUI codebase was changed, the GUI was tested when built using ./run ide build.

[kazcw]

(Stacked on #8891).

[farmaazon]

Shouldn't NodeMetadataFields also inherit MetadataFields? Nodes have externalId too.

[kazcw]

I'm not sure. Although all nodes are Asts, I wouldn't consider the externalId part of the node metadata. Their updates are currently handled separately, and in practice IDs have to be handled a bit specially--if an Ast has only one concrete child, that child should inherit the parent's externalId (this will be more important once we have AssignmentExpression nodes as a stable place to attach node metadata).

I'm not opposed to making MetadataFields the root of a metadata hierarchy if we find it would simplify anything, but currently (with the little metadata we have so far) I'm not seeing that to be the case.

[somebody1234]

approving the ide-desktop changes, which changes the bundled format for the Electron entrypoint code from CJS to ESM. i haven't tested though, so make sure Electron still works fine for both ide and ide2.

[somebody1234]

is there a reason why we need the client to be ESM, btw?

[kazcw]

is there a reason why we need the client to be ESM, btw?

@somebody1234
The client now transitively depends on the WASM parser (because it's now used by the ydoc-server). With ESM bundling, we can use top-level await in the FFI module to allow other modules to use the WASM module safely--otherwise we would have to be careful to explicitly await the parser being loaded before doing anything that uses it.

[kazcw]

Update (@farmaazon): The only blocker left is packaging. Here's what I've found:

• We should bundle the WASM module as an asset and fetch it--that way it should work the same way in Vitest (node) / Electron (node, but with different restrictions on module loading) / browsers. The current runtime-checked approach doesn't work in Electron because Electron doesn't allow node modules to be imported dynamically. [Switching between different static WASM wrappers with export conditions seem like a viable alternative, but not any simpler I think.]

To get top level await working by switching to ESM bundling:

• The main configuration changes are in this PR
• We'll also need to upgrade to Electron 28 (to load ESM)
• We might have to configure vite differently when building for Electron: Disable the topLevelAwait plugin and set build.target to something like esnext that lets native top-level await be used without polyfill.

Alternatively instead of top-level await we could explicitly await the parser loading, but we'd have to do so in various places--when starting the ydocserver, the app/mock, and tests that need the parser.

[farmaazon]

The problem with switching to ESM is our ide-desktop/lib/client dependencies: many of them have require statements which esbuild does not handle well.

Trivial to replace:

• opener
• mime-types
• portfinder

Harder to replace:

• create-servers
• tar