[Progress Discussion] Initial Progress Discussion

[MikeAliG]

Progress discussion #363

Hello, I hope you are well and Happy New Year. First of all, Thank you for waiting for the transition modules (@udecode/slate-plugins-common and @udecode/slate-plugins-test-utils) the work was taking a toll and thankfully over the Christmas I have some time to go over the transition. As for the progress, we have had great progress specially in terms of compartmentalisation and module control and it is almost ready for the initial commit but there's a problem with the new module build and I dug down into it but to no avail. The built files in the @udecode/slate-plugins-common and @udecode/slate-plugins-test-utils and their types are moved in a subfolder with the package name which you can find below:

This issue could be on the rollup for which I found the closest issue in here which is still open but what's interesting is the problem doesn't occur for none of the other packages and it just happens to happen for the new packages (@udecode/slate-plugins-common and @udecode/slate-plugins-test-utils) which is probably something I'm missing but I checked the rollup.config.js and the config seems fine but I have no idea this issue occurs so please let me know if you have any solution to this issue.

Have a great day,
Ali

[MikeAliG]

I also temporarily pushed the changes into modularisation branch if you'd like to take a look or if you'd like anything changed, please let me know, I'll be happy to change it.

[zbeyens]

Thanks! I'll take a look.

[zbeyens]

The problem was that common was importing files outside the package: core types. It's fixed by using the module import with:

"compilerOptions": {
    "paths": {
      "@udecode/slate-plugins-core": ["./packages/core/src"],
      "@udecode/slate-plugins-test-utils": ["./packages/test-utils/src"]
    }
  },

I'm pushing the fix to the branch.

[MikeAliG]

Interesting, I must have missed it while fixing circular dependency issues. I'll pull the changes and run tests once this is pushed. Thanks.

[zbeyens]

I was thinking about slate-plugins-helpers in our latest discussion. I think this package is not needed: let's keep everything that is tree-shakeable in slate-plugins and the rest will be extracted in separate packages (e.g. slate-plugins-x).

• slate-plugins-core: no deps
• slate-plugins-common deps:
  • core
• slate-plugins deps:
  • common
  • core
• slate-plugins-x deps:
  • common
  • core
  • slate-plugins (e.g. for the paragraph type)

[MikeAliG]

Oh I just understood what you meant. You meant something like below as per previous discussion, correct?

• slate-plugins: having all the below dependencies (requirement: tree-shaking) as a quick start.
• core: core functionality providing EditablePlugins
• common: I would rather call it like this as it contains many types of function: transforms, queries, utils,...
• slate-plugins-preview-md
• slate-plugins-deserialize: only has these dependencies
  • deserialize-html
  • deserialize-md
• slate-plugins-serialize: only one dependency (a bit useless then)
  • serialize-html
• slate-plugins-dnd
• slate-plugins-elements
  • a package for each element
• slate-plugin-marks
  • a package for each mark
• a package for each handler, normalizer, widget

Either way, I am positive. I will apply those changes once the common package is done.

Thanks,
Ali

[zbeyens]

The commit has been pushed. I've moved common/constants and common/plugins to slate-plugins, and pipe to core.

Indeed, let's start with:

• core: for those only using EditablePlugins and pipe. No deps.
• common: for those using the helpers and not the plugins. This could be in slate-plugins when thinking only about tree-shaking, but it's keep it. Deps:
  • core
  • lodash
  • utility-types
• slate-plugins: for those using all the tree-shakeable plugins. Deps:
  • common: reexport all
  • core: reexport all
  • image-extensions
  • is-hotkey

Following plugins have slate-plugins (no need of common and core as it's reexported) as deps:

• slate-plugins-md: (de)serialize markdown. Deps:
  • remark-parse
  • remark-slate
  • unified
• slate-plugins-html: (de)serialize html. No deps.
• slate-plugins-dnd deps:
  • slate-plugins-components?
  • @react-hook/merged-ref
  • @tippyjs/react
  • react-dnd
  • react-dnd-html5-backend
• slate-plugins-preview-md (better name? anyway this plugin is less important) deps: (decorateCodeBlock should be removed from slate-plugins)
  • prismjs
• slate-plugins-components deps:
  • react-use
  • @tippyjs/react
• slate-plugins-components-fluent deps:
  • slate-plugins-components
  • @uifabric/styling
  • @uifabric/utilities

Not sure which of these third-party deps are tree-shakeable. Which could be kept in slate-plugins otherwise. Feedback is welcome!

[MikeAliG]

This is great! However, about the slate-plugins-preview-md and slate-plugins-md we can make a package for both of them under the same name as slate-plugins-md or slate-plugins-markdown since they are very similar in their function and usually used together and also it'll follow the slate-plugins-html naming pattern but let me know if you have other plans for them.

So now that the common module is done, which package would you suggest to start with next? If we start with slate-plugins-components, we can start thinking about the new API as well.

On a different topic, I saw the collaborative editing in the todo list for V1 which is great. Do you still have plans to support collaborative editing? If so we can keep that in mind for the new API and we can start supporting it in the new release.

Please let me know your ideas, I'd like to hear them.

[zbeyens]

Great suggestion, let's put the preview in slate-plugins-md.

Extracting all the components slate-plugins-components(-fluent) is good next step.

As for the collaborative editing, it's still in the roadmap, not necessarily for V1, but that would be great!

[MikeAliG]

Fantastic! I'll start the modularisation of slate-plugins-components and hopefully posting the progress as I move along.
As for collaborative editing, I have some ideas based on our previous use cases but since it's not a priority, I'll just have that in mind for useSlatePlugin() design to tackle possible breaking changes that might arise and I'll send the proposals as we progress.
Please let me know if there's anything you'd like me to keep in mind with the modularisation.

[zbeyens]

@MikeAliG Let me know when you take a break so I can work on this in my free time

[MikeAliG]

Hello! I hope you are well. Thank you very much for your wait and sorry if I took a lot of time to process the code, but please feel free to change/work any part that I'm working on, I am totally open to that and in fact I would thank you a lot as well.

As for the new API, I've been trying to understand the underlying issues (if they rise) but I just pushed a small commit about the new design of slate-plugins-components about the new design which goes hand to hand with the zustand's API but I don't know if it's better to keep it in slate-plugins-components or I should move it to slate-plugins as a main package. There are several things I had in mind with this new API but please feel free to change any part as you wish:

1. The new API removes most of the boilerplate. For example the following code could be simplified dramatically without performance issues (Although I have some other considerations for performance which I will talk about below.)

import React, { useMemo, useState } from "react";
import { createEditor, Node } from "slate";
import { withHistory } from "slate-history";
import { Slate, withReact } from "slate-react";
import {
  ParagraphPlugin,
  BoldPlugin,
  EditablePlugins,
  ItalicPlugin,
  UnderlinePlugin,
  pipe,
} from "@udecode/slate-plugins";
import { useSlatePlugins } from "@udecode/slate-plugins-components";

export const Editor = () => {
  const { editor, editorValue, plugins, setEditorValue } = useSlatePlugins({
    plugins: [
      ParagraphPlugin(),
      BoldPlugin(),
      ItalicPlugin(),
      UnderlinePlugin(),
    ],
    withPlugins: [withReact, withHistory],
    initialValue: [
      {
        type: options.p.type,
        children: [
          {
            text: "This text is bold, italic and underlined.",
            [options.bold.type]: true,
            [options.italic.type]: true,
            [options.underline.type]: true,
          },
        ],
      },
    ],
  });

  return (
    <Slate
      editor={editor}
      value={editorValue}
      onChange={(newval) => setEditorValue(newval as SlateDocument)}
    >
      <EditablePlugins plugins={plugins} placeholder="Enter some text..." />
    </Slate>
  );
};

2. The new API is future-proof for collaborative editing as with this approach, we are in control of the state internally as we can swap the useState with an immutable store which handles merging of the Node and documents with cursor control.
3. We can make defaults accessible without importing for example BasicPlugins can encapsulate withPlugins: [withReact, withHistory] and plugins: [ ParagraphPlugin(), BoldPlugin(), ItalicPlugin(), UnderlinePlugin(), ],
4. As Previously discussed, it reduces render time and handles unwanted processes (Zombie child processes)
5. Internally everything can be immutable and we can satisfy the packages to be used in React API based frameworks such as Fre and Preact with no breaking changes.
6. It allows the customisation of components at a higher level without complexity.

So right now, I am working to make all the basic elements renderless to be able to test these theories and do some benchmarking to make sure that nothing is being lost and everything is behaving perfectly but as it might take some time, please let me know if you want me to work on some other parts or if you want to change anything in any part, I will gladly change it.

As always thank you for your attention, and please feel free to not only let me know about your ideas but also please change/work on any part, it will help me as well.

Many thanks,
Ali

[MikeAliG]

I am currently transferring the components [elements, marks, widgets] into the new slate-plugins-components package.
We are currently two options we can have in order to have a smooth transition:

1. Accepting the component in the Plugin instance itself:

export const ParagraphPlugin = (
  // These options would be required because they should receive the component: 
  options: ParagraphPluginOptions
): SlatePlugin => ({
  renderElement: renderElementParagraph(options),
  deserialize: deserializeParagraph(options),
  onKeyDown: getOnHotkeyToggleNodeTypeDefault({
    key: 'p',
    defaultOptions: DEFAULTS_PARAGRAPH,
    options,
  }),
});

where options receive:

export interface RenderNodeOptions
  extends RootProps<RenderNodePropsOptions>,
    HotkeyOptions {
  /**
   * Type of the node.
   */
  type?: string;

  /**
   * React component of the node.
   */
// In this approach this option would be required in here.
  component: React.ElementType;
}

2. We can have placeholder component such as a <div /> and we can replace them by the global store.

Both are possible with their own advantages and disadvantages but please let me know your ideas on this, which one would you prefer?

[MikeAliG]

Ok, I think the multiple editor issue is solved. I've pushed the code into the modularisation branch if you'd like to take a look. With the following approach we can have hook-level/component-level global store with fallback on useState in which case the users can instantiate multiple useSlatePlugins with no issues/perf costs and they can mix and match the components of those editors with each other. The new approach is like this:

import { useEffect, useMemo, useState } from 'react';
import { pipe } from '@udecode/slate-plugins-core';
import type { UseSlatePluginsOptionType } from './types';
import { slatePluginStore } from '.';

/**
 * @description This plugin is the base of your Editor, you can dynamically change the contents as you wish and internally, it will be automatically reflected on your editor
 * @see @todo LINK TO THE DOCS PAGE
 * @requires key
 * @param options UseSlatePluginsOptionType
 */
export const useSlatePlugins = (options: UseSlatePluginsOptionType) => {
  /**
   * @description This will allow component level store with fallback which allows multiple editor compositions.
   */
  const [
    {
      editor: stateEditor,
      withPlugins,
      plugins,
      setComponents,
      setWithPlugins,
      setEditor,
      setPlugins,
    },
  ] = useState(() => slatePluginStore());

  useEffect(() => {
    if (options.editor) setEditor(options.editor);
    if (options.components) setComponents(options.components);
    if (options.withPlugins) setWithPlugins(options.withPlugins);
    if (options.plugins) setPlugins(options.plugins);
  }, [
    options.editor,
    options.components,
    options.withPlugins,
    options.plugins,
  ]);

  const [editorValue, setEditorValue] = useState(options.initialValue);

  const editor = useMemo(() => pipe(stateEditor(), ...withPlugins), []);

  return {
    editor,
    withPlugins,
    plugins,
    setComponents,
    setWithPlugins,
    setEditor,
    setPlugins,
    editorValue,
    setEditorValue,
  };
};

Although, I put the key: string option in the options, it is not much needed but just in case. Also, with this approach we can have lazy initiation of the stores as well since useState is lazy.

Please let me know about your thoughts and ideas as well.

[zbeyens]

Thanks, will review this and PR some ideas

[lukesmurray]

I noticed in your example you set withPlugins in useEffect. A potential problem here is decorate is called on render and useEffect runs after render. Normally that's fine but decorate will be called on void nodes in the initial render since any withPlugin which supplies isVoid semantics will not be run until the effect fires. I've been trying to find a solution to this myself.

Ideally you are able to set withPlugins in the useMemo hook which creates the editor. But then if your withPlugins have dependencies (for example dynamically setting autoformat rules) the editor is no longer a singleton. Since you all are thinking about APIs in this discussion and are trying to be future proof I wanted to raise this issue here. At the moment it's not clear to me that a withPlugin can be dynamic.

[zbeyens]

Good to know! I was thinking that we can maybe save the history, reset the editor reference (useMemo deps) and restore the history.

[lukesmurray]

@zbeyens I got plugins with dependencies to work with my hook but I had to be a bit clever.
If you have free time and are interested here is a link to the relevant code. I would be so grateful if you had an alternative approach or ideas for improvement. I documented the implications of my design decisions in this section of the readme and show how to safely preserve history. The best part of this design is there are zero calls to useEffect which means renders always work.

I know this may not be immediately relevant to you since your plugins seem to be fairly static but I think once you or your library consumes start building plugins which use state or data from apis then this dependency issue is fairly challenging to deal with.

[MikeAliG]

Thank you for the follow ups and PR. So, now, I think we can move on to slate-plugin-components. What is the best approach in terms of component organisations? As per:

By feature: element, mark, etc. (all the current plugins), components-fluent
By component: core, plugins, components

I have some ideas but please challenge them:

Now that we have a global store, we can share the same store with the end user so that they can even modify it to their needs and overwrite the store values if they wish to do so.

In which case, we don't need the ParagraphPluginOptions since everything is being controlled by one central store:

/**
 * Enables support for paragraphs.
 */
export const ParagraphPlugin = (): SlatePlugin => {
    const {
      main: {
        components: { p },
      },
    } = useSlatePluginsStore();

  return({
  renderElement: renderElementParagraph(p),
  deserialize: deserializeParagraph(p),
  onKeyDown: getOnHotkeyToggleNodeTypeDefault({
    key: 'p',
    defaultOptions: p.DEFAULTS_PARAGRAPH,
  }),
})};

So now, if they need to change DEFAULTS_PARAGRAPH, they can change it in the global store and internally we'll use immer to prevent nested data immutability to provide better UX to the end user.

To summarise, this approach has some Pros:

1. A global store with immutability to change everything, even the DEFAULTS_OPTIONS and ease of use.
2. Prevents prop drilling for component composition for the end user and plugin creators.
3. Improves reactivity for on-the-fly editors*
4. Better observable pattern which we won't need to pass node types for every plugin.
5. Better error handling (if required later on)

*As for the on-the-fly editors, I mean the editors which the plugins need to change on-the-fly. An example of this would be the environment we need for the interactive docs where users can play around with the editor and write code to reflect the options.

Although these are fine, but please let me know if you have any thoughts on any of this. I'd like to know your thoughts.

[zbeyens]

Looks to be a cleaner approach to me!

This has also some cons:

• the store will be mainly (at least in the docs) accessible from the hooks, we need to also export the vanilla store to be usable without React (e.g. SSR): https://github.com/pmndrs/zustand#using-zustand-without-react
• it's not functional programming anymore (e.g. all the queries helpers with editor parameter and options like type). I think we can still apply the functional programming to the common/queries and common/transforms helpers for easy usability and testing.

Next steps:

• let's maximize the use of the store for the common/repeating SlatePlugin options (e.g. types, components) without losing the types. Internally we will use both store selectors and functional programming.
• only a hook / React component can use a hook, so all the plugins will be renamed to use...Plugin
• SlatePlugin will need to return nodeKeys: string[] which will be the keys to store the components/types in the store. These are constants defined by the plugin. We can use the ones used in DEFAULT_...(e.g. p, li, ul, ol, h1, h2, etc.). useSlatePlugins will merge all the nodeKeys plugins. f.e. the HeadingPlugin will return nodeKeys: ['h1', 'h2', 'h3', 'h4', 'h5', 'h6']
• SlatePlugin will need a
• new store fields:
  • nodeTypes = Record<nodeKeys: string[], nodeType: string>: reminder that we let the user modify the type because this will be stored in the slate document, i.e. the database.
  • components = Record<nodeKeys: string[], component: FunctionComponent>
• new selectors (I think we can omit doing useSlatePlugins... as it gets long):
  • useNodeKeys(key)
  • useNodeComponent(nodeKey, key)
  • useNodeType(nodeKey, key)

Thoughts:

• currently we only have SlatePlugin type. We can create sub types of plugins like SlatePluginElement, SlatePluginMark, etc.
• f.e. nodeKeys will be returned only by SlatePluginElement, SlatePluginMark
• think about next options to put in the store/useSlatePlugins: hotkey, rootProps (this one may be tricky as it's strictly typed)

[MikeAliG]

That's great!

I agree with all of them. I'll start using separate selectors as my main task.

I will keep you updated.

[zbeyens]

@MikeAliG I'm currently working on this branch, I'll keep you updated!

[zbeyens]

@MikeAliG v1 alpha is released, completing all the requested features and even more! See the sandbox. I still need to write the migration guide and new docs.