Move default language setting to Tokenizer

[saint1991]

1. Add a preserve feature to CodeNode

In the original implementation, CodeNode is removed when the backward deletion occurs at the beginning of the Node.
In some use cases, implementing a code editor, such behavior is not needed.
The "Preserve" setting is for making it controllable.

2. Move the default language setting to Tokenizer

The former PR #3243 enabled to use arbitrary Tokenizer other than Prism and 'javascript' is not always appropriate to the default so make it a Tokenizer setting.

This PR contains 2 changes but I'm happy to divide them if required.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated
lexical	✅ Ready (Inspect)	Visit Preview	Dec 1, 2022 at 10:49AM (UTC)
lexical-playground	✅ Ready (Inspect)	Visit Preview	Dec 1, 2022 at 10:49AM (UTC)

[thegreatercurve]

@saint1991 The default language changes seems fine, but I think we'd want to leave out the preserve flag. Users can just implement this themselves by extending the CodeNode and overwriting the collapseAtStart method.

[saint1991]

@thegreatercurve

Thanks for your kind review!
Thanks to #3367, the code something like below works as I expected.

Thank you so much for the great work!
I've removed commit 1. which is no longer needed as you mentioned.

export class CodeEditorNode extends CodeNode {
  static getType(): string {
    return 'code-editor';
  }

  collapseAtStart(): true {
    return true;
  }
}

export const $createCodeEditorNode = (language?: string | null | undefined): CodeEditorNode => {
  return new CodeEditorNode(language);
}

const editor = createEditor({
  nodes: [
    CodeEditorNode,
    CodeNode: {
      replace: CodeNode, 
      with: (node: CodeNode) => $createCodeEditorNode(node.getLanguage())
    },
    CodeHighlightNode,
  ]
})

[thegreatercurve]

This should just be enough:

Suggested change

	declare export function $createCodeNode(language?: string | null): CodeNode;
	declare export function $createCodeNode(language: ?string): CodeNode;

[saint1991]

@thegreatercurve
Thanks. Fixed it.

[fantactuka]

I'm curious if it's better to have default language as a param to registerCodeHighlighting and CodeHighlightingPlugin. Then you can have single tokenizer but custom default language based on specific surface needs

[saint1991]

I think it's also OK, but I put the setting here because the supported languages depend on Tokenizer's implementation. The setting 'javascript' is one of the proper choices for PrismTokenizer because Prism supports it.

In terms of the use of the library, the difference is that users should specify the default language even when they use PrismTokenizer by default.