Working with a custom editor extension

[skbitsp]

Description

Hi Folks, We at LinkedIn have created a custom extension using a Notebook Cell (a single cell which acts as a workbook). We want to enable the jupyterlab-lsp extension for the same editor. We got to the "Fully Initialised" state, but the LSP functionality like completion/highlighting, etc. doesn't work.

Example: on pressing "Tab", it doesn't get registered as a Keydown event and the extra spaces come up.

JupyterLab: 3.6.3
jupyterlab-lsp version: 4.2.0
Attached screenshots above.

Steps

To get it to "Fully Initiallized" state, overrode the existing FileEditorAdapter (@jupyter-lsp/jupyterlab-lsp/lib/adapters/file_editor/file_editor):

class SwbAdapter extends FileEditorAdapter {
  type = 'raw';

  constructor(extension, editor_widget) {
    super(extension, editor_widget);
    this.editor = editor_widget.content;

    this.initialized = new Promise ((resolve, reject) => {
      this.init_once_ready().then(resolve).catch(reject);
    });
  }

  context_from_active_document() {
    let editor = this.widget.context.editor;
    let ce_cursor = editor.getCursorPosition();
    let root_position = PositionConverter.ce_to_cm(ce_cursor);
    return this.get_context(root_position);
  }

  is_ready = () => {
    return (
        !this.widget.isDisposed &&
        this.widget.context.isReady &&
        this.widget.content.isVisible &&
        this.widget.context.sessionContext.session?.kernel != null
    );
  };

  get editors() {
    return [this.widget.context.editor];
  }

  get mime_type() {
    return MIME_TYPE;
  }

  get language_file_extension() {
    return ".swb";
  }

  get ce_editor() {
    return this.widget.context.editor;
  }

  get activeEditor() {
    return this.widget.context.editor;
  }

  create_virtual_document() {
    return new VirtualDocument({
      language: "python",
      file_extension: this.language_file_extension,
      path: this.document_path,
      overrides_registry: {},
      foreign_code_extractors: {},
      standalone: true,
      has_lsp_supported_file: true,
      console: this.console
    });
  }
}

and then registered it in the adapter manager:

adapterManager.registerAdapterType({
  name: 'workbook',
  tracker: workbookTracker,
  adapter: SwbAdapter,
  entrypoint: FileEditorContextMenuEntryPoint,
  get_id: function (widget) {
    return widget.id;
  },
  context_menu: {
    selector: '.swb-editor-cell', rank_group: 0, rank_group_size: 4
  }
});

Queries/ Doubts

• Provided the entrypoint as: FileEditorContextMenuEntryPoint. Didn't understand the significance for it. Should we create our own entry point?
• Provided the selector as selector: '.swb-editor-cell'. Not sure if it needs to be registered somewhere or we have to use any of the existing selectors.

Would really appreciate any pointers.
Thanks.

[krassowski]

The entrypoint indicates which set of commands from features should be used, it should not be a problem. Your selector looks fine too - assuming that the cell does have such a class (but it would not prevent diagnostics/highlights from working).

• Does the LSP server of your choice at the backend recognise files with .swb suffix correctly? Some servers will reject files with unknown extensions (which is why we ensure to append it).
• As of now we hook up into the JupyterLab native completer and extend it heavily; if a completer is not setup for a widget in the first place, this extension will not have a completer to enhance
• FYI In JupyterLab 4.0 the LSP adapter API got simplified moved into JupyterLab proper, and completer APIs got much improved for extensibility. The port of this extension is almost here (JupyterLab 4.0 migration #949). The core APIs are still getting finalised. To get a quick idea, see the new FileEditorAdapter and it's activation.

[skbitsp]

Thanks for your response. We are using JL 3.6.3 as of now.. Not sure when we will move to 4 :)

• Nice pointer of the .swb suffix. (which is why we ensure to append it) - how to do this?

• We are using the "Notebook raw cell", so maybe it doesn't come with the jupyter completer and we will have to implement it on our own? - Any pointers on how to import/ implement it?

But I think even if we implement the completer, the syntax highlighting, etc. should atleast work, which doesn't so the suffix thing also might be the issue.

We actually need to have it integrated with sql-language-server but was trying with jedi-language-server for a simple POC.

Attaching screenshots for reference:

[krassowski]

how to do this?

I meant changing

  get language_file_extension() {
    return ".swb";
  }

to

  get language_file_extension() {
    return ".py";  # and ultimately to `.sql` for `sql-language-server`
  }

Any pointers on how to import/ implement it?

In 3.x you need to add a completion connector and commands (against, getting simpler in 4.0). I would start by hooking up ContextConnector which suggests based on tokens in the editor and checking if it works (regardless of LSP), for 3.x see here.

[skbitsp]

Thanks for your inputs, after registering the CompletionConnector, the completion works:

const connector = new CompletionConnector({ session, editor });
const handler = completionManager.register({ connector, editor, parent: widget });

But syntax highlighting, hover, etc does not work. Will we have to register some additional connectors for it?

[krassowski]

No, no additional connectors required. Can you record messages sent between server and client? Are there any? There are some logging options in settings but these do not cover all messages unfortunately so best choice is to listen on the websocket (e.g. in Firefox devtools) or adding a logger on the server side.

[skbitsp]

So, I was trying to connect it with sql-language-server, have installed the server using "npm add sql-language-server -W". It works perfectly with notebooks. But for workbooks, I have changed its language to sql and the file extension to ".sql". So it gets connected to SQL (as can be seen from the Initialized box). But it still gives suggestions for python. Any ideas what can be the reason?

This is the updated swbAdapter:

class SwbAdapter extends FileEditorAdapter {
  type = 'raw';

  constructor(extension, editor_widget) {
    super(extension, editor_widget);
    this.editor = editor_widget.content;

    this.initialized = new Promise ((resolve, reject) => {
      this.init_once_ready().then(resolve).catch(reject);
    });
  }

  context_from_active_document() {
    let editor = this.widget.context.editor;
    let ce_cursor = editor.getCursorPosition();
    let root_position = PositionConverter.ce_to_cm(ce_cursor);
    return this.get_context(root_position);
  }

  is_ready = () => {
    return (
        !this.widget.isDisposed &&
        this.widget.context.isReady &&
        this.widget.content.isVisible &&
        this.widget.context.sessionContext.session?.kernel != null
    );
  };

  get editors() {
    return [this.widget.context.editor];
  }

  get mime_type() {
    return MIME_TYPE;
  }

  get language_file_extension() {
    return ".sql";
  }

  get ce_editor() {
    return this.widget.context.editor;
  }

  get activeEditor() {
    return this.widget.context.editor;
  }

  create_virtual_document() {
    return new VirtualDocument({
      language: "sql",
      file_extension: this.language_file_extension,
      path: this.document_path,
      overrides_registry: {},
      foreign_code_extractors: {},
      standalone: true,
      has_lsp_supported_file: true,
      console: this.console
    });
  }
}

[skbitsp]

We also added custom commands for the completor to work:

    app.commands.addKeyBinding({
      selector: '.swb-editor-cell .jp-InputArea-editor.jp-mod-completer-enabled',
      keys: ['Tab'],
      command: 'darwin:swb-complete'
    });

    app.commands.addKeyBinding({
      selector: `.swb-editor-cell .jp-mod-completer-active`,
      keys: ['Enter'],
      command: 'darwin:swb-select',
    });

    app.commands.addCommand('darwin:swb-complete', {

      execute: args => {
        console.log('inside workbook completer');
        app.commands.execute('completer:invoke', { id: workbookTracker.currentWidget.id });

      }
    });

    app.commands.addCommand('darwin:swb-select', {

      execute: args => {
        console.log('inside workbook select');
        app.commands.execute('completer:select', { id: workbookTracker.currentWidget.id });

      }
    });

Any ideas if we should we do something additional for SQL language?

[skbitsp]

Noticed 1 thing.. that the completer working in SQL workbooks is the default file editor completer and is not powered by LSP. Which might explain the reason why Syntax highlighting, hover, etc. doesn't work.
Notice the difference between autocomplete suggestions in notebooks and workbooks.

Also confirmed that the breakpoint never hits in completion_handler.js in the jupyter-lsp extension, which hits there incase of file editor or notebooks.

Any pointers on what we might be missing? The extension shows fully initialised in the bottom left panel.

[krassowski]

I would really need more details to say, e.g. logging from browser console (Is there "Cold not find a VirtualEditor suitable for the provided set of editors" message?), list of messages on the websocket wire, etc. Also, if you could share a minimum standalone reproducible example, I could take a look. If you cannot share publicly feel free to reach out via email/other platform.

[skbitsp]

Thanks for all the help! Was able to get this working by exporting all the required files in index.ts of the LSP extension:

export {FileEditorAdapter} from './adapters/file_editor/file_editor';
export {FileEditorContextMenuEntryPoint} from './adapters/file_editor';
export {PositionConverter} from './converter';
export {VirtualDocument} from './virtual/document';
export {EditorAdapter} from './editor_integration/editor_adapter';
export {LSPConnection} from './connection';
export {WidgetAdapterManager} from './adapter_manager';

and then importing in my extension. All functionalities working including Hover, Diagnostics Panel, autocomplete, etc. 😊