JupyterLab 4.0 migration

[krassowski]

References

Related to #917.

Features:

• signature (works)
  • active argument highlighting
• hover (works)
• completion
  • users now need to manually enable "autocompletion" and "documentation panel" in settings - and there are two sets of completer settings; we probably should probably override the default one.
  • start/end ranges are set as expected
  • model and renderer precedence:
    • wait for Adds rank to ICompletionProvider jupyterlab/jupyterlab#14800
    • two upstream issues:
      • Completion providers list is not refreshed dynamically jupyterlab/jupyterlab#15016
      • Completer model and renderer are not updated on completer update jupyterlab/jupyterlab#15017
  • results are correctly filtered (with subject to the above
  • completer CSS class
  • continuous hinting detection based on trigger characters:
    • shouldShowContinuousHint needs context, fetch needs reason jupyterlab/jupyterlab#14806
    • Pass completion context to shouldShowContinuousHint() jupyterlab/jupyterlab#15015
• diagnostics
  • integration
  • tags (e.g. deprecated or unused) - upstream question
  • robustness - upstream issue
  • fix tooltip positioning CodeMirror should be configured to use absolute positioning of tooltips jupyterlab/jupyterlab#14857
  • enable gutter?
  • update CSS for new class name
  • tooltip dark mode styling is off
  • bump @codemirror/lint version in JupyterLab
• rename
• highlights
• syntax (implemented by swapping mimeType)
• jump to (works, needs more testing)

General:

• disabling features
• statusbar
• settings
  • when settings are open during startup the form renders generic fields; this may be resolve in lab 4.1 as we may be able to remove delayed transformation of settings

Testing/CI:

• builds
• tests pass:
  • unit
  • integration
• documentation
  • builds
  • updated

Nice to have:

• re-enable immutable installs for yarn 3 (the problem is that hash for file: protocol includes any rubbish and needs to be recomputed on each commit; there probably should not be a hash at all, and indeed portal: does not have it but is not supported by learna; official recommendation is to use workspace:, see comment which we did. Related issue and SO.
• upstream: should ICompletionContext.widget be IDocumentWidget rather than Widget? (because we want to access .context.path); depends on where we believe completer will be used
• add a default type for WidgetLSPAdapter<T> generic to avoid WidgetLSPAdapter<any>

Upstream PRs:

• Separate LSP settings handling out to another plugin jupyterlab/jupyterlab#14786 (required for custom settings)
• Expose serverCapabilities, provides, and updateLogging jupyterlab/jupyterlab#14712 (needed to remove @ts-ignores on non-exposed methods)
• Port LSP connection manager fixes/enhancements jupyterlab/jupyterlab#14714 (needed to catch up with fixes here, including no shadow FS for pyright)
• Fix CodeMirror tooltip positioning and tooltip background jupyterlab/jupyterlab#14858
• Pass completion context to shouldShowContinuousHint() jupyterlab/jupyterlab#15015

Issues:

• LSP follow-up checklist jupyterlab/jupyterlab#14711
• this context gets lost in modelFactory jupyterlab/jupyterlab#14805
• How to get WidgetLSPAdapter/VirtualDocument from CodeMirror event? jupyterlab/jupyterlab#14710
• There is no sufficient access to LSP adapters (WidgetLSPAdapter) from widgets jupyterlab/jupyterlab#14783
• shouldShowContinuousHint needs context, fetch needs reason jupyterlab/jupyterlab#14806
• Completion providers list is not refreshed dynamically jupyterlab/jupyterlab#15016
• Completer model and renderer are not updated on completer update jupyterlab/jupyterlab#15017

Code changes

User-facing changes

• Diagnostics gutter can be enabled in settings (this is experimental, just exposing what @codemirror/linter gives us out of the box)
• Highlights now distinguish between read (variable access)/write (variable definition)/text (fuzzy matches) - please feed back on styling:
  
• To enable autocomplete two settings need to be enabled, see Autocompletion configuration is a tedious two-step process jupyterlab/jupyterlab#15022
• But autocomplete does not work yet (PR is open - Pass completion context to shouldShowContinuousHint() jupyterlab/jupyterlab#15015)
  The following settings got removes:
• disableCompletionsFrom and kernelResponseTimeout - these needs to be implemented in JupyterLab itself, see discussion in this upstream issue
• showDocumentation - an identical setting is now present in core JupyterLab

Backwards-incompatible changes

• .lsp-completer class got removed and instead body[data-lsp-completer-layout] selector is being used; this is suboptimal CSS performance-wise and may be changed.
• feature registration is now handled by ILSPFeatureManager from upstream @jupyterlab/lsp package

[github-actions]

👈 Launch a binder notebook on branch krassowski/jupyterlab-lsp/jupyterlab-4.0

[krassowski]

The lerna run test failures are odd - there are no errors and it all passes locally.

[krassowski]

I plan to merge this soon and iterate on remaining tasks after releasing an alpha. The robot tests heavily rely on diagnostics including the title in DOM but this is now what lint extension in CM6 does. One way forward would be to rely on diagnostics panel and only check for presence/absence of diagnostic ranges in the editor as needed.

[krassowski]

Merging as it is getting uncomfortably long:

• unit tests, lint, smoke tests pass
• works locally
• all remaining items are tracked in issues (here or upstream)