Semantic highlighting for F# and C# cells

[Jonathan-Schaefer-git]

A cell editor currently colors code with Monaco’s lexer alone, which gives a type, a function, a module and a value the same color. In a C# or F# cell that hides most of what the code is about. I would like to add semantic highlighting. I have a working build for C# and F# cells, for both verso serve and in the VS Code webview. I built it to test the notebook experience for my own library, but I think it is a change every Verso user would welcome.

This discussion is mainly to agree on the shape of the contract, so that other languages can build on a shared foundation.

The kernel side is an optional interface beside ILanguageKernel:

public interface ISemanticTokenProvider
{
    Task<IReadOnlyList<SemanticToken>?> GetSemanticTokensAsync(string code);
}

public sealed record SemanticToken(int Line, int StartColumn, int Length, string Type,
    IReadOnlyList<string>? Modifiers = null);

Tokens are cell-relative and typed by the LSP’s standard names (SemanticTokenTypes: namespace, type, class, struct, enum, interface, typeParameter, parameter, variable, property, enumMember, event, function, method, keyword, operator; SemanticTokenModifiers: mutable, static). null means “could not analyze, keep what is shown”, as diagnostics already do. Verso.Host exposes it as kernel/getSemanticTokens. F# uses FCS’s GetSemanticClassification, C# uses Roslyn’s Classifier.

In the editor, a Monaco semantic tokens provider follows the diagnostics rules. As in, nothing is asked while a cell runs, and requests go one at a time. Verso’s themes color each token type with their existing syntax colors. In VS Code the tokens take the theme’s own semantic colors, as in a source file.

Two questions on the contract:

Is an optional interface acceptable, or should it be a method on ILanguageKernel?
Union cases and enum members require one new syntax key for enumMember in the built-in themes. Is that fine? Without it they fall to the editor’s text color.

The PR would build on the fixes for #128 (PR #129) and #130 (PR #131), which I would like merged first.

[TorreyBetts]

Thanks for opening this before the PR, and for building it for both serve and VS Code. Highlighting types, functions and modules differently would be an improvement for C# and F# cells.

An optional interface is acceptable, and an enumMember key is fine. If a theme doesn't define it, please fall back to an existing color.

For the contract:

• Document positions as 0-based and cell relative like Diagnostic, with Length in UTF-16 characters, and add the token type names as constants in Abstractions.
• Share one type check between semantic tokens and diagnostics, so a pause in typing doesn't compile twice.

Both #129 and #131 are merged, so you're now able to open the PR.